Reviewed and updated October 8, 2026.
Finding: Across these engagement records, our qualitative interpretation was that visible evidence helped people review a score, status or figure. This is a practitioner reading of the records, not a measured change in trust or adoption.
Summary
- We reviewed seven engagement or workstream records through the lens of the production gap: the distance between a working model and output people can evaluate and use. These are not seven independent production systems or seven distinct clients. The practice and certification programs share one production voice platform, and the retrieval reports describe the same product-help proof.
- The seven covered voice practice and certification, product help retrieval, conversation capture, a business case model, an operations platform, and an architecture exploration.
- One platform, hosting the two voice programs, is in production. The others are in development, built and awaiting live data, built as a proof or model, or exploratory.
- The live platform records scored practice sessions and participant status. Manual review suggested recurring delivery and questioning habits, but their prevalence and any resulting training improvement were not measured.
- The key lessons were about trust and access: tighten a grader that rewards surface mentions, build views from one data source, trace every figure to its source, define each metric once, and plan access as part of launch.
What these systems let an organization see
A scored record of every practice conversation
Status: live. On a voice practice platform, reps talk with an AI buyer, and every session is transcribed and graded. Each recap gives an overall score, category scores such as professionalism, and written feedback that counts filler phrases and flags closed questions posed as open ones. Scenarios can hold facts the AI buyer reveals only when a rep asks good questions, so a session can show whether the rep uncovered needs or only delivered a pitch. Reps use the recap to decide what to fix before the next attempt. Managers see their own people's sessions, scores, and completion status. They can read the transcript and play back the call, so follow-up starts from the call itself.
Patterns no single manager could see
Status: session data live; miss-rate reporting built and demonstrated. A manual review of scored recaps suggested recurring delivery and questioning habits across sessions. That review informed possible training priorities; it does not establish the prevalence of a skill gap or a training improvement. Reporting of criteria missed by team was also built and demonstrated, so trainers could inspect those patterns against the underlying sessions.
A status for every person
Status: live; per-leader report planned. For a time-boxed certification on the same platform, the program owner gets one row per person: invited, logged in, practicing, attempted, or certified, with last login. Each row also shows the due date and a past-due flag, so the owner can see who to remind before the deadline. A rollout view shows these counts per group, which helps the owner pace invitations and follow-up. Raw exports let the owner share status with each team leader, and a report that rolls each person up under their leader is planned.
Evidence behind scores, answers, and figures
Status: certification trail live, with recordings rolling out; the other two built as a proof and a model. Certification attempts keep the transcript and a pass or miss grid with a reason for each miss, and call recordings are being added to the same trail. Admins can review and, where needed, override a result using the recorded evidence. In a retrieval proof, every answer carried a citation trail so support staff could judge where it came from. A business case model tagged every input as client-provided, quoted in a meeting, external research, or assumption, so reviewers can see which numbers to challenge.
Signals about the system itself
Status: learner flags and response review live; usage reporting in progress. Learners can flag any session as too hard or unrealistic. Comparing flag rates with session volume by group shows two things: scenarios that draw few too-hard flags may be too easy, and a cluster of flags can point to knowledge the AI buyer lacks. Reviewers can also mark any AI buyer response and write the one it should have given, which builds a correction record that shapes future sessions. Usage reporting is designed to show practice volume over time and which activity drives AI cost.
Customer conversations as reviewable data
Status: prototype demonstrated; first release in development and testing. A conversation capture platform has been demonstrated inside a mock CRM. It turns each customer meeting into a record filed under the account and the deal. Each record holds the transcript, key points, action items, and attendees, and is meant to tell reps what to follow up on and who owns it. Competitive mentions go to the rep to accept or reject. Each rejection needs a reason, which becomes labeled feedback. In the prototype, extracted pain points, tools, and stated figures each carry the exact quote and a confidence score. A demo executive view ranks pain points by how many accounts mention them, normalizes mention rates per 100 conversations, and requires a minimum sample before any comparison. It is meant for executives deciding what to sell, build, or investigate, and runs only on synthetic data so far. See the aggregation architecture.
Operations data in one place
Status: built; live data connections planned. An operations platform for a process-driven team brings organizations, people, and work into one place. It shows where each deal stands, each person's tasks, and checks that warn when an item breaks a rule the team sets. It is built so each person can see what they need to do next, and the process owner can see what each deal needs at its current stage and which items to fix.
What we learned
Scoring: make the grader earn trust
Our AI grader gave credit for surface mentions: naming a required topic could earn points without the quality the criterion intended. We caught this in early calibration and tightened the grading criteria. Define the model's permitted criterion classifications, validate them against the rubric and a human-reviewed reference set, and compute totals and scale conversions in tested deterministic code. A formula in the prompt does not make model arithmetic reliable. Keep buyer instructions separate from grader-only material and test for instruction leakage. Transcription errors on names can mislead the persona or grader, so test domain vocabulary. Set the pass policy explicitly and review learner flags alongside session evidence. See calibrating AI graders.
Numbers: check claims against what was said, and label every input
Building a business case, we checked six load-bearing claims in our working drafts against verbatim meeting transcripts instead of AI meeting summaries. One was confirmed as stated, three were partly right or needed correction, and two could not be found and were flagged. One range an AI summary had altered was corrected to what the speaker actually said, and uncertain figures were modeled as ranges, not points. Every input in the model carried a source label, including plain assumptions. We also fenced speculation. Sensitivity analysis targeted the weakest assumptions, and speculative value sat in a separate tier, apart from the core total. One workbook fed every deliverable, and the dashboard was reconciled to it.
Reporting: one source of truth, plain labels
Views built at different stages can drift apart, so rebuild them from one shared data source. Label each measure plainly, and keep practice activity separate from formal results. Ship raw, exportable data first, then build the views people ask for. Where data sources define the same metric differently, name one source of truth for each metric and show that source next to the number. Imported data ages, so connect live data or label its age before users review it.
Retrieval: match how people actually ask
People do not phrase questions the way documentation reads. In the product-help proof, work focused on everyday shorthand and product-specific terms. The team replaced a flat 6,000-token context budget with design targets of about 1,000 for lookups and 5,000 or more for diagnostic questions. Engineers estimated that a fallback which counted words instead of tokens left technical-text context 10 to 20 percent under-filled when it ran; this was not a measured utilization result. Unmapped product terms fell back to broader search. Evaluate retrieval separately from answer quality and check mapping completeness before release.
Conversation data: capture for questions not yet asked
Label conversation data as it arrives and tie each item to its quote and speaker. Assess the evidence for the particular claim: a customer statement is evidence of what that person said, not automatic proof of an outcome or a financial return. Check source scope and corroboration where needed. Keep the cost of a problem separate from the return on a solution, and retain human review for system-of-record changes. Plan consent and an alternative for meetings where recording is not permitted.
Adoption: plan access as part of launch
Access is often the first adoption hurdle for any new tool. Treat sign-in, invitations, and device support as part of the launch plan, and give people a short guided introduction before their first session.
Exploration: access rules are the hard part
An exploratory architecture engagement demonstrated, on a sample database and demo data, agents built from product documentation that followed its workflows, and a role-scoped view of what each person's AI assistant may see. Deciding who may see what is the hard part of internal AI, and it stays a human judgment.
What the lessons have in common
A working model was only the start. Before launch, check three things for every number people will act on: a visible source, one definition, and a named person who can verify it. Then make sure the people who need that number can reach it without friction.
Method and limitations
This is a qualitative review, not a survey. For each engagement we read the engagement record, tracking history, and meeting transcripts, and labeled each data source as live, rolling out, built and demonstrated, or planned. Identifying details and client-internal figures are left out.
- Seven workstream records provide a narrow qualitative base. Some share a platform or proof, so they are not independent implementation samples. The lessons may reflect these organizations' ways of working more than the wider market.
- Only one platform is in production, so our observations about how people use data rest on a narrow base.
- We have not measured these effects against a control group.
- Most systems are still in development or review, so long-term outcomes are unknown. We will update this report.
For the broader path, see From prototype to production AI.
Related reading
For delivery scope, see custom AI development and private AI infrastructure and deployment. The reports on intent-aware retrieval and voice-assessment grading set out individual methods and limitations.
About ALLTIPLY Labs
ALLTIPLY Labs publishes what we learn delivering AI systems. To define the evidence your users need before acting on AI output, request a project review with the workflow, baseline and acceptance decision.
