Executive summary
- Client: an enterprise client.
- Problem: practice relied on live roleplay, which used manager time and gave leaders no easy way to see how reps handled buyer conversations.
- What the client can see now: a scored, transcribed record of each practice call, team progress for each manager, and learner flags on scenario realism. Reading recaps in volume also shows patterns across the sales organization.
- Main lesson: the value is in the aggregate. Scored practice at volume can show patterns that one-off roleplays never surface.
- Status: live in production. Some team reporting views are still rolling out, and a report ranking the most-missed scored components is built and demonstrated, not yet live.
The problem
Reps practiced by roleplaying with a manager or a peer. Practicing in front of colleagues is not always the best setting to learn.
The bigger gap was evidence. Sales leaders wanted to see how reps handled buyer conversations, so they could target training before customer meetings. Live roleplay gave them no easy way to see that across the sales organization.
What the client can see now
ALLTIPLY built a voice AI roleplay platform where reps talk to an AI buyer in the browser, and each call is transcribed and scored.
Each practice call, scored and on the record
After each session the rep gets an overall score on the scale set for that scenario (a graded scale for practice, a pass threshold for assessed scenarios), category scores such as professionalism, and written feedback. The feedback counts filler phrases and comments on questioning, including closed questions posed as open ones. The full transcript sits alongside it for context.
Managers and the program owner can open a rep's session, read the transcript, and usually replay the audio, alongside the score and duration. They work from evidence instead of memory, spot-check the grader, and can see which skills need work.
Team progress for each manager
In the core practice program, a manager sees only their own direct reports: who has started, how many sessions each rep has run, and their scores. Team views for programs with a pass mark, and broader roll-ups, are still rolling out. An executive dashboard shows the utilization trend over time. Usage reports for capacity planning are rolling out.
For programs with a pass mark, a rollout view shows the share of each group that has passed, how many practice sessions each group has run, invitation status, and logins. The program owner uses it to track rollout progress and can export the underlying data.
Patterns across the sales organization
With calls scored and transcribed at volume, the program owner can read recaps side by side and find patterns across the whole sales organization that no single manager sees. That separates gaps shared across the organization from gaps that belong to one rep.
Leaders can then aim the next round of training at the skills that need it most, based on evidence from practice rather than anecdote. ALLTIPLY has also built team reporting that ranks the scored components each group misses most often. That report is built and demonstrated, not yet live. Once it is live, those patterns can be read from a report as well as from recaps.
Learner flags on realism and difficulty
Learners can rate any session and tag a reason such as too hard or unrealistic. Reasons are counted for each group against session volume. The ratios point to different fixes: a very low "too hard" rate can mean a scenario is too easy, while a high rate can mean the AI buyer lacks knowledge reps need. The program owner and scenario builders use them to decide where to tune difficulty or add knowledge to the buyer.
A correction record for the AI buyer
Reviewers mark buyer responses up or down and write the response it should have given, building a record of corrections that shapes later sessions. Scenarios also hold hidden facts the buyer reveals only to good questions. That makes discovery testable: reading the scored transcript, a manager can see whether a rep uncovered the need or just pitched.
What we learned
- The value is in the aggregate. One scored roleplay helps one rep. Read together, many show which gaps recur across the organization, so training goes there first.
- Spoken practice changes what people retain. Leaders reported that saying material out loud to a buyer helped it stick better than reading it. Practicing with an AI buyer is also lower pressure than performing in front of peers.
- Collect a learner flag on every session, and set strictness on purpose. Flag-to-session ratios are a cheap calibration signal, and "too hard" often points to missing buyer knowledge, not only difficulty. Strictness is a policy choice, so decide it explicitly for each program.
- Keep what the grader knows away from the buyer. Keep grading instructions out of the buyer's prompt entirely. A buyer that knows the rubric can drift into acting like a trainer. ALLTIPLY enforces this with a hard boundary in the prompt design.
- Let domain experts tune the buyer. A wrong fact reaches every learner on that scenario, so vocabulary and facts should come from people who know the market. Also tell the buyer what it cannot check, such as a rep's personal experience, so it accepts those examples rather than arguing.
- Keep arithmetic out of the model, and keep scoring schemes apart. Models guess at formulas, so compute totals in code. Where the model must reason about scores, give it the exact formula. Keep scoring schemes on different scales in separate display logic.
- Give each role its own clearly labeled view. Each audience should have one current view, with counts labeled by what they measure.
- Give occasional authors a guided path. Scenario authors range from regular builders to occasional ones. Offer a structured mode for regular builders and a guided one for everyone else, and make clear which scenario text learners see and which only the AI sees.
- Set expectations before first use. Tell learners what each session tests so the first score is useful.
- Treat access as part of the product. Sign-in, invitations, and notifications are the first thing every learner meets, so test them on the devices learners use.
How it works
- Voice session: a rep talks to an AI buyer in the browser, mainly on desktop, with mobile browser access supported. The call is transcribed and a language model grades it against the scale chosen for that scenario. Scenarios with a pass mark add pass/miss criteria, a pass threshold, and a trainer override.
- Scenario builder: goal, sales-cycle phase, objections, difficulty, buyer personality, hidden and volunteered facts, and grading scale. AI-assisted drafting from a brief or file is rolling out, and ALLTIPLY has also built a seven-step guided builder for occasional authors.
- Product knowledge: documents embedded for vector search in PostgreSQL, with tags suggested on upload and an in-app assistant for questions.
- Controls: grading scales and admins for each group, an audit trail with restorable deletes, and a prompt library editable without code.
- Operations: model calls run through a router with a standard API, so the model provider can be swapped. Third-party voice and optional avatar services run alongside LLM tracing and error monitoring.
Related
Want this kind of visibility into your own sales team? Talk to us.
