AI-scored voice sales certification: data and lessons

Executive summary

Client: an enterprise client.
Problem: the client had completion data, not skill data.
Per rep, the client can now see: a criterion-level record of each scored session, and each rep's status from invite to certified.
Across the program: rollout funnels by team, organization-wide communication patterns, learner difficulty flags, and usage reports for AI capacity planning. A view of the criteria teams miss most is built and demonstrated.
Main lesson: AI graders give credit too easily when a criterion can be met by naming a topic. Criteria must describe what good sounds like, backed by human review and override.
Status: certification, scoring, and most reporting are live. Reporting-line views and dispute replay are rolling out, and a per-leader PDF report is planned.

The problem

Course completion shows who watched. The client could not see who had started practicing, who could handle a skeptical buyer, or which skills each team kept missing.

What the client can see now

The certification runs on a voice AI roleplay platform ALLTIPLY built. Reps talk to an AI buyer, and an AI grader scores the session. The value is in the data that scoring produces. Each view below is live unless its status line says otherwise.

A scored record of each session

Each practice and certification attempt produces an evaluation: the score against a set pass mark, and a pass or miss result for each criterion, with the reason for each miss. The record also holds the transcript and a read on professionalism and filler words. There is one record per rep, per session.

Decisions: pass or fail, and what a rep should fix before the next attempt.

Each person's status and practice pattern

One row per rep shows their stage: not invited, assigned, invited, logged in, practicing, attempted, or certified. Each row also shows the due date, a past-due flag, and last login. Practice data sits alongside it: sessions per rep, whether a rep started and stopped, and how many sessions it took to certify.

Decisions: who to remind before a deadline, who has never logged in, and who is practicing versus struggling.

A rollout control view

For each part of the organization, a control view shows the funnel from headcount to assigned, logged in, and certified, team by team. It adds "needs attention" and "ready to invite" queues, plus leaderboard and at-risk lists. These are lists to check, not automated alerts. It is the main operating view for a program owner and sponsoring leaders.

Decisions: how fast to widen the rollout, which teams to push, and how each team is tracking as the deadline approaches.

Status by reporting line

A hierarchy view shows each person's status, session count, and scores, rolled up through the reporting line. Raw data can be exported with hierarchy columns, which the program owner splits into one report per leader.

Decisions: what to send each leader, and whose teams are behind.

Status: raw export is live. The hierarchy view, and views that show each leader only their own people, are rolling out. A per-leader PDF status report is planned.

The criteria teams miss most

Team reporting was extended to show which criteria a team, or a larger group, misses most often.

Decisions: which skills to retrain, team by team.

Status: built and demonstrated to the client.

Patterns across the whole organization

Pooling session results surfaces recurring patterns across the organization that no single manager sees, such as common phrasing or questioning habits.

Decisions: where to focus the next round of sales training, backed by evidence instead of anecdote.

Status: a reading across pooled session results, not a standing dashboard.

Learner difficulty signals

Reps can rate each session up or down with a reason such as "too hard" or "unrealistic". The reasons are counted against session volume for each part of the organization. "Too hard" flags can line up with reps mentioning things the AI buyer does not know, which points to a content gap, not only to difficulty.

Decisions: where to adjust difficulty, and where to add knowledge to the AI buyer.

An evidence trail for disputes

Admins can review any certification attempt with its transcript and override the AI result where that review finds it wrong. Session review adds the audio recording and per-criterion evidence, so a disputed call can be replayed with a leader.

Decisions: whether a disputed result stands, and whether a low score reflects the rep or a tool fault.

Status: review and override are live. Audio replay and the per-criterion evidence view are rolling out.

AI usage and capacity

Usage reporting shows how AI demand changes over the course of a program.

Decisions: how to plan AI capacity.

Status: ALLTIPLY analysis, shared with the client as usage reports.

What we learned

Write criteria that describe quality, not topics. In a side-by-side human review of an early transcript, we found the grader had credited a rep just for touching each required topic, without checking whether the questions were open or the rep delivered the point in the expected way. We tightened the grading instructions on both. More in Calibrating AI graders.
Make dependent criteria explicit. In testing, the grader gave credit for a supporting example even though the rep never raised the topic it supports. A supporting criterion should not pass when the one it depends on was missed.
Add custom vocabulary early. The speech-to-text layer misheard a company name, and the AI buyer then "corrected" a rep who had said it right. Product and company names belong in the vocabulary from day one.
Practice and certification need different feedback. A live tally of criteria helps in practice sessions but gives answers away in a certification attempt, which has no hints. We kept the tally for practice and put the per-criterion result in the post-session report.
Set the pass bar on purpose. A baseline certification and a skill-development track need different pass bars, and learner feedback is a useful check on either. One certification rarely fits every role, so we built role variants, each with its own criteria, buyer persona, and strictness.
Plan content refreshes for the AI buyer. The buyer knows only the material it was given, while product content keeps growing. We told the buyer to accept claims it cannot check, such as a rep's own experience, instead of disputing them.
Try to break the AI buyer before reps do. Deliberate attempts turned up a voice that switched accents mid-session, a conduct strike that did not fire, and a call the AI ended with no explanation, which looked like a system error. A session the AI ends must say so on screen. To tune a persona faster, we ran voice tests alongside AI-to-AI sessions.
Treat sign-up as part of the product. Give invite links a sensible lifetime, test sign-up on the devices people actually use, and state clearly which devices are supported.
Stage the rollout. An early pilot group that takes the certification under the same conditions as reps surfaces wording and scoring problems while stakes are low. Inviting in waves lets invitation and login issues surface before everyone is invited, and planned release windows keep fixes from disrupting sessions.
Separate import, assignment, and notification. Accounts can then be loaded ahead of any announcement, and notifications go out only when the program owner chooses.
Keep one source of truth and name metrics precisely. Every screen should draw on the same data, and each metric should have a single, unambiguous name, for example keeping total sessions distinct from certification attempts.
Plan AI capacity for the peak. Demand is highest while a certification window is open, so size for that peak, not for average volume.

How it works

A fixed set of criteria that reps can cover in any order, each scored pass or miss, with grading scales set for each certification variant.
An AI buyer that keeps to a set call length, ends the call in character when time runs out, and ends off-topic sessions after repeated strikes.
Bulk import that builds the reporting hierarchy, with notifications that can be switched off.
A reusable certification template and AI-assisted scenario drafting, both rolling out.

Planning a certification program? Talk to us.

What AI-scored voice certification showed about a sales organization