Calibrating AI graders for voice assessment

Reviewed and updated October 8, 2026.

Finding: Once an AI grader's traceable errors are found and addressed, scored voice practice can show leaders the evidence behind each result, where reps struggle, and how their reps speak to buyers.

Summary

The sales organization now has data it did not have before: a scored record per session, with the reason for each missed criterion on certification attempts, manager replay of practice sessions with transcripts, learner difficulty flags, and per-session delivery feedback that, read across reps, shows which speaking habits recur. A fuller evidence trail for certification attempts is rolling out, and reporting on the criteria each team misses most has been built and demonstrated.
In a pilot before wider rollout, a human review of one transcript found the grader had scored it well above what the performance warranted. It gave credit for mentioning a required topic, even when the point was delivered poorly and no real open-ended question was asked. The evaluation prompt was tightened before wider rollout.
The problems we traced were specific, not vague model leniency: grading rules the grader was never given, display logic and labels inherited from another scale, and flaws in the conversation being graded (a transcription error on a proper name, and a buyer persona that hinted at answers).
Treat the pass bar as a documented policy choice, which can differ by program purpose.
Managers can replay practice sessions, and admins can review and override certification results. A transcript quote behind every criterion is the next practice we recommend.

Where this comes from

ALLTIPLY built and has operated a voice AI practice platform for a client's sales organization. One platform carries two programs: a roleplay training program scored on a points scale, and a sales certification program scored against pass-or-miss criteria and a set pass mark. The production voice-platform operating report describes the platform record. This field report covers the grading observations from both programs. In both, a rep speaks with an AI buyer in the browser, the conversation is transcribed, and a language model grades the transcript.

What the organization can now see

A trusted grader turns each practice conversation into data. Each view below does two jobs: it tells someone what to do next, and it gives the program owner a way to check the grader. Each view is live unless its status says otherwise.

A scored record per session. Training sessions get an overall score on a points scale, with category scores such as professionalism. Certification sessions are scored against the pass mark, with a pass-or-miss grid and the reason for each miss. Scored sessions also carry written feedback on delivery habits: filler phrases with counts, non-sequiturs, and closed questions posed as open ones. Reps use it to decide what to fix next, and trainers and managers use it to decide where to spend their time. The reason on each miss is also where a reviewer starts when a score looks wrong.
An evidence trail behind each result. Status: live for practice sessions, rolling out for certification. Managers can already replay practice sessions on their team with the transcript, score, and duration. For certification attempts, admins can review the transcript and override the result. Audio and a per-criterion evidence view for certification are rolling out. The trail lets a program owner separate a tool or scenario fault from a weak performance. As general practice, capture audio and evidence for every session so reviews have a complete trail.
Learner difficulty and realism flags. Any session can be flagged "too hard" or "unrealistic", and flags are counted by group against session volume. Comparing rates across groups shows where a scenario sits against its intended difficulty. A very low rate can mean a scenario is too easy. A high rate can reflect difficulty, or a buyer that lacks knowledge reps need to discuss. Read that way, the flags show scenario builders where to add knowledge to the buyer.
A correction record for the AI buyer. Reviewers mark buyer responses up or down and write the response it should have given, which fixes wrong facts and unrealistic vocabulary. Because a correction reaches every learner who takes that scenario, it works best in the hands of domain experts.
Criteria missed most, by team. Status: built and demonstrated. Team reporting was extended to show which criteria teams and groups miss most often. Sales leaders and trainers can use it to choose what to retrain, and for whom.

What volume surfaces

Read across many sessions, per-session feedback on delivery habits shows whether a habit belongs to one person or recurs across a team. The kinds of habits this feedback can flag include filler phrases, sentences that change direction midway, weak open-ended questions, and check questions that invite a yes or no instead of opening discovery.

Reading feedback across scored sessions can help a reviewer find recurring habits that one-off roleplays may miss. A shared rubric aims to apply consistent criteria to reps taking the same scenario; consistency does not establish objectivity, fairness or grading accuracy. Training priorities such as questioning technique and concise delivery can then be checked against session evidence. An inflated grader weakens that picture, because credit given too easily can hide the gaps that criterion results should expose.

What it took to trust the scores

Of the problems we traced, two were grading flaws. The rest were display problems and flaws in the conversation being graded, which make scores harder to trust even when the grading logic is sound. Some we fixed by tightening the evaluation prompt, walling trainer-only content off from the buyer, and changing how the buyer handles claims it cannot verify. For others, including explicit rules between dependent criteria and custom vocabulary for transcription, we set out the practice we recommend.

Mentioning a topic earned credit for delivering it

The grader gave credit for naming a required topic, whether or not it was delivered well or followed by a real open-ended question. We tightened the evaluation prompt to grade question quality and how well each point was delivered, not just coverage. Mentioning a topic is not the same as delivering it, so each criterion has to say what good delivery sounds like.

Dependent criteria had no rules

In testing, the grader credited a supporting point for a parent point the rep never actually made. A supporting point cannot count without its parent. Write that dependency as an explicit rule rather than expecting the model to apply it.

Scores inherited another scale's display and labels

One early score displayed far too low because percentage scoring reused display logic from another scale. A leftover readiness label from that scale also appeared beside a near-perfect percentage. These were display problems, not grading errors. Compute and display totals in code with a tested formula per scale, and review labels and defaults when adding a program. Model-generated arithmetic can be inconsistent, so validate it rather than treating a supplied formula as a reliability guarantee.

Transcription errors changed the facts

Speech-to-text misrendered a proper name in one conversation. The AI then "corrected" a rep who had said it correctly. An error like this can mislead the buyer and the grader alike. We recommend custom vocabulary for the product names and domain terms reps rely on, in the pipeline from day one.

The buyer gave away the answers

Trainer-only scenario content, including instructions meant for grading, reached the AI buyer. It sometimes acted like a trainer or seller and volunteered points reps should have had to draw out. That can make a test easier than designed. ALLTIPLY separated buyer instructions from trainer-only material in the prompt design. This is a design control, not a tested security boundary; adversarial and regression tests would be needed to establish resistance to instruction leakage. Separately, the buyer disputed examples from reps' own past work that it could not verify. It was adjusted to accept those examples as statements from the rep, without treating them as independently verified facts. Parallel voice and agent-to-agent tests were used to support persona tuning.

Treat the pass bar as policy

A pass bar is a policy choice, not a model setting. Programs with different purposes can reasonably set different bars, and programs that certify a specific skill usually warrant stricter criteria. Document the choice so later reviewers can tell a policy decision from a change in the grader. Learner difficulty flags then show where each program actually sits against its intended bar. The platform supports this directly: each group can have its own grading scale, criteria, buyer type, and difficulty, and an AI assistant helps draft new scales.

What we recommend

The fixes to the evaluation prompt and the buyer are in place. The practices below are what we recommend for any AI grading program:

A transcript quote behind every point. The grader quotes the words that satisfy each criterion, or states that none do. Reps and reviewers then see exactly what earned each result.
Explicit rules between dependent criteria. A supporting criterion cannot pass when its parent was missed. Enforce this in the rubric or in code.
Grade criteria separately or in small groups. Each result becomes easier to explain. We have not tested whether it also reduces inflation here.
A human-graded reference set. Two or more people grade clear passes, clear fails, and borderline cases. Every grader change must match it before release.
Overrides as data. Record a reason with every override. A cluster on one criterion shows where to work next.
Drift checks. Re-run the reference set after any change to the prompt, model, scenario or vocabulary. Investigate a sudden pass-rate change before attributing it to improved rep performance. Review scenario mix and participant changes as well as the grader.

Practices that help people act on scores

Separate practice from certification. On this platform, practice tiers show a running tally of criteria met as the rep speaks, and the more advanced practice tier adds hints on demand. The certification tier shows no hints and no live tally. After every attempt, the full criterion grid with reasons tops the report.
Tune live cues against normal speech. A live cue that fires on pauses can penalize considered replies. Tune its thresholds against normal speaking pace before release. We tuned one such cue down on this platform.
Set expectations first. We recommend a short brief on what the session measures. It helps first-time participants perform to their ability and read the result correctly.
Pilot with experienced people. We recommend having a small group of experienced people take a new certification before wider release. It surfaces wording and scoring problems while stakes are low.

Method and limitations

This paper draws on ALLTIPLY's recorded working sessions and meeting notes from building and operating one voice AI platform with two programs during production use. The over-scored pilot transcript is one documented case, not an average. This paper does not report statistical grader-to-human agreement. The speaking habits described are examples of what reading feedback across many scored sessions can surface, not measured findings from a controlled study. The recommendations are general practice for any AI grader, and this paper does not report on any one organization's adoption of them. For a broader treatment of evaluation evidence, see evidentiary evaluation for AI certification.

About ALLTIPLY Labs

ALLTIPLY Labs publishes what we learn building and operating AI systems. To scope a grading evaluation, request a project review with the rubric, representative transcripts and decisions the score will support.

Calibrating AI graders: what we learned and what the scores reveal