Define the evidence that makes an AI release acceptable

A pilot appears useful, but executives and delivery teams disagree about readiness. A broad benchmark score does not show whether the system handles the organization's consequential tasks. Average quality can hide rare severe errors. Reference answers can be wrong, reviewers can disagree and an automated judge can reward fluent but unsupported responses.

We build a release evaluation tied to your own tasks and the consequences of getting them wrong.

• The task panel is segmented by business consequence and failure type, including difficult and prohibited cases with domain-owner reference answers and severity labels.
• References and scoring rules are frozen before candidates are compared.
• Deterministic checks cover fields and permissions, expert review covers meaning, and calibrated model judging is used only where appropriate. Disagreements are reviewed before results are published.
• An invalid reference is corrected with a dated rationale, and affected comparisons are rerun. A severe failure blocks the affected release scope even if the average improves. A new source, model or tool permission triggers the relevant regression panel.
• Reference sets are restricted and traces are sanitized. We decide where protected evaluation data and judge-model calls run, and private execution still needs valid references.

• A versioned evaluation panel and rubric. Tasks segmented by consequence and failure type, with frozen references and scoring rules.
• Category-level results. Results and a failure register by category, so rare severe errors are not hidden by an average.
• A release recommendation. The release decision with exclusions, limitations and the changes that trigger a retest.

• Domain owners approve expected outcomes and critical failure definitions.
• Reviewers judge uncertain cases without knowing which candidate produced them, where practical.
• The authorized release owner makes the final decision.

For a low-risk deterministic workflow, ordinary software checks may cover the requirement. Custom evaluation work fits systems whose generated judgments or actions affect consequential decisions.

• Calibrating AI graders: what we learned and what the scores reveal