Decide what to do with a stalled AI pilot

The pilot has a sponsor and a working interface, yet users still correct answers or return to the old process. Another round of tuning would spend budget without a clear reason to expect improvement. Answers fail for different reasons: the right source was never found, the source was out of date, the task was misunderstood or a system action never completed. A better model cannot fix all of these.

We build a diagnostic that finds where the pilot actually breaks and what to do next.

• We freeze a baseline configuration and replay representative tasks against it, using approved reference answers, current prompts, retrieval settings and the integration map.
• Each failure is separated into source availability, retrieval, answer quality or integration behavior, then mapped to the component responsible.
• We test one bounded change against the same task panel, so the comparison is fair.
• If logs or source versions are missing, the cause is marked unknown and reproduced in a separate test environment. A regression restores the baseline. An inconclusive result leads to a narrower investigation, not a success claim.
• The diagnostic works from read-only exports under the original access rules. We also inspect where traces, source indexes and candidate models run. A private deployment alone does not repair task quality.

• A failure map. Each failure is linked to the representative task where it appeared and to the component that caused it.
• An approach comparison. Current and candidate approaches are tested against the same task panel, so the difference is fair.
• A next-scope recommendation. Repair, rebuild or stop, with operating dependencies and unresolved assumptions stated plainly.

• The process owner approves what counts as a successful task.
• A subject expert settles disputed answers, and the sponsor accepts the next scope.
• The diagnostic system cannot change production settings.

If the process is stable and an existing product passes the required tasks and controls, replacing the pilot may make more sense than a custom rebuild.

• Rebuild or tune? A decision framework for stalled AI systems
• The production gap: lessons from seven AI workstream records