Who Owns Quality: The Benevolent Dictator and AI Incident Response

The hard part of an AI product is not the code. It is the judgment about what "good" means, and that judgment has to have a single owner, not a committee. A normal software team organises around features. An AI product team has to organise around the eval loop, because the thing that decides whether the product is good is not "did the code run" but "is the output actually right, useful, and safe," and that is a judgment call someone has to own and keep making, over and over, as the product changes. This post is who that someone is, how the rest of the team works around them, why annotation tooling is the highest-leverage thing you can build for them, and why an AI incident should end in a new eval example.

This is part ten of the From Demo to Product series, and the close of its main arc. The whole series has been building to one claim: the product is the loop, not the model, and the loop needs people. Two architecture deep-dives follow this post as parts eleven and twelve.

Organise around the eval loop, not the model#

If nobody owns "what good means," you get the classic failure, and it is remarkably consistent across teams. Engineers optimise a metric that does not quite match what users need. The eval set drifts from reality because nobody is tending it. And everyone points at a green dashboard while users quietly churn, because the dashboard measures the wrong thing well. The dashboard is not lying; it is answering a question nobody checked was the right question.

So the org question is not "who writes the agent." It is "whose judgment defines quality, and how do we capture it cheaply and repeatedly?" Everything below flows from that one reframe.

The roles: who catches what#

Three roles, and the value of naming them this way is that each one catches a different class of failure that the others are blind to:

The single-owner model is the part people resist, so it is worth stating plainly: appointing one domain expert as a benevolent dictator who owns the eval criteria is the most effective approach for most small-to-medium teams. A psychologist for a mental-health tool, a lawyer for legal analysis, a support leader for support automation. Engineers catch the technical issues and PMs catch the product ones, but domain expertise is the scarce input, and the best way to empower the domain expert is to give them custom annotation tools.

Why one arbiter and not a committee: quality on a fuzzy task is a matter of judgment, and judgment-by-consensus produces mush, a rubric everyone half-agrees with and nobody can apply consistently. One owner keeps the bar coherent. Engineers and PMs inform it; they do not vote on it.

Annotation is the product improvement mechanism, not overhead#

Error analysis (the discipline from the evals post) is not a solo task done once at kickoff. It is the recurring ritual that keeps the eval suite honest, and it needs a real workflow, not good intentions.

• Give the domain expert a custom annotation tool, not a spreadsheet of raw JSON. A purpose-built view that shows the input, the output, the retrieved context, and offers one-click labels plus a free-text note. The cheaper it is to capture a judgment, the more judgments you get, and error analysis is bottlenecked entirely on the expert's time. This is the single highest-leverage tooling investment for eval quality, and making the expert read JSON blobs in a spreadsheet throttles exactly the input you most need.
• At small scale, the benevolent-dictator model is the annotation model. One expert labels, and their call is final. You do not need agreement machinery for a team of one labeler.
• At larger scale, with multiple annotators across domains, measure agreement. Do not assume two people label to the same mental model. Measure inter-annotator agreement, discuss the disagreements, and iterate the rubric until consistency improves. And read a disagreement correctly: two experts diverging usually means the rubric is ambiguous, not that one of them is wrong. Fix the rubric rather than averaging the noise.
• The output loops back. Every labelled failure becomes a golden-set row and, once patterns emerge, an evaluator, feeding the data flywheel. Annotation is not overhead you tolerate; it is the improvement mechanism itself.

The rituals that keep it alive#

An AI product decays without recurring practices, not just artifacts sitting in a repo. Four are load-bearing:

• The error-analysis review, roughly weekly: the domain expert and an engineer read a fresh sample of production traces together, name new failure modes, and decide which become evaluators. This is where drift gets caught before a metric alarm fires, because a human notices the new kind of wrong before a threshold does.
• The ship-review artifact. Every quality change is presented as "here is the eval delta and the product metric it moved." Model metrics justify why a change should work; product metrics prove it did. "It felt better in testing" is not a ship review.
• The eval-set refresh cadence. Scheduled harvesting of new examples from production so the golden set tracks reality instead of yesterday's problems.
• Owned design-partner contact. Someone owns the relationship with real users, because that is the source of the messy real inputs a synthetic set never contains.

A hallucination can be a Sev#

Standard on-call assumes deterministic failures: a service is up or down. AI products add a class of incident that classic monitoring misses entirely, where the system is up, fast, and confidently wrong. Treat these as first-class incidents with their own severity ladder and runbook. The types worth a severity, beyond normal outages:

The runbook adapts normal incident practice to a non-deterministic core:

DETECT      an online-eval / cost / latency / intervention-rate alert, or a user report
   ▼
CONTAIN     one-switch ROLLBACK at the gateway, or kill the feature flag (rehearsed)
   ▼
COMMUNICATE status to stakeholders and users; honesty beats a silent confident-wrong system
   ▼
DIAGNOSE    pull the trace; the run, step, and tool spans localize it
   ▼
FIX + GATE  fix, prove it clears the offline eval gate, re-ship via staged rollout
   ▼
POSTMORTEM  the failing input becomes a NEW golden-set row, and if it is a pattern, a new evaluator

The distinctive move is the last step. An AI postmortem's deliverable is an eval-set example, not just an action item. That is what makes the incident loop and the data flywheel the same loop: every incident permanently hardens the eval suite, so that exact failure can never silently regress again.

Gotchas, or what actually bites teams#

• Quality-by-committee produces an incoherent bar. Three stakeholders each with a veto and no single owner yields a rubric nobody can apply consistently. Appoint the benevolent dictator.
• Engineers owning the eval criteria. Engineers catch technical failures brilliantly and domain failures poorly. A retrieval fix that passes every code check can still return an answer a real practitioner knows is subtly wrong. The domain expert owns "is this good," not the engineer.
• Annotation on raw data. Making the domain expert read JSON blobs throttles your highest-leverage input. Build the annotation tool early; it pays for itself in eval quality.
• Multiple annotators without measuring agreement. Two experts labeling to different mental models produce a noisy golden set that looks fine on the surface. Measure agreement and fix the rubric; do not average the noise.
• Treating a hallucination as "just a bad output." Without a severity ladder and a rehearsed rollback, a viral confident-wrong answer becomes an all-hands scramble instead of a contained incident with a runbook.
• Postmortems that produce action items but no eval example. If the failing input does not enter the golden set, the fix did not stick, and the failure regresses on the next change.

Closing the arc#

That is the main arc of "From Demo to Product," end to end. The demo is 5% of the work. The model is not the moat. Evals are the product discipline, and not every problem needs an LLM. The model is a flaky dependency you wrap, not trust. You ship on containment, you operate the loop and build the flywheel, you model the unit economics, and you detect, contain, and monitor the risks. And you put one owner on quality, with a runbook that turns every incident into a test that makes the product permanently harder to break. Do all of that, and you have crossed the gap from a demo to a product a business can actually depend on.

From here the series turns to the architecture underneath all of it: the reference architecture for a production AI app draws the whole system on one page, and designing UX for a confidently-wrong intern designs the box that faces the user.