Evals Are the Product Discipline (and Writing Them First Backfires)

The intuitive move is to write your evaluators first and build to them, like TDD for prompts. It feels disciplined. It also backfires, and the reason is worth understanding, because it is the single most common way "software best practice" gets misapplied to AI products. This post is about what to do instead: build evaluators from failures you discover, not failures you imagine, keep the suite small and calibrated, and gate at three levels. It is the practical companion to my LangSmith evaluation guide, which covers the tooling; this one is about the discipline.

This is part three of the From Demo to Product series. Evals are why the earlier posts keep saying "measured pass rate" instead of "looks right."

Why writing evals first fails#

TDD and BDD work on deterministic software because the failure grammar is closed. A function has a finite set of branches, and "given X, when Y, then Z" can be enumerated by reading the code. An LLM has no such grammar. Its input is unrestricted natural language and its output distribution shifts with every prompt tweak, retrieved document, and model swap.

Hamel Husain and Shreya Shankar name this directly: "Eval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional software where failure modes are predictable, LLMs have infinite surface area for potential failures. You can't anticipate what will break."

Writing a full suite before anyone has used the system is like a doctor ordering every test in the hospital before the patient walks in. Expensive, unfocused, and it still will not catch the specific way this patient is sick. What works is a differential diagnosis: look at what is actually presenting, narrow down, test for that.

So the discipline is not specs first, code second. It is: ship a narrow slice, watch it fail for real, name the failures, build one evaluator per named failure, gate on that, repeat. The eval suite is always downstream of observed reality, never upstream of it. Which is also why you cannot do any of this without observability first; you cannot error-analyse traces you never captured.

Error analysis is the actual work#

Hamel and Shreya report spending 60 to 80% of total development time on error analysis and evaluation across the projects they have advised. Not a rounding error, the majority of the work. Their replacement for upfront specs is blunt: "Error analysis is the most important activity in evals. Error analysis helps you decide what evals to write in the first place."

In practice it means reading your own outputs. Run the system on real or realistic inputs, read something like a hundred traces, write a freeform note on the first failure in each, then cluster those notes into a named taxonomy: "hallucinates dates," "ignores the retrieved doc," "too verbose." That taxonomy is what tells you what to build.

And it is not a one-time kickoff. It is the recurring loop that keeps the suite honest. At proof-of-concept, you error-analyse a handful of runs against a hand-picked set. At pilot, real design-partner usage surfaces failures your golden set never imagined. In production, sampled live traffic surfaces drift: new failure categories appearing weeks after ship.

A good suite is smaller than you expect#

People imagine twenty metrics. A real suite is a cost hierarchy, cheapest first, and it is usually about two or three code checks plus one or two LLM judges per surface.

Code-based checks come first: is the format valid, are the required facts present, is forbidden content absent, was the right tool called, is latency and cost within budget. Cheap to build, free to run forever. The heuristic is simple: if a property is mechanically checkable, write code for it. Do not spend a model call and a maintenance burden on something a regex or a JSON-schema validator already solves.

LLM-as-judge is reserved for the genuinely fuzzy stuff code cannot touch: faithfulness, helpfulness, task success. It needs 100 or more labelled examples to build with confidence and ongoing maintenance as the task shifts. Most real failure taxonomies, once named, sort into "a rule would catch this" far more often than intuition predicts, which is why the suite stays small.

Calibration is not optional#

An LLM judge you have not validated against your own labels is theatre. It produces a number that looks rigorous and is not, and it is strictly worse than no eval, because it changes ship decisions on a number nobody checked.

Hamel and Shreya: "Focus on achieving high True Positive Rate and True Negative Rate with your judge on a held-out labeled test set... When you use multiple people, you'll need to measure their agreement using metrics like Cohen's Kappa, which accounts for agreement beyond chance." The mechanics of kappa live in my LangSmith post; the point here is the artifact you carry into a ship decision. It is not "the judge said 92%." It is "judge calibrated on N human labels, agreement X%, kappa Y, trustworthy." A ship gate built on an uncalibrated judge is a ship gate built on nothing.

Three levels of validation, none optional#

Level	Runs on	Catches
Offline	a versioned golden set, in CI, pre-ship	regressions you can enumerate, the quality gate
Human	real users on a design partner or closed beta	what numbers miss, a system that passes every metric and still feels wrong
Online	sampled live production traffic, reference-free	drift, quality degrading weeks later because the input mix changed

None substitutes for another. Offline-only teams ship things real users hate. Numbers-only teams miss the qualitative signal a five-minute usability session surfaces instantly. Ship-and-forget teams get blindsided by drift a month later. Every failure any of the three catches becomes a new dataset example, so the golden set grows instead of going stale. A golden set that never grows has stopped tracking reality.

"Good enough to ship" is not "perfect"#

A non-deterministic system that has to clear "perfect" before shipping never ships. The actual bar, for a narrow first slice:

Better than the current baseline, plus safe within the guardrail floor, plus cheap enough for the unit economics, plus a fallback for when it is wrong.

If all four hold on the narrow slice, ship it gated, and let human validation and online evals drive the rest of the roadmap. Widening scope and chasing edge cases before the first ship is the classic way a team never ships at all.

One more thing to carry into any ship review: track model metrics and product metrics side by side, and connect them. A four-point faithfulness gain means nothing to a stakeholder until it is paired with "and escalation rate dropped 15%." Model metrics justify why a change should work; product metrics prove it did.

The traps#

Generic prefab evals are worse than no evals. As one of Hamel and Shreya's TAs puts it, "in the best case they waste your time and in the worst case they create an illusion of confidence that is unjustified." A 100% pass rate is a red flag, not a milestone. And a small-n delta, "82% to 84%" on a handful of examples, is usually noise, not a win; check the statistics before you call it a ship-worthy improvement.

Where this goes next#

Evals tell you whether a thing works. They cannot tell you whether it should have been an LLM at all. That question comes earlier, and skipping it is expensive: the next post is does your problem even need an LLM. If you want the tools behind this post, the LangSmith guide is the hands-on version.