When Building Gets Cheap, Judgment Gets Expensive

Building software used to be the expensive part. You had an idea in an afternoon and spent the next quarter making it real, and that slow, costly quarter did you a favour you never noticed: it filtered out half-formed ideas before they shipped. That filter is gone. An agent can now turn a rough idea into a convincing, working feature faster than you can properly decide whether it was the right idea, which sounds like pure upside until you notice what it does to the cost of being wrong. When building is cheap, the penalty for building the wrong thing does not shrink. It grows, because you reach a polished version of the wrong thing before anyone has stopped to ask if it was wrong.

So the bottleneck moved. The scarce skill stopped being implementation and became judgment: deciding what to build, whether it is working, and when to stop. Put more bluntly, when the cost of producing code collapses, engineering shifts upward, from producing code to managing uncertainty.

This is written for the engineers and product leads shipping AI features who can already build faster than they can decide what to build. By the end you will have eight concrete moves for shaping a build before you write it, a worked example that runs all eight on one feature, and a way to tell mid-project whether your next fix belongs in the prompt or somewhere else entirely. It is the front half of the story I told in From Demo to Product: that series was about building, shipping, and operating an AI product. This is about shaping it first.

The spine: eight moves that reduce uncertainty in order#

The point of what follows is not to produce eight documents. It is to reduce uncertainty step by step, where each move produces a concrete piece of evidence the next one consumes:

outcome ─▶ workflow evidence ─▶ assumptions ─▶ slice ─▶ spec ─▶ metrics ─▶ stage ─▶ feedback
   ▲                                                                                   │
   └───────────────── a new product need loops back into the next outcome ──────────┘

They are one pipeline, and each move hands the next a piece of evidence to work from. Nothing here is a document that sits in a drawer. Every step is a decision with evidence attached, and the loop at the end is the whole point: shaping does not stop when coding starts, it runs continuously through delivery.

I will use one running example the whole way, because a concrete case beats abstraction: an assistant that helps an on-call engineer figure out which service is failing when a production alert fires. Keep that in your head.

The most useful debugging instinct I have on an AI product: when it keeps getting things wrong, check whether the evidence is pointing at the model or at what you fed it. It is usually what you fed it, and no amount of prompt-tuning fixes a thin input.

What AI actually changes here#

Strip the words "AI" and "agent" out of the eight moves and most of the discipline still holds for any software. That is a feature, not a bug: this is requirements thinking, and requirements thinking was always right. But AI changes the weighting, and it does so in ways that punish teams who skip the shaping:

• Behaviour is probabilistic, so "it worked once" proves almost nothing, and evaluation replaces some of what tests used to do.
• Context quality becomes a product dependency, so a stale source or a missing field is a product failure, not a data-team footnote.
• Model choice changes both capability and failure modes, and a model upgrade can silently change behaviour after you shipped.
• Authority boundaries become load-bearing, because the thing deciding what to do is non-deterministic and occasionally wrong with total confidence.

That last cluster is why the moves about assumptions, specs, and staging carry more weight for an AI build than they would for a CRUD app. Hold that thought; it recurs.

Move 1: Define the outcome before you name the output#

It helps to separate three layers that teams collapse into one. The problem is that on-call engineers spend too long working out which service caused an alert. The outcome is that the engineer identifies the likely failing service and a safe next action within a couple of minutes. The solution is an AI incident assistant. Most teams skip straight from problem to solution and never write the outcome down, which is exactly how you end up building the wrong thing quickly.

"Build an incident assistant" is an output. It names an artifact and smuggles in a solution nobody has earned yet. An outcome says the result instead:

When a production alert fires, the on-call engineer identifies the failing service and a safe next action within a couple of minutes, and the whole diagnosis stays read-only and auditable.

That sentence can be satisfied by software, a runbook, a data fix, or a smaller change to an existing screen. It keeps the team attached to the result rather than the first artifact someone pictured. A frame like that has six parts:

The failure mode to watch for is solution leakage: an outcome that already contains a product form, an interface, a model choice, or an architecture nobody has earned. "Users get a weekly AI summary" leaks both the summary and the cadence. "Deploy a vector database" leaks infrastructure. AI teams leak solutions in a very particular way: we anchor on the fashionable piece of the stack, RAG or fine-tuning or a vector store, before checking whether plain keyword search or a deterministic rule would have solved the user's problem outright. State the capability ("relevant policy evidence is available during review") and let the evidence pick the machine.

Constraints and non-goals are part of the goal, not garnish. A build that hits the outcome by writing to production during diagnosis has not hit the outcome, because read-only was the point. And a non-goal like "no automatic remediation in this slice" is what stops a useful small thing from swelling into a platform before it has proven anything.

Move 2: Discover the workflow people actually perform#

Requirements are not sitting in a meeting waiting to be collected. They are scattered across what people actually do: the workarounds, the copy-paste, the side-channel Slack message, the step everyone stopped noticing years ago. So you do not start by asking what features people want. You reconstruct what happens now, step by step, and you attach evidence to each step.

The move that keeps this honest is grading your evidence, because not all "we know this" is equal:

All four are useful. Only the top two prove what happens today. Label the bottom two so confidence does not inflate from "someone said" to "it is known." Then go hunting for the four things that break AI features: friction (repeated effort, delay, re-entry), hidden state (facts living in someone's head, a chat thread, a personal note), authority (who or what is actually allowed to make the consequential change), and exceptions (where the normal path stops being normal).

Here is the part I would underline from experience. The happy path is almost never where an AI feature gets hard. In the agent systems I have built, the trouble starts at the missing context field, the stale source, the ambiguous permission, and the exception nobody thought worth writing down. Those are all hidden state, authority, and exceptions, and they matter more for an LLM than for deterministic software, because a deterministic system fails loudly at the gap while a model cheerfully hallucinates straight past it.

One warning that took me a while to respect: do not average away disagreement. If two experienced people do the same job differently, that is data, not noise. They might be different roles, different risk levels, an old process and a new one, or a real policy fight nobody resolved. An averaged workflow can end up describing nobody who exists.

Move 3: Map the assumptions and resolve the riskiest one first#

A roadmap hides uncertainty inside features. An assumption map drags it out. Every build is a stack of bets that must be true for it to be worth anything, and you write each one as something that could be proven false. "The feature is useful" cannot be tested. "Eight of ten on-call engineers identify the correct service faster with the read-only result" can. Spread the bets across five classes so you notice the one you skipped:

For an AI build, three assumptions hide inside Feasibility, Viability, and Safety and get skipped the most. Latency: can the model answer fast enough to matter, given that a slow chain-of-thought during a live severity-one incident is worse than useless? Consistency: does the output hold across the real variation in incident telemetry, or only on the three examples you tried? Cost: does the per-task economics survive real volume, or is every diagnosis quietly unprofitable? None of those show up if you only ask "can it work at all."

Then rank the assumptions on more than one axis, because risk is not a single number. Score impact (damage if it is false), uncertainty (how weak your evidence is), and irreversibility (how expensive it is to learn the truth after committing). The point is not the arithmetic, it is being forced to say why one unknown deserves resolving before another. For the incident assistant it falls out like this:

The shape of the build should follow the shape of the uncertainty. A read-only replay comes before a production integration; a human-approved recommendation comes before automatic action. High-consequence, hard-to-undo choices need their evidence earliest.

Move 4: Choose the smallest slice that can change the decision#

"Ship an MVP" is where good intentions go to die, because "minimum" gets optimised into "smallest thing we can demo" rather than "smallest thing that proves something." The reframe that fixes it: a slice is defined by the assumptions it proves, not by how little code it takes. Take your highest-risk open assumptions, turn them into a required proof set, and a candidate slice is eligible only if it covers that set. Then, and only then, compare eligible slices on value, uncertainty reduced, effort, consequence, and reversibility.

To be clear about a distinction that is easy to blur: a slice is a scope and hypothesis boundary (which assumptions am I testing, and how narrowly), while the prototype/pilot/production choice in Move 7 is the deployment and containment environment you run that slice in. A single slice usually maps to a prototype or a bounded pilot, never straight to production.

The eligibility gate is the whole game, because it rejects the seductive false minimums:

For the incident assistant, a read-only replay across ten real past incidents tests the two things that actually matter, whether it identifies the right service and whether operators trust it, at almost zero consequence. A polished dashboard on synthetic data would test comprehension but dodge the data-feasibility question. A production auto-remediator would test everything at once with a consequence you cannot take back.

Then, before you write any of it, write the stop rule: what happens if the slice fails? Abandon the outcome, change the target user, try a different mechanism, gather better evidence. If every possible result leads to "keep building," you did not design an experiment. You designed a plan wearing an experiment's clothes.

Move 5: Write specifications that preserve judgment#

There are two ways to hand work to a coding agent badly. Underspecify, and it guesses at a system it cannot see. Overspecify, and you force it to transcribe a design that may already be wrong, throwing away the judgment that was the whole reason to use a capable model. The useful middle is a spec that reads like a contract, not a screenplay: it fixes what must be true and leaves the reversible choices open.

The move that makes this real is tagging every decision with how much freedom it gets. Lock consequence, delegate detail:

For the incident assistant, that is not abstract. It looks like this in the actual instruction you hand the build:

LOCKED
- read-only: GET requests to telemetry APIs only, no writes during diagnosis
- no new external dependencies
- must preserve the existing alert-payload contract
 
BOUNDED
- at most 3 tool calls per diagnostic run
- hard timeout at 15 seconds (latency is a product requirement here)
- data sources limited to the incident log and deployment record
 
DELEGATED
- how the diagnostic summary is formatted for the engineer
- internal module structure and function names
- retry and backoff strategy within the bound above

That single distinction lets you keep a human checkpoint exactly where consequence lives and nowhere else. Two supporting habits carry weight. Specify behaviour through examples, because "helpful" and "production-ready" are not executable but a handful of normal, edge, failure, and forbidden cases are. And match your proof to your claim: a unit test proves a local function, a replay set proves behaviour over representative cases, an audit log proves an authority boundary held. Do not accept a lower layer as evidence for a higher-layer claim.

Move 6: Design the success metrics before the result exists#

If you decide what "good" looks like after you see the results, you will move the goalposts to protect what you built. Everyone does; it is gravity, not a character flaw. So derive the metrics from the goal up front, and treat each metric as a contract, not a number: a name, a direction, a threshold, a window, a source, a population, and whether it is an outcome or a guardrail. Without a source and a window, nobody can reproduce the number. Without a threshold, it cannot drive a decision.

A single outcome metric on its own is a trap, and it is a worse trap for AI than for anything else, because a model will happily optimise into confident nonsense. So pair every outcome metric with a guardrail and a counter-metric. The AI-native pairs are the ones to reach for: correctness against latency (a fast wrong answer is not a win), and recall against hallucination rate (flagging every possible failing service "just in case" is not diagnosis, it is noise). For the incident assistant: speed is the outcome, but "no production writes," correct-service rate, and operator workload are the guardrails and counter-metrics that stop a fast, unsafe, noisy result from looking like success.

Then write the pass, fail, and ambiguous decisions before you generate a single value: pass if the correct-service rate clears a high bar and the time clears its budget; fail on any production write or a correctness rate below a floor; ambiguous if the gain is small with wide variance, which means go get a bigger sample.

Move 7: Choose prototype, pilot, or production on purpose#

These three are not levels of polish. They are different learning environments, each answering a different question:

A prototype can be technically finished and still be disposable; keep it discardable, isolated, and honest about the one thing it is trying to learn. A pilot can run on real production data while staying tightly limited in audience and authority. Production is not a deploy button; it starts the moment the organisation accepts continuous responsibility.

Treat the moves between stages as explicit gates, not vibes. Prototype to pilot needs evidence that the mechanism works, the safety boundary holds, and the outcome is measurable. Pilot to production needs a named owner, a reliability target, a cost envelope, a security review, rollback, and monitoring. The danger to design against is stage drift: a prototype that picks up users, real data, and authority without ever picking up ownership. A warning banner does not stop that. Make the stage observable from the system itself, in access control and telemetry, so a prototype physically cannot reach production data by accident.

Move 8: Build the feedback ratchet#

A feedback ratchet is a loop where recurring failures become stronger system constraints instead of recurring human interventions, so the system can only get harder to break, never looser. Shipping closes one loop and opens this one. A team can collect traces, evals, and incident logs forever and learn nothing, because the missing piece is promotion: a defined path from a raw signal to a durable, owned change with proof attached.

The mechanism is to route each signal to the earliest layer that owns the cause:

The reflex I have to fight in myself is fixing everything with a prompt tweak. If a permission boundary or a test can make a failure impossible, that beats one more sentence of instruction the model may or may not honour. Every ratchet action needs an owner, a priority set by how bad and how frequent the failure is, the artifact to change, the verification that proves it worked, and a retirement condition. Controls accumulate into contradictory, expensive policy if nothing ever removes them, so retire a control when the architecture changes, when a lower-level invariant replaces it, or when it blocks more legitimate work than harm. Retirement needs evidence too. Do not delete a rule just because it feels old.

The whole thing, on one incident#

Frameworks are easy to nod along to and hard to feel, so here is the incident assistant run through all eight moves as one short arc, with an ending that is the entire point.

The outcome was to identify the likely failing service and a safe next action within a couple of minutes, read-only. The two riskiest assumptions were that the alert payload carried enough to identify the service, and that engineers would trust a recommendation they did not derive themselves. So the slice was a read-only replay over ten past incidents, no remediation tools, sources limited to the incident log and the deployment record. The spec locked read-only, bounded the run to three tool calls and fifteen seconds, and delegated the formatting. The success rule, written before any results existed, was eight of ten correct within the time budget, with any production write counting as an outright fail.

It came back six of ten. The misses were not the model being dim. In four of the cases the alert payload simply did not carry enough to identify the service at all. The fast, cheap move would have been to keep editing the prompt until the demo looked better. The right move was to see that the evidence was pointing at the input, not the model, and to fix the layer that actually owned the failure: route the signal to the context layer and enrich the alert. Judgment, not implementation. That single decision is the whole post in miniature.

Gotchas, or what actually bites teams#

• Solution leakage. If your "outcome" names an interface, a model, or an architecture, you locked in the first idea instead of the right one.
• Confidence inflation. "Someone told me" hardens into "it is known" unless you label reported behaviour and inference as what they are.
• The averaged workflow. Two people doing the job differently is a finding, not noise to smooth over.
• The false minimum. A slice that drops the risky part of the workflow is not small, it is incomplete.
• The spec that kills judgment. Locking every decision turns a capable agent into a transcriptionist of a design that may already be wrong.
• The moved goalpost. Metrics decided after the results exist will bend to protect the build, so write pass, fail, and ambiguous first.
• Stage drift. A prototype that gains users, data, and authority without gaining ownership is the most common way a quick experiment becomes an unowned production system.
• Telemetry with no owner. Feedback with no promotion path and no owner is just better-formatted logging.

Where this goes next#

This is the judgment layer that comes before the build. Once you have shaped the thing, an outcome you can defend, a slice that proves the risky part, a spec that keeps judgment where it belongs, and metrics you committed to in advance, you are ready for the part I have already written about: From Demo to Product, on building, shipping, and operating it without the demo lying to you. Two threads connect the halves. The feedback ratchet here is the same loop as the data flywheel, and the read-only, human-approved, earn-it-gradually instinct running through these eight moves is the same one behind shipping on containment.

A note on where this comes from: I rebuilt this arc, in my own words and with my own AI-engineering framing, from the Product Judgment and Delivery path of the AI Engineering from Scratch course. Naming that feels more useful than pretending I invented the pipeline. The worked example, the AI-specific angle, and the opinions are mine.