Detect, Contain, Monitor: The Risks That Sink AI Products

A traditional bug has a fix: find the bad branch, patch it, close the ticket. An LLM risk is not a branch, it is a distribution. The model will hallucinate some fraction of the time, obey some fraction of injected instructions, and drift as the world shifts, and no amount of patching drives any of those to zero. So the senior move is to stop trying to eliminate the failure rate and instead bound the blast radius and watch the rate. Airbags, not a promise the car will never crash. This post is a tour of the risks that actually sink AI products, and the single idea that runs through all of them: every mitigation is a detect, contain, and monitor control, never a cure.

This is part nine of the From Demo to Product series. It pairs with treat the model as a flaky dependency, which goes deeper on the vendor-risk slice.

The mental model: detect, contain, monitor, never "solve"#

Because an LLM risk is a distribution rather than a bug, "we fixed it" is the wrong frame. The mitigations below do three things instead: they detect the failure when it happens, they contain the blast radius so a failure is survivable, and they monitor the rate so you know when it is getting worse. Read each of the eight risks that follow through that lens.

1. Hallucination and wrong output#

The defining reliability risk, and it comes in four distinct flavours worth telling apart because they need different catches:

Flavour	What it is
Factual	Invents a fact that is simply not true
Grounding	Ignores the retrieved document and answers from memory anyway
Citation	Fabricates a source or attaches a real source that does not support the claim
Reasoning	Produces a valid-looking chain of reasoning that is wrong

Hallucination rates have fallen substantially from a few years ago, but they are not zero, and they never will be, because the mechanism (generating from a lossy compression of training data) produces invention as a side effect. So you design for it: ground answers in retrieval with citations, run faithfulness evals offline and online, and set a confidence threshold that routes low-confidence cases to "I'm not sure" plus a human handoff. Never emit a confident wrong answer with no escape hatch. And match the autonomy to the cost of being wrong, a point that recurs everywhere in this series: a wrong summary is cheap, a wrong money transfer is not.

2. Prompt injection and security#

Direct and indirect injection, jailbreaks, tool misuse, and data exfiltration. There is no complete fix here, and it is important to be honest about why: an LLM cannot reliably separate instructions from data, so any text it reads is a potential instruction. Detection helps but cannot be the load-bearing layer, because the thing you are detecting is adversarial and adaptive.

The load-bearing defence is least privilege. If the agent cannot do the dangerous thing, the injection asking it to is inert. That is the one control that does not depend on out-guessing an attacker. Layer input and output guardrails, sandboxed execution, and human approval on high-impact actions on top, and treat all retrieved and tool content as untrusted by default.

3. Reliability and non-determinism#

Same input, different output; and the provider itself has outages, latency spikes, and rate limits. Treat the model as a flaky, expensive external dependency and wrap it in the rigor you would give any third party you do not control: retries with backoff, timeouts, circuit breakers, deterministic fallbacks, idempotency, and graceful degradation, all centralised at the gateway. And prefer the simplest orchestration that does the job, since a plain workflow is more reliable than a RAG pipeline, which is more reliable than a free-roaming agent. Complexity is itself a reliability risk.

4. Cost blowups#

Per-request cost of goods plus agent loops means a stuck loop or a viral spike can multiply the bill dramatically overnight. Mitigate with per-run step, token, and dollar budgets, routing and caching, rate limits at the gateway, and cost alerts on daily spend that page you like an outage, because a runaway cost is an outage of a different kind. The full treatment is the unit economics post.

5. Privacy, governance, and compliance#

PII and secrets leaking into prompts, logs, traces, or memory; regulated data; residency rules. And regulation now has real teeth. Major AI regulation applies in phases rather than all at once, with some obligations already in force and others deferred to later dates, and those dates have moved more than once. Because of that, I am not going to pin a specific deadline in this post: re-verify the current in-force status before you plan around any of it.

What does not change is the design advice, regardless of the exact date. Redact PII before storage and sending, minimise what you keep and set retention limits, isolate per tenant, use data-residency-aware providers, and assess whether your use case falls into a high-risk category early. The obligations that come with high-risk classification (a risk-management system, documentation, human oversight, logging) are far cheaper to design in from the start than to retrofit under a deadline, and any extra runway a deferral gives you is time to build them in, not permission to ignore them.

6. Model deprecation and vendor lock-in#

Providers deprecate models and change prices on their schedule, not yours. A model change is a when, not an if. The mitigations are structural: route all calls through a provider-neutral gateway, keep an eval harness so you can re-qualify a replacement model in hours rather than weeks, treat deprecation as routine roadmap vendor-risk rather than an emergency, and consider keeping a fallback provider warm. This risk has its own full post in the series, because it is the one builders most consistently underweight.

7. Quality drift after launch#

Quality degrades weeks after launch with no code change, as input distributions shift, providers silently update models, and data goes stale. You run online evals and drift monitoring on live traffic, refresh the eval set on a cadence, and pin model versions where you can and re-qualify before adopting a new one. This is covered end to end in LLMOps and the data flywheel.

8. Product, market, and trust risks#

The thin-wrapper risk that a better base model or a lab feature eats your product; over-automation that erodes user trust; a single public failure that becomes a reputational event. You build a data, integration, and system-of-record moat; you earn autonomy gradually rather than switching it on; and you design for graceful, honest failure, starting supervised in low-blast-radius surfaces.

The risk register#

The through-line is that none of these is fixed once. So the artifact is not a one-time audit, it is a living register you keep in the repo and review at every launch:

Risk	Likelihood	Impact	Mitigation (where it lives)
Hallucinated answer reaches a user	High	High	Citations + faithfulness eval + confidence-to-handoff
Prompt injection via a retrieved doc	Med	High	Input guard + least privilege + sandbox
Runaway agent cost	Med	Med	Step and dollar budgets + cost alert + usage caps
PII leak into logs, memory, or traces	Med	High	Redaction + retention limits + tenant isolation
High-risk regulatory obligation missed	Low-Med	High	Classify high-risk early; build oversight, logging, docs in
Primary model deprecated or repriced	High	Med	Gateway + provider-neutral + re-qualification evals
Post-launch quality drift	High	Med	Online evals + drift alerts + eval-set refresh
Thin-wrapper, commoditised	Med	High	Data, integration, and system-of-record moat

Fill in an owner and a status for each row, and revisit it every launch, because likelihoods and impacts move as the product and the vendors change.

Gotchas, or what actually bites teams#

• The fail-open guardrail. A guardrail that errors and gets treated as "allow" disappears exactly when input is most adversarial. Fail closed on high-stakes paths.
• Silent model updates you do not control. A "same model name" can behave differently after a provider update. Pin versions where the provider allows it and re-qualify before adopting.
• Stale regulatory claims. Compliance deadlines move; a confident date from a year-old blog post may be wrong. Re-verify against the primary source before you plan around it.
• Margin risk hiding in an average. A per-request cost that looks fine in aggregate can be margin-negative for a power user, so carry a worst-case-power-user row in the register alongside the technical risks.
• Treating deprecation dates as fixed. Specific retirement dates are accurate only as of when you read them. Re-check the provider's own calendar before planning around any of them.

Where this goes next#

Every risk here has an owner, and the last post in the series is about the people: who owns quality, the benevolent dictator and AI incident response, including the incident runbook whose postmortem output is a new eval example. The vendor-risk slice of this post is treat the model as a flaky dependency.