The Reference Architecture for a Production AI App

A single call to a chat model is about a day of work. Everything that makes that call survive contact with real users is the other ninety percent, and it is the same ninety percent no matter which model you call. Surviving a bad prompt, a provider outage, a pricing change, a user trying to talk the thing into leaking a secret: none of that lives in the model call. It lives in the boxes around it. This post is the map of those boxes, how they bolt together, and the one rule that decides how complicated the middle of the diagram gets.

This is part eleven of the From Demo to Product series. The earlier posts kept pointing at pieces of this architecture: the gateway, the fallback, the cost levers. This one draws the whole thing on one page.

The mental model: build the boxes in the order they fail#

The temptation is to build the boxes in the order they are fun to build: the clever agent loop first, the boring plumbing never. Reverse that. Build them in the order they fail in production. Nobody's demo dies from a missing gateway or a missing fallback. Every real incident report has one of those holes in it. So the mental model for the whole diagram is: the model call is the easy part, and the unglamorous checkpoints around it are what turn a demo into something a business can lean on.

The reference architecture#

Here is the shape almost every production AI app converges on. Roughly ten boxes, one of which is the model:

 client ─▶ app / API ─▶ [input guard] ─▶ orchestration ─▶ [AI GATEWAY] ─▶ model provider(s)
                │                        (workflow /          │   routing · cache · retries ·
                │                         RAG / agent)         │   rate-limit · fallback · log
                │                             │                │
                │                      retrieval (vector + lexical)
                │                      tools / actions [approval]
                │                      memory (short / long)
                │                             │                │
                ▼                             ▼                ▼
         [output guard] ◀──────────── response + citations   observability (tracing / evals / cost)
                │                                                     │
                └── user ◀── deterministic fallback on error/low-conf ┘

The useful thing about this map is that almost every box is a solved problem with an obvious owner. You do not invent a vector database or a tracing backend, you adopt one. What is left for you to actually design is a much shorter list:

Two boxes in that list are the ones teams consistently under-build: the AI gateway and the deterministic fallback. They get skipped because they have no obvious home. "Add a guardrail" and "add tracing" are muscle memory now. "Add the box that decides what happens when the model is slow, wrong, or down" is not, so it waits until an outage or a price hike makes it mandatory. Build both from the start.

The one rule: workflow, then RAG, then agent#

The single biggest architecture decision in that diagram is the orchestration box, because it decides how predictable, how cheap, and how debuggable the whole system is. There are three patterns, and they are not equal:

The rule is to prefer them in that order: workflow first, RAG when you need grounding, agent only when the task truly cannot be scripted. Choosing the boring pattern is a senior move, not a timid one. Most things pitched as "AI agents" are workflows wearing an agent's marketing.

This is not just my house style. It is the same order two independent practitioner sources land on: the major model labs' own guidance on building agents says to start with simple prompts, evaluate them properly, and add multi-step agentic complexity only when simpler approaches demonstrably fall short, because agents trade latency and cost for flexibility. Platform-architecture write-ups from practitioners reach the same order from a different direction: context, guardrails, gateway, cache, and only then agentic patterns. When two sources with different priors converge on the same ordering, that is a rule worth trusting.

Most of what people reach an agent for is covered by a handful of workflow patterns that keep your code in charge of the control flow: chaining prompts in sequence, routing an input to the right specialised path, running calls in parallel and aggregating, an orchestrator that fans work out to workers, and an evaluator that critiques and improves a draft. All of those are debuggable because your code owns the steps. RAG and agent are not mutually exclusive either: "agentic RAG," where retrieval itself makes dynamic decisions, exists for when a fixed retrieve-then-generate pipeline fails your eval. The rule just applies one layer down: make the retrieval step dynamic only when the simple version demonstrably breaks.

Model choice: your eval beats the leaderboard#

Two habits keep model choice sane. First, choose on your own eval, not a public leaderboard. A leaderboard measures a benchmark distribution; your product has a different task distribution, and the only number that matters is your own harness on your own golden set, scored on task success against latency against cost. This is the eval discipline applied to a procurement decision.

Second, start strong and optimise down. Prove the feature is even feasible on a frontier-tier model, then push work down to a cheaper, faster tier only where your eval shows quality holds. Think in tiers, not a single pick: a cheap fast tier for high-volume, low-difficulty calls, a production-default tier where most work should land, and a frontier tier reserved for the hardest slice. That is a routing decision, covered in treating the model as a flaky dependency, not a one-time choice.

And stay provider-neutral from day one. Never hard-code one vendor's SDK into your business logic, because model deprecation and price changes are a when, not an if. Providers retire models on a published calendar and change prices on their own schedule, so a model name buried in application code is a migration waiting to fire. Keep it behind a swappable adapter instead.

Buy the commodity, build the differentiated core#

Not every box is worth your engineering time. The default for each layer:

The principle underneath the table: every hour spent on undifferentiated plumbing, a retry wrapper or a token counter, is an hour not spent on the evals and workflow logic that are actually your moat. Buy the commodity so you can afford to build the core.

The two boxes teams skip#

Both under-built boxes exist for the same reason: the model is a dependency that will occasionally be slow, wrong, expensive, or simply unavailable, and "what happens then" is a design decision, not an afterthought.

The AI gateway is a single internal choke point every model call routes through. Put routing, caching, retries, rate limits, key management, per-request logging, cross-provider fallback, and one-switch rollback in that one place instead of reimplementing a slice of each in every service. Its real payoff is strategic: it turns "the model changed" into a one-place config change instead of a codebase-wide fire drill. The full argument is in treat the model as a flaky, expensive dependency.

The deterministic fallback is the explicit path for when the model errors, times out, or returns low confidence: a canned response, a simpler non-AI path, or a human handoff. The failure it prevents is not "the feature breaks." A feature that says "I can't help with that right now" is fine. The failure it prevents is a broken feature that answers anyway, fluently and wrong. One trap worth naming: a fallback that is itself a model call is not deterministic, because it inherits the exact failure it was meant to contain. The fallback has to be a rule, a cache, a simpler system, or a human. The rollout side of this lives in ship on containment.

Four commitments for a non-deterministic core#

A system whose core is non-deterministic and latency-variable needs four things that a deterministic CRUD app does not. Stream responses, because perceived latency matters more than raw latency on a multi-second call. Design the failure UX explicitly, so low confidence surfaces as a handoff rather than a confident wrong answer. Make retries idempotent, because non-determinism plus a flaky network means you will retry, and a non-idempotent retry double-sends the email or double-charges the card. And cache aggressively, since a stable prefix re-sent on every call is money left on the table. The UX half of this is the whole of the next post.

Gotchas, or what actually bites teams#

• "We'll add the gateway later" becomes never. Every call gets hand-written against one SDK, and six months later a price change forces a rewrite touching every call site instead of one router. Route through an abstraction from call number one, even a thin one.
• Agent creep. A workflow gains "just one more step the model could decide" until it is quietly an unconstrained agent nobody designed. Least privilege and step budgets erode one convenient tool at a time, not in one dramatic decision.
• No fallback means a confident wrong answer is the default failure. Teams build the happy path, skip the deterministic exit, and the first outage or low-confidence response ships straight to the user with no escape hatch.
• Model choice by leaderboard, not eval. A benchmark-topping model can lose on your task distribution. The only number that matters is your own harness on your own golden set.
• Caching treated as a nice-to-have. Skipping prompt caching on a stable system prompt leaves a large per-call discount on the table, and it compounds fast at volume.
• Retries without idempotency duplicate side effects. A retried tool call that sends an email or charges a card a second time is worse than the timeout it was papering over.

Where this goes next#

The diagram has one box that is entirely about the human on the other side of it: the failure UX, the streaming, the citations, the approval surfaces. That box is its own discipline, and it is where most AI products under-invest. The next architecture post is designing UX for a confidently-wrong intern. If you want the strategy layer above this diagram, start with why the demo is only 5% of the product.