The Unit Economics of an AI Product

A line of application code runs for effectively nothing, a million times over. That is why classic SaaS gross margins sit high: the marginal cost of one more user is a rounding error. An LLM call is not like that. It is a metered utility, and you pay per token, every time, forever. That single fact reshapes the whole business, because it means you can build something users genuinely love that loses money on every single use, and never notice until the invoice arrives. This post is how to model that reality and how to keep it from sinking you.

This is part eight of the From Demo to Product series. Cost is the operations topic engineers most reliably ignore until it becomes a crisis, so it is worth doing the thinking early.

The mental model: the model is a metered utility, not free code#

You have to treat inference the way a cloud team treats compute: instrument it per request, attribute it per feature, and optimise it deliberately. The reason "it worked in the demo" tells you nothing about profitability is that a demo runs a handful of requests, and the cost only becomes real at the scale and the usage pattern of actual customers. So the first move is not to optimise anything. It is to measure, so you know which requests actually cost you money.

The unit is the task, not the call#

Providers bill per token, split into input (your prompt and context) and output (the generation), and the two are not priced the same. Output is the expensive side by a wide margin, several times the input price, which already tells you something: verbose generations cost far more than verbose prompts.

But the deeper trap, the one almost everyone falls into, is pricing against a single generation. The unit that actually matters is cost per task, and a task is usually many calls. Consider a typical retrieval-augmented answer: you embed and retrieve, you rerank, you generate, you run an LLM judge to check faithfulness, and you retry once when something fails. That is several model calls for one user-visible answer. A multi-step agent is worse: it can be an order of magnitude or two more expensive than a single call, because each step in the loop is its own round trip.

So the thing to instrument is the sum across the whole task, not the price of one generation:

cost_per_task ≈ Σ over calls ( input_tokens × input_price + output_tokens × output_price )
              + retrieval and rerank cost
              + tool costs

Instrument this per request in your tracing. You cannot optimise what you do not measure, and the number that a demo makes feel cheap is exactly the number that scales into a problem.

AI is not SaaS on margins#

Because every request carries real cost of goods, AI gross margins land well below the SaaS bar that investors are used to. The reported figures vary a lot by source and by how heavily the product is used, from comfortably-positive-before-optimisation at the optimistic end to actually-negative-before-optimisation at the sober end. I am not going to quote a single number as gospel, because the honest answer is that it depends on your task, and the range is wide enough that a single figure would mislead. What every source agrees on is the direction: structurally lower than SaaS, and dangerous if you leave it unmodeled.

Two implications follow, and both are easy to miss:

• Model the margin per pricing plan, not just the top-line price. A flat-fee plan plus a power user running a very high volume of agent tasks can be margin-negative on that user alone. Your heaviest users subsidise themselves straight into your loss column, and they are invisible in an average.
• Investors increasingly accept lower AI margins if the moat is real. "Moats before margins" is a defensible stance, but only if you actually know the number and can say why it is acceptable, not because you never calculated it.

The cost curve is a tailwind, not a plan#

There is real good news here. Inference cost for a given capability tier has been falling steeply, and the trend is expected to continue: a capability that is expensive today tends to be markedly cheaper in a year or so. So an app that looks marginal at today's prices often becomes comfortably profitable as the curve drops underneath it.

But two cautions. Do not bet the business on an unrealised future price cut; plan around today's cost and treat the decline as upside, not as your break-even assumption. And do not over-optimise prematurely, trading speed to market for savings you might not need, on a curve that may make the savings irrelevant anyway. Ship, measure, then optimise the part that is actually hot.

The levers, in ROI order#

When you do need to bring cost down, the levers are not equal, so pull them in this order:

1. Model routing, the biggest lever by far. Send routine work to a cheap, small model and reserve the frontier model for genuinely hard requests. Either classify difficulty up front, or try the cheap model first and escalate only when it fails. This compresses the inference bill more than anything else you can do, which is why it is first.
2. Caching, which takes a large fraction off repeats. Prompt caching on a stable system-and-instruction prefix reads back at a small fraction of the base input price. Result caching handles repeat queries outright. This is enormous for RAG and agent apps, where a big fixed prompt is sent over and over.
3. Context discipline. Fewer, better tokens are a direct cost cut, often at the same quality. Do not stuff the window "just in case"; every token you send you pay for, and most "just in case" context never gets used.
4. Cap the loop. Step, token, and dollar budgets per run stop runaway agent costs. A stuck loop is not a curiosity, it is a cost incident, and without a cap it can multiply a single task's cost dramatically before anyone notices.
5. Right-size the output. Output is the expensive half, so ask for concise, structured responses and do not let the model ramble.
6. Batch or use smaller models where latency allows and where your evals show the smaller model holds.

Do all of this behind a gateway so the levers are centralised and swappable, rather than reimplemented at every call site where they will drift out of sync.

Build a cost model before you price#

Do not price on vibes. Fill this in with your own task's numbers before you commit to a plan:

Per task:
  calls_per_task            = ___   (retrieve + rerank + generate + judge + retries)
  avg_input_tokens_per_call = ___   ;  avg_output_tokens_per_call = ___
  retrieval / tool cost      = ___
  cost_per_task             ≈ ___
 
Per user / month:
  tasks_per_user_month = ___  →  cost_per_user = cost_per_task × tasks
  price_per_user       = ___  →  gross_margin  = 1 − (cost_per_user / price_per_user)
  worst-case power user (N× tasks): margin = ___  →  cap or meter?
 
Levers applied: [ ] routing  [ ] prompt cache  [ ] result cache  [ ] context trim  [ ] loop budget  [ ] concise output

With that in hand, the pricing choice is a margin-safety decision as much as a UX one:

Gotchas, or what actually bites teams#

• Per-call thinking hides the real number. People price against a single generation; the real unit is the task, and agent tasks are many calls. Instrument cost per task or you will misprice.
• The intro-pricing cliff. A model's introductory rate expires on a calendar date, and a cost model built on intro pricing silently goes underwater that day. Date every price and re-check on a schedule.
• Cache misses you did not notice. A timestamp or an unsorted JSON blob in the prompt prefix quietly breaks prompt caching, and the big discount you were counting on disappears. Verify your cache hit rate; do not assume it.
• Optimising before you have shipped. The cost curve is falling fast, so over-engineering cost before you have real usage trades speed to market for savings you may not need. Ship, measure, then optimise the hot path.
• Quoting a single margin figure as gospel. The numbers are quoted pre-optimisation and vary by source and usage. Model your task's cost and margin rather than borrowing someone else's headline.

Where this goes next#

Cost blowups are one item on a longer list of risks that sink AI products, and they all share the same shape: you do not fix them once, you detect, contain, and monitor them. The next post is detect, contain, monitor: the risks that sink AI products. The operations loop this cost work plugs into is LLMOps and the data flywheel.