Treat the Model as a Flaky, Expensive Dependency

Here is a fact that sounds like a metaphor but is not: the model your product depends on is a third party that will change its pricing, retire itself, and even alter how it counts tokens, all while your product is live, and none of it will show up as a code diff in your repo. You will find out from a shocked finance team or a broken integration test, not a heads-up. Once you internalise that the model is an external dependency exactly like any other API you do not control, the engineering that follows is obvious. This post is that engineering.

This is the final post in the From Demo to Product series. It is also the most operational, and it pairs closely with my post on monitoring model deprecation in production, which is the hands-on version of one slice of this.

It is a scheduling fact, not a metaphor#

Providers run this on a calendar. As of mid-2026, Anthropic publishes a deprecation schedule: models retire on fixed dates, and once a model is gone, requests against it fail. Pricing moves on a calendar too. Sonnet 5, for example, shipped at an introductory rate that steps up around 50% to its sticker price on a published date, and the newer models use a tokenizer that produces roughly 30% more tokens for the same text, which means a per-request cost that lands 20 to 35% above what a naive before-and-after comparison would suggest.

I am quoting specific numbers from mid-2026 to make it concrete, but the exact figures are not the point, and they will be stale by the time you read this. The point is the shape. None of that is a bug in your code. It is the vendor changing the terms under a running product, which is exactly what any external dependency does. Go check the current pricing and deprecation pages before you trust any number in this paragraph; the fact that you have to is the whole lesson.

The mitigations are the boring ones#

The good news is that treating the model as a dependency does not require anything exotic. It requires the same discipline you would give a payments API or a third-party search service.

Retries with backoff. Transient failures, rate limits, connection blips, 5xx responses, should retry with exponential backoff. Permanent failures, a bad request or bad auth, should fail fast without retrying. This is table stakes for any network dependency, and a surprising number of LLM integrations skip it.

Caching. Identical or near-identical requests should not pay full price twice. A cache in front of the model cuts both cost and latency, and it is the first lever to pull when the bill jumps.

Versioning. Pin the model version you validated against, and treat a version bump like a dependency upgrade: run your evals before you roll it out, not after users notice. A model that silently changes behaviour is a regression you did not test.

Fallbacks. When the primary model is down, deprecated, or rate-limited, route to a backup. This is the difference between a degraded response and an outage.

Monitoring. Watch cost per request, latency, and quality as first-class production signals, not afterthoughts. The tokenizer change above is invisible unless something is watching the token count; the pricing step is invisible unless something is watching the bill.

The gateway is where this lives#

If you build one thing out of this post, build a single choke point that every model call flows through. Call it an AI gateway. It is where routing, caching, retries, fallback, versioning, cost logging, and one-switch rollback all live, so they exist once instead of scattered across every call site.

The value of the choke point is that vendor drift becomes a config change instead of a code change. A model retires: you repoint the gateway. Pricing jumps: your cost dashboard, fed by the gateway, alerts you the same day instead of at the end of the month. A new model looks better: you route a slice of traffic to it, compare on your evals, and promote or roll back from one place. Without the gateway, each of these is a scavenger hunt through the codebase.

The model is the least defensible part anyway#

There is a strategic reason this framing matters beyond reliability, and it ties back to an earlier post in this series. The model is not only a flaky dependency, it is the least defensible part of your product. It is rented, commoditised, and available to every competitor. So wrapping it in a gateway is not just an ops nicety; it is what keeps the swappable, commodity layer swappable. The day a cheaper or better model appears, the team with a gateway changes a line of config, and the team without one changes their architecture.

That is the quiet payoff. Treating the model as a dependency you can retry, cache, version, and replace is the same thing as refusing to let the model be your moat. The value you own lives in the layers around it: your data, your evals, your workflow. The model is electricity, and the gateway is the fuse box.

The traps#

The failures here are all failures of not watching:

Vendor drift while you are not looking. A pricing page changing on a date, a tokenizer swap inflating your token count, a model retiring on a schedule you did not track. If cost and behaviour monitoring are not on, you find out the hard way.

A model version bump that skips your evals. Treating "same model name, new snapshot" as a no-op is how a silent regression ships.

Hardcoding the model at every call site. It feels fine with one model and one call. It becomes a re-architecture the day you need a fallback.

Closing the loop#

That is the series. The demo is 5% of the product; the 95% is the evals, guardrails, gateway, failure UX, rollout, and drift monitoring that turn "worked in the demo" into "still working at 3am." The model is not your moat; your data and workflow are. Evals are the product discipline, discovered from real failures, not imagined up front. Not every problem needs an LLM, and the fit gate is the cheapest check you can run. And the model itself is a flaky, expensive dependency you wrap rather than trust. Do all five, and you land on the right side of the gap between a demo and a product.