The hard part of production AI has quietly stopped being model quality. Frontier and open-weight models are close enough on most tasks that the differentiator is the system around them — whether it stays up, what it costs per request, and how it behaves on the worst day rather than the average one. This is a working guide to the three constraints that actually bind, and the arithmetic to size each.

August 2026 · ~12 min read

The bottleneck moved

For most of the last few years, the answer to “our AI feature isn’t good enough” was a better model. That answer has largely run its course for everyday workloads. A 2026 open-weight release like DeepSeek-V4 or Kimi K3 will handle the great majority of production tasks competently. What separates a demo from a service is no longer the weights.

It is three things, and they are all systems problems:

• Scale is bounded by memory, not compute. The intuition that “more GPUs means more throughput” breaks down because the binding constraint on a serving node is usually KV cache capacity and memory bandwidth, not FLOPs.
• Correctness is a distribution, not a boolean. Traditional fault tolerance assumes a call either succeeds or fails. A model call can succeed, return in 200 ms, and be wrong — and no HTTP status code will tell you.
• Optimization has a strict payoff order. Teams routinely spend weeks on a 15% kernel win while a routing change sitting one layer up would have cut cost by 60%.

The rest of this article takes each in turn.

Scale: memory is the budget

The single most useful piece of arithmetic in LLM serving is the size of the KV cache, because it — not parameter count — usually determines how many concurrent requests a node can hold.

For a transformer using grouped-query attention, the cache for one token is:

bytes_per_token = 2 (K and V)
                × n_layers
                × n_kv_heads
                × head_dim
                × bytes_per_element

Take a 70B-class model with 80 layers, 8 KV heads, head dimension 128, served in FP16:

2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 320 KiB per token

At an 8K-token context, one sequence holds roughly 2.5 GiB of KV cache. On an 80 GB accelerator that already has ~140 GB of weights sharded across the group, the practical ceiling is a couple of dozen concurrent sequences — long before you run out of arithmetic throughput.

Three consequences follow, and they are where the 2026 architecture work has concentrated.

Attention design is now a serving decision. Multi-head latent attention and the compressed-sparse hybrids in recent open models exist specifically to shrink this number. DeepSeek reports its V4 hybrid attention needing roughly 10% of the KV cache of V3.2 at 1M-token context. That is not an academic result — it is a direct multiplier on how many users fit per node.

Continuous batching beats static batching decisively. Static batching waits for a full batch, then runs it to completion, so every sequence pays the cost of the longest one. Continuous batching admits and evicts sequences at each decode step, keeping the accelerator saturated.

Prefix caching turns a cost centre into a rounding error. Agent and RAG workloads resend a large, stable prefix — system prompt, tool definitions, retrieved context — on every turn. Caching that prefix converts most of the prompt from full-price prefill into a near-free cache read. The catch is that caching is a prefix match: one byte changing early invalidates everything after it. A timestamp interpolated into the top of a system prompt is enough to make the cache useless, and the failure is silent — you simply keep paying.

Fault tolerance when correctness is a distribution

Classical resilience patterns — retries, timeouts, circuit breakers, bulkheads — all still apply, and most teams implement them. The gap is that they were designed for a world where a call either works or throws. Model calls have a third state, and it is the dangerous one.

The bottom two rows are the ones that break naive clients. Code that reads response.content[0] unconditionally works fine for months and then produces a blank message or an exception the first time a safety classifier declines a request — because the call succeeded. Any client that talks to a model needs to inspect the stop reason before it touches the content.

For the last row there is no transport-level signal at all, which means reliability has to be built at the application layer:

• Ground it. Retrieval with citation enforcement converts “is this true?” into “is this supported by the retrieved span?”, which is checkable.
• Verify selectively. A second, cheaper model checking a claim against its source catches a meaningful share of errors at a fraction of the cost of a second full generation. Spend it on the high-consequence paths only.
• Give it somewhere to abstain to. A system with no “I don’t know” branch will always fabricate, because that is the only path you left open.

Degrade in steps, not off a cliff

The most useful reliability property is a ladder of reduced service rather than a binary up/down. Each rung should be independently triggerable by a health signal.

Two details make the difference between a ladder that works and one that exists only in a design document. First, the bottom rung cannot depend on the thing that failed — a “fallback” that calls the same provider with a smaller model is not a fallback during a provider outage. Second, every rung needs a test that exercises it, because untested fallback code is reliably broken exactly when it is needed.

Idempotency is not optional

Agentic systems retry. The moment a workflow can call a tool that sends an email, charges a card, or writes to a ledger, retry semantics become a correctness issue rather than a convenience. Give every side-effecting operation an idempotency key derived from the logical task, not from the attempt:

# Wrong — a retry produces a second charge
key = uuid4()

# Right — the same logical action collapses to one effect
key = sha256(f"{workflow_id}:{step_id}:{payload_digest}").hexdigest()

The same reasoning applies to queues. When ordering and exactly-once handling actually matter — a payment sequence, a state machine — a FIFO queue with deduplication is worth the throughput cost. When they do not, do not pay it.

The optimization order that pays

Optimizations are not commutative in value. The ones furthest from the hardware usually pay the most, and they are the ones teams reach for last. A rough ordering by typical return on effort:

Levers 1 and 2 are architectural and cost almost nothing to try. Lever 8 is where the fun is, and it is where a great deal of engineering time goes before anyone has measured whether the request needed to happen at all.

Routing deserves special mention because it compounds with everything else. Real workloads are heavily skewed — a large share of traffic is easy, and a small tail is genuinely hard. Sending all of it to your most capable model prices the easy majority at the cost of the hard minority.

A reference shape

Putting it together, the architecture that holds up in production has a recognisable shape. Nothing here is exotic; the discipline is in having each layer present and independently testable.

What to instrument

You cannot operate what you cannot see, and the default metrics are the wrong ones. Averages hide the failures that matter; a mean latency of 900 ms is compatible with 5% of users waiting eight seconds.

• Latency at p95 and p99, never the mean. Also split time-to-first-token from total time — they have different causes and different fixes.
• Cost per request, broken down by route. Aggregate spend tells you that something changed; per-route cost tells you what.
• Cache hit rate. If it drops toward zero across identical-prefix requests, something is silently invalidating the prefix. This is worth an alert.
• Stop-reason distribution. A rising share of truncations or refusals is a real regression that no error-rate dashboard will show, because those responses are all HTTP 200.
• Quality on a fixed evaluation set, on every deploy. Treat a regression here exactly like a failing test: it blocks the release. This is the single practice that most separates teams who ship confidently from teams who ship and hope.
• Fallback exercise rate. If a degradation rung has not fired in ninety days, you do not know whether it works. Fire it deliberately.

Closing

None of this is about models, which is the point. The field spent several years where the dominant term in system quality was model quality, and that era is ending — the weights are good, increasingly open, and increasingly commoditised.

What remains is ordinary, demanding engineering: knowing that memory rather than compute sets your ceiling, that a successful HTTP response is not evidence of a correct answer, and that the cheapest request is the one you found a way not to make. The systems that hold under load in 2026 are the ones built by people who took those three facts seriously before traffic forced them to.

Written August 2026. Corrections and disagreement welcome — dkiran238@gmail.com. Back to Articles.