Everyone assumes serving an LLM is bottlenecked by raw compute — by how fast the GPU can multiply matrices. Character.AI's engineers will tell you the real ceiling is something almost nobody mentions: memory for remembering what was already said.
Character.AI serves around 20,000 queries per second at a cost of less than one cent per hour of conversation — and they got there by attacking not the model's math, but its memory. Specifically, a structure called the KV-cache, which turned out to be the thing standing between them and affordable scale.
When an LLM generates text, it produces one token (roughly, one word-piece) at a time, and each new token depends on everything that came before. To avoid re-reading the entire conversation from scratch for every single token, the model stores a summary of the prior context — the KV-cache (key/value cache) — in fast GPU memory.
Here's the squeeze: that cache grows with the length of the conversation and with the number of people chatting at once. GPU memory is finite and expensive. So the number of conversations you can serve simultaneously isn't limited by how fast the chip computes — it's limited by how many KV-caches you can fit in memory. Throughput is memory-bound, and almost everyone's intuition says it should be compute-bound.
Why the obvious lever is the wrong one
Faced with "we can't serve enough users," the instinct is buy more GPUs or a faster GPU. But if each conversation hogs memory, a faster chip just sits idle waiting — you've paid for compute you can't use because memory filled first. The leverage isn't in the processor. It's in shrinking the KV-cache so more conversations fit in the memory you already have.
Character.AI stacked three techniques to cut the KV-cache by more than 20x:
Multi-query attention. Standard attention keeps a separate set of keys and values for every attention "head." Multi-query attention shares one set of keys/values across all heads, slashing the per-token memory footprint with minimal quality loss.
Cross-layer KV sharing. A model has many layers, each normally storing its own KV state. Character.AI shares KV state across layers, cutting the cache further.
Int8 quantization. Store the cache (and weights) using 8-bit integers instead of 16-bit floats — roughly half the memory for each cached value.
Then a fourth idea attacks repeated work. In a chat, every new message re-includes the whole conversation history. Naively, the model re-processes that shared prefix every turn. Using a RadixAttention-style prefix cache, Character.AI keeps the already-computed KV state for the shared conversation prefix between turns — so a long conversation doesn't pay to re-read itself on every message. This is, in spirit, the same insight as semantic caching: don't recompute what hasn't changed.
Worth knowing
The headline "<1¢ per hour of conversation" is what makes consumer-scale conversational AI economically viable at all. And it came almost entirely from memory engineering, not from a better model or more silicon. When a workload is memory-bound, every byte you save in the hot structure converts directly into throughput — which is why the KV-cache, not the GPU, was the right thing to obsess over.
The gap it reveals
As LLMs enter every system design, the naive mental model is "it's a compute problem, throw GPUs at it." The 2026-relevant realisation is that LLM serving is dominated by KV-cache memory, and that the highest-leverage optimisations (multi-query attention, KV sharing, quantization, prefix caching) all target memory footprint. Engineers who understand this reason about AI infrastructure correctly; everyone else reasons about a problem that isn't the bottleneck.
In the interview room
"Design a chatbot serving system" is the new hotness in design rounds, and the discriminating move is to identify the bottleneck correctly: "throughput here is bound by KV-cache memory, not compute, so I'd focus on shrinking the per-conversation cache and caching shared prefixes between turns." Most candidates will talk about model size and latency; naming the memory wall sets you apart immediately.
The reframe
Every new technology arrives wrapped in a wrong intuition about where its limits are. With LLMs, the wrong intuition is "compute." The real constraint — for serving, at least — is the memory cost of context. Character.AI's edge wasn't a secret model; it was correctly identifying which resource was actually scarce and engineering relentlessly against that one.
The GPU was never the bottleneck. Remembering the conversation was.
Primary source →
blog.character.ai — Optimizing AI Inference at Character.AI