01Weights are not the constraint

People size inference hardware by asking whether the model fits. For a 12B model in bf16 that is roughly 24 GB, or about 7 GB at 4-bit, and on a 40 GB card either answer looks comfortable.

Then the service goes live, a handful of users hold long conversations, and it runs out of memory. The weights never changed. The KV cache did.

During generation the model stores the key and value projections for every token it has already processed, so it does not recompute them for each new token. That store is the KV cache, and it grows linearly with context length and linearly with the number of concurrent requests.

02What the KV cache costs

For standard multi-head attention, per token:

2 × layers × hidden size × bytes per element

The 2 is for keys and values. Take a 32-layer model with a hidden size of 4096, served in bf16: 2 × 32 × 4096 × 2 bytes, which is about 512 KB per token.

Eight concurrent conversations at 4,096 tokens is about 16 GB of cache — more than the 4-bit weights of the model serving them. At 8,192 tokens and sixteen users you have exceeded a 40 GB card on cache alone.

03Grouped-query attention changes the number

The difference is not marginal. An eight-to-one ratio turns 512 KB per token into 64 KB, and the sixteen-user, 8,192-token case from 64 GB into 8 GB. Capacity planning that ignores GQA will either buy far too much hardware or, if you assume GQA where there is none, fall over in production.

The models we serve on Medusa sit in the 12B to 14B class, where architecture choices vary between families. There is no substitute for checking the config of the specific model you intend to serve.

04Working out your real ceiling

A procedure that takes about ten minutes and replaces a great deal of guesswork.

1. Read the config for layer count, hidden size, and the query-to-KV head ratio.
2. Compute cost per token using the formula above, divided by the GQA ratio.
3. Subtract weights and overhead from your card. Leave about ten per cent free for the allocator; planning to use the last gigabyte is how you get intermittent failures under load.
4. Divide the remainder by cost per token, then by your intended context length. That is your concurrency ceiling.
5. Compare against the length you actually see. Most services assume the maximum context; real conversations are usually far shorter, and sizing for the maximum wastes capacity.

05Levers that raise the ceiling

Cheapest consequence first.

• Cap context length. The most direct lever. Look at your actual token-length distribution before setting it — a long maximum is often driven by a handful of outliers you could truncate or handle separately.
• Quantise the cache. Serving the KV cache in 8-bit roughly halves it. Quality impact is usually small, but it is a change worth evaluating rather than assuming.
• Quantise the weights. Moving from bf16 to 4-bit frees memory that becomes cache headroom. This trades some model quality for concurrency.
• Use a serving stack with paged attention. Allocating cache in blocks rather than contiguous reservations substantially reduces waste from fragmentation and from requests that finish early.
• Add a card. The honest answer when the arithmetic does not close.

06What to measure in production

Three numbers tell you whether your ceiling estimate was right.

Peak concurrent requests, not average. Averages hide the moment that breaks you.

Token-length distribution, at the ninety-fifth and ninety-ninth percentiles rather than the mean. Your cache is sized by the tail.

Memory headroom at peak. If this regularly drops below ten per cent you are one traffic spike from failures, whatever the average says.

On a dedicated card these numbers are stable and attributable, because nothing else is competing for the memory — which is the practical argument for single tenancy when you are trying to characterise a service.

If you want the arithmetic checked against a specific model and traffic pattern, describe it: contact@vijaycloud.com.