Quantisation for serving

The short version

Serving in 8-bit roughly halves weight memory against bf16; 4-bit roughly quarters it. That freed memory becomes KV cache, which is what limits concurrency. Quality loss is usually small on general tasks and concentrated in long-tail and reasoning-heavy cases, so evaluate on your own hard examples rather than on averages.

01What quantisation actually does

A model stores each weight as a number. Serving in bf16 uses two bytes per weight. Quantisation stores those numbers with fewer bits and keeps a small amount of extra information — scaling factors, typically per block of weights — so the original values can be approximated when needed.

Two consequences follow, and they pull in opposite directions.

The memory footprint drops roughly in proportion to the bit width. That is the gain, and it is large and certain.

Each weight is now an approximation. The error is small per weight, but it compounds through the layers, and it compounds unevenly — which is why the quality effect is not uniform across tasks.

02The memory arithmetic

Weight memory by precision, approximate
Precision	Bytes per parameter	7B model	12B model	14B model
bf16 / fp16	2	~14 GB	~24 GB	~28 GB
8-bit	~1	~7 GB	~12 GB	~14 GB
4-bit	~0.6	~4 GB	~7 GB	~8 GB

The 4-bit figures are above 0.5 bytes because of the scaling constants, which are real and worth including in a capacity plan.

What matters is not the saving in isolation but what it becomes. On a 40 GB card serving a 14B model, bf16 weights leave roughly 12 GB for cache after overhead; 4-bit leaves closer to 32 GB. That is not a marginal difference in concurrency — it is a different service. Our post on how the KV cache sets your user ceiling turns that headroom into a user count.

03Where the quality cost shows up

Aggregate benchmark scores tend to move very little at 8-bit and modestly at 4-bit, which is why summary numbers make quantisation look almost free. Averages hide where the loss lands.

In practice degradation concentrates in a few places:

Multi-step reasoning. Small errors early in a chain propagate. A task that requires holding several constraints at once is more sensitive than one that requires recall.
Rare tokens and specialised vocabulary. Domain terms, code identifiers, names and non-Latin scripts are represented by weights that see less training signal, and they degrade first.
Long contexts. Errors accumulate over more tokens.
Instruction adherence at the edges. Strict format compliance can loosen in ways that average metrics do not capture but downstream parsers certainly do.

The practical implication

If your product does retrieval and summarisation, quantisation is likely close to free. If it does multi-step reasoning over specialised vocabulary and you parse the output strictly, test carefully at 4-bit and consider 8-bit as the default instead.

04Weights, cache and activations are separate decisions

People say “we quantised the model” and mean only the weights. There are three distinct places precision can be reduced, with different trade-offs.

Weights. The largest fixed saving, and the best understood. This is what almost every quantisation discussion means.

KV cache. Often the better lever for a serving workload, because on a concurrent service the cache can exceed the weights. Storing the cache in 8-bit roughly halves it. Quality impact is usually smaller than weight quantisation at the same bit width, since the cache is transient rather than compounding across layers.

Activations. Reducing precision during computation can improve throughput on hardware with the right support, but it is the most sensitive of the three and the least worth doing casually.

If memory is your constraint and concurrency is your goal, quantising the cache before the weights is often the better first move, and it is the option most teams overlook.

05Choosing a format

We would not recommend a specific library, because the landscape moves faster than any article. The properties to compare:

What to check when comparing quantisation approaches
Property	Why it matters
Requires a calibration dataset?	Calibrated methods usually preserve quality better, but the calibration data must resemble your traffic
Supported by your serving stack?	A format your server cannot load is not an option, whatever its benchmarks say
Throughput effect	Some formats save memory but add dequantisation overhead, trading latency for capacity
Group or block size	Smaller groups preserve quality and cost more overhead bytes
Does it quantise the cache too?	Often the more valuable saving for concurrent serving

On a dedicated card you can hold two quantisations of the same model and compare them against identical traffic, which is a far more reliable comparison than published benchmarks against a different workload.

06How to evaluate it honestly

The failure mode here is evaluating on the easy cases, concluding quantisation is free, and discovering the cost in production.

Build a hard set. Fifty to a hundred real inputs chosen because they are difficult: long, multi-step, domain-heavy, format-strict. Averages will not surface the loss; these will.
Compare against the unquantised model on identical inputs, same seed, same decoding settings. Changing two things at once is how quantisation gets blamed for a sampling change.
Measure format compliance separately from answer quality, if anything downstream parses the output.
Measure latency and time-to-first-token, not just throughput. A format that raises total throughput while worsening first-token latency can be wrong for an interactive product.
Decide the concurrency you actually gained, using the freed memory. That is the benefit side of the trade, and it should be a number rather than an impression.

If you want help sizing that trade for a specific model and traffic pattern, describe it: contact@vijaycloud.com.

Quantisation for serving: what you gain, what you give up