Batching and streaming vs latency

The short version

Batching raises tokens per second across all users and can add waiting to any individual request. Prefill and decode have opposite bottlenecks, so a setting that helps one can hurt the other. For interactive products optimise time to first token and stream; for batch work optimise throughput and ignore latency.

01The two goals pull apart

Two numbers describe a serving system, and improving one usually costs the other.

Throughput is total tokens per second across everyone. It decides how many users a card supports, and therefore your cost per request.

Latency is what one user waits. For anything interactive it is what they perceive as quality, and a fast system that feels slow is a product problem regardless of its throughput.

Most tuning advice optimises throughput, because that is what benchmarks measure. If your product is a chat interface, following that advice will make it worse. So the first decision is not a setting but a question: which of these two are you actually optimising?

02Prefill and decode behave differently

Generation has two phases with almost opposite characteristics, and conflating them causes most confusion about serving performance.

Prefill processes the whole prompt at once. All tokens are handled in parallel, so the phase saturates compute and is generally compute-bound. A long prompt makes prefill expensive.

Decode generates one token at a time, each depending on the last. There is little parallel work per step, so the phase is dominated by reading weights and cache from memory — memory-bandwidth-bound, which is why bandwidth matters more than TFLOPS for generation speed.

Two phases, two bottlenecks
	Prefill	Decode
Work per step	Whole prompt, parallel	One token, sequential
Usually limited by	Compute	Memory bandwidth
Scales with	Prompt length	Output length
Benefits from batching	Modestly, already parallel	Substantially
Determines	Time to first token	Tokens per second after that

The consequence: batching helps decode considerably and prefill much less, so its benefit depends on the ratio of prompt length to output length in your traffic.

03What continuous batching changes

Naive batching collects a fixed group of requests, runs them together, and waits for all to finish. One long generation holds the whole batch, and requests arriving mid-batch wait for the next one.

Continuous batching lets requests join and leave the running batch on token boundaries. A finished request frees its slot immediately; a new arrival joins at the next step rather than waiting for a boundary.

This is the largest improvement available in modern serving, and unusually it improves both numbers at once. It raises utilisation because slots do not idle, and lowers waiting because arrivals are not queued to a boundary.

If your stack does not do this, that is the first thing to change rather than tuning anything else.

04Time to first token is what users feel

For interactive products, total generation time is close to irrelevant and time to first token is nearly everything.

A response that begins in 300 milliseconds and takes eight seconds to finish feels fast. One that begins after four seconds and finishes in five feels broken, despite being faster overall. People read at a limited rate; once output flows faster than they read, extra speed is invisible.

Time to first token is dominated by prefill, and prefill scales with prompt length. So the levers are prompt-side: shorten system prompts, avoid stuffing context that does not change the answer, and cache prefill for shared prefixes if your stack supports it. A long fixed system prompt paid on every request is the most common avoidable cost here.

05Streaming buys perceived speed

Streaming does not make generation faster. It makes waiting shorter, which for an interactive product is the thing that matters.

Two implementation details quietly undo the benefit.

Do not buffer. A proxy, load balancer or framework that accumulates the response before forwarding converts a streaming API into a blocking one. The symptom is that streaming appears not to work while the server insists it does.

Do not wait for a parseable structure. If your client needs complete JSON before rendering, you have serialised the whole generation again. Stream prose and deliver structured fields separately.

06Settings that trade one for the other

Common settings and which way they push
Setting	Raise it	Lower it
Maximum batch size	More throughput, more variance per request	Steadier latency, lower utilisation
Maximum context length	Handles longer inputs	More concurrency from the same memory
Prefill chunk size	Faster long prompts	Decode interrupted less by arrivals
Memory reserved for cache	More concurrent users	Headroom against spikes
Speculative decoding	Faster decode when guesses land	Simpler, more predictable behaviour

The one that surprises people

Raising maximum batch size can worsen the experience for every individual user while improving your throughput graph. If your dashboard tracks tokens per second and your users track how long they waited, you can optimise the dashboard and degrade the product.

07What to measure

Four numbers, and none of them is an average.

Time to first token, at the fiftieth and ninety-ninth percentiles. The tail generates the complaints.
Inter-token latency separately from time to first token. Different causes, different fixes.
Total throughput at your target latency, not at maximum batch. A throughput figure with no latency constraint attached is not a useful number.
Queue depth over time — the earliest warning that arrival rate has exceeded service rate, as covered in what to monitor on a GPU box.

Tune on a dedicated card if you can. On shared hardware your latency percentiles include someone else’s traffic, and you will spend time attributing variance that was never yours to fix.

If you want help picking a target latency and sizing for it: contact@vijaycloud.com.

Batching and streaming without hurting latency