01One request, many calls

In a chat product the unit of work is a turn: the user sends a message, the model generates a reply, the request ends. Capacity planning follows naturally from concurrent turns.

An agent inverts that. One user action becomes a loop: the model decides on a tool, the tool runs, the result is appended to the context, the model is called again, and this continues until it reaches an answer or a limit. A single user request might be five model calls, or fifty.

Every call in that loop carries the accumulated context of all previous steps. So the loop is not just more calls — it is calls with monotonically growing inputs, which is the more important property.

02The traffic shape changes

Chat traffic is roughly proportional to active users. Agent traffic is proportional to users multiplied by an unpredictable step count, and that step count is data-dependent.

The practical consequence: a load test that fires single-turn requests tells you very little about how the same hardware behaves under agent traffic.

03Memory behaves differently

Because each step appends to the context, the KV cache for an agent run grows as the run proceeds. A step that starts at two thousand tokens may be at twelve thousand by step ten, and the cache scales with that.

Worse for planning: the cache is often held during tool execution. If your agent calls an external API that takes four seconds, that run’s cache occupies memory for those four seconds while doing no computation at all. Ten concurrent agents mid-tool-call can consume the memory of ten long conversations while the compute units idle.

This is the specific mechanism by which agent workloads look like low GPU utilisation and simultaneously run out of memory — a combination that seems contradictory until you separate memory occupancy from compute occupancy, as our post on the utilisation problem does.

04Tool waits are not free

The instinct is that tool latency is somebody else’s problem: the GPU is idle, so nothing is being consumed. That is true of compute and false of memory.

Two mitigations are worth knowing.

Evict and recompute. Some serving stacks can drop a paused run’s cache and recompute the prefix when it resumes. That trades compute for memory, which is the right trade when memory is your binding constraint.

Cap the loop. A hard limit on steps per run bounds worst-case context growth. Without one, a single pathological task can consume the memory budget of many well-behaved ones.

Whether your stack supports the first is a question to settle before you design around it. Our post on KV cache sizing covers the underlying arithmetic.

05Failure modes you did not have

Agents introduce failures that chat products do not have, and most are capacity-adjacent rather than model-quality issues.

• Runaway loops. An agent that cannot make progress may retry indefinitely, consuming memory and compute for a task that will never complete.
• Context exhaustion mid-run. The run hits the context limit at step twelve and fails, having already spent the compute of eleven steps.
• Head-of-line blocking. A few long-running agents occupy the memory that many short requests needed, so unrelated users see queueing.
• Correlated bursts. If agent runs are triggered by a scheduled job rather than human behaviour, load arrives all at once rather than spread out.

06Planning capacity for agents

A procedure that gets closer than treating agents as chat.

1. Measure steps per run on your real tasks, and record the distribution rather than the mean. The tail sizes your hardware.
2. Measure context length at the final step, not the first. That is the peak the cache must hold.
3. Measure tool wait as a fraction of run duration. That fraction is the share of your memory doing no work.
4. Compute concurrency from peak context, using the KV cache arithmetic, then divide by the tool-wait factor to get sustainable concurrent runs.
5. Set a step cap and a context cap deliberately, and treat exceeding them as a product decision rather than a crash.

On a dedicated card these measurements are stable and attributable, because no other tenant is competing for the same memory — which matters more for agents than for chat, since the variance you are trying to characterise is already high.

If you are sizing for an agent workload and want the arithmetic checked, describe the task shape: contact@vijaycloud.com.