01Three problems, one request

“We need more GPUs” is a conclusion, and it arrives attached to one of three quite different observations.

• It does not fit. The job fails with an out-of-memory error.
• It is too slow. The job runs but takes longer than the team can tolerate.
• We cannot serve everyone. Requests queue or are rejected under load.

These have different diagnoses and different cheapest fixes. Only the first is a hard constraint, and even it usually has software answers before it has hardware ones.

02Memory: it does not fit

The only genuinely binary case. A training step that needs more memory than the card has will not run, at any speed.

Before adding hardware, the levers in sizing VRAM apply: reduce physical batch size and use gradient accumulation, shorten sequence length if the data allows, enable gradient checkpointing, switch from full fine-tuning to LoRA or from LoRA to QLoRA, and consider a memory-efficient optimiser.

Those levers span a very wide range. Full fine-tuning a 7B model needs roughly 112 GB; QLoRA on the same model needs roughly 4 GB. If you have not walked down that list, a second card is buying you something a configuration change would have given free.

When the arithmetic genuinely does not close — full fine-tuning at scale, or a model whose weights alone exceed one card — that is a real multi-GPU requirement, and the honest answer is that no amount of tuning fixes it.

03Time: the run is too slow

This is the case most often misdiagnosed, because slowness feels like a compute problem and usually is not.

Before assuming you need more compute, check whether the GPU is actually busy. If compute utilisation sits at thirty per cent during training, adding a second card will give you two cards at thirty per cent. The bottleneck is elsewhere, and our post on the specs that throttle training and are not the GPU covers the signatures — small-file overhead, page cache misses, preprocessing in the training loop.

If the GPU genuinely is saturated and the run is still too slow, a second card helps, but sub-linearly. Gradient synchronisation costs bandwidth every step, so two cards deliver meaningfully less than twice the speed, and the gap widens as model size grows relative to interconnect capability.

Two questions worth asking before paying that overhead. Does the run need to be faster, or does it need to start sooner — because queueing and scheduling might be the real problem? And is the experiment loop the constraint rather than the run, in which case running several variants concurrently on one card may serve you better than running one variant faster.

04Throughput: demand exceeds one card

For serving, the question is whether you have exhausted the card or exhausted your configuration.

Concurrency on a single card is bounded by memory, and specifically by KV cache. Before adding hardware, work through the levers in the cache article: cap context length to what your traffic actually uses, quantise the cache, quantise the weights to free cache headroom, and ensure your stack does continuous batching and paged allocation.

Those changes can multiply concurrent users on the same hardware. A team serving eight users on a 40 GB card in bf16 with a long context cap may serve several times that after tuning, which is a cheaper outcome than a second card and less operational surface.

Serving does scale more gracefully across cards than training does, because requests are independent — no gradient synchronisation, so two cards genuinely approach twice the capacity. So when you do need it, this is the easier case.

05What multi-GPU costs you

Worth stating plainly, because it is usually underestimated.

06What to try first

A short procedure that resolves most of these requests without new hardware.

1. Identify which of the three problems you have. Out-of-memory, slow, or insufficient concurrency. They are not the same and do not share a fix.
2. If out-of-memory: work down the memory levers. Most jobs that appear not to fit are a batch size and a checkpointing flag away from fitting.
3. If slow: check compute utilisation first. Below roughly fifty per cent, fix the data path rather than buying compute.
4. If concurrency: cap context, quantise the cache, confirm continuous batching. Then re-measure.
5. Re-measure after each change rather than applying several at once, so you know what worked.

07Signals it is genuinely time

Four situations where a second card is the correct answer rather than an avoidance of tuning.

• Model weights alone exceed one card. No configuration change fixes this.
• You need full fine-tuning at a size where the arithmetic cannot close — anything from 7B upward.
• Compute utilisation is genuinely high and sustained and the run is still the constraint on your team.
• Serving concurrency is memory-bound after tuning, and traffic is still growing.

Multi-GPU and bare-metal configurations are the Enterprise conversation on our side, sized per workload rather than picked from a plan. If you want the arithmetic checked before committing to it, describe the workload: contact@vijaycloud.com.