01The GPU is waiting

The single most common surprise for teams new to their own hardware is that the expensive component is idle most of the time. Compute utilisation sits at twenty or thirty per cent, step times are inconsistent, and nothing in the training code looks wrong.

What is happening is that each step needs a batch of data prepared and delivered, and something in that chain cannot keep up. The GPU processes its batch and then waits.

This is worth diagnosing rather than tolerating, because the fix is usually configuration rather than hardware — and because a bigger card makes the ratio worse, not better.

02Storage: throughput and small-file pain

Two different storage problems get confused with each other.

Sequential throughput is what you need when streaming large shards. NVMe local storage handles this comfortably; network-attached storage may not, particularly if the link is shared.

Small-file overhead is the one that catches people. A dataset of two million individual images or JSON files pays per-file cost on every read, and that cost is dominated by operation count rather than bytes. The disk reports low throughput and appears idle, while the loader is bottlenecked on operations.

The fix for the second is to stop reading small files. Pack the dataset into a small number of sequential archives, or a format designed for streaming, and the same disk delivers many times the effective rate. This single change resolves more starvation than any other on this list.

03Host RAM and the dataloader

System memory does three jobs during training, and running short on any of them shows up as GPU idle time.

• Page cache. If your dataset fits in free RAM, the operating system serves repeat epochs from memory rather than disk. A dataset slightly larger than available RAM performs far worse than one slightly smaller, which is a sharp cliff rather than a gradual slope.
• Prefetch buffers. Loader workers prepare upcoming batches while the current one trains. Those queued batches live in RAM, and a shallow queue means the GPU waits at every step boundary.
• Worker copies. Each loader process carries its own memory. Raising worker count to fix a CPU bottleneck can create a RAM one, and the symptom is the job being killed rather than slowed.

The practical guidance: leave real headroom. A machine at ninety-five per cent RAM is one large batch away from a failure that will look mysterious in the logs.

04vCPU and preprocessing

Everything that happens to a sample before it reaches the device runs on the CPU: decoding, tokenising, augmenting, collating. If that work costs more per sample than the GPU takes to process it, the CPU sets your training speed.

Text pipelines are usually cheap once tokenised — and the word to notice is once. Tokenising inside the training loop, every epoch, is common and wasteful. Tokenise once, cache the result, and the CPU load largely disappears.

Image and audio pipelines are heavier, because decoding and augmentation are genuinely expensive per sample. These are the workloads where core count matters, and where moving augmentation onto the GPU can be worth the complexity.

A rough check: watch whether all cores are pinned during training. If they are, the CPU is your ceiling. If they are idle alongside the GPU, look at storage instead.

05The bus between host and card

Prepared batches still have to cross from host memory to the device. This is rarely the bottleneck for text, and occasionally is for high-resolution image or video work with large per-sample payloads.

Two things help when it is. Pinned host memory allows faster transfers than ordinary allocations. And overlapping transfer with computation — moving the next batch while the current one trains — hides most of the cost, which is a framework setting rather than a hardware one.

We mention it for completeness rather than emphasis. Diagnose storage, RAM and CPU first; this is the least likely of the four.

06Diagnosing which one it is

07What our plans provide

So you can check the arithmetic against your own dataset rather than discovering it later.

Two things to check against those numbers. Does your dataset fit in host RAM, and therefore in page cache? And does it fit on local NVMe alongside your checkpoints — remembering that a full-precision checkpoint of a 12B model is on the order of twenty-odd gigabytes, and that keeping ten of them adds up.

If your data does not fit either budget, that is worth raising before you start rather than after. Describe the dataset and we will size it: contact@vijaycloud.com.