01What is actually filling the memory

Before you change anything, know which of the four consumers is the problem. A training step holds weights, gradients, optimizer state and activations at the same time, and the techniques below only touch some of them.

This matters because the two most commonly recommended fixes, accumulation and checkpointing, only reduce the last row. If your problem is that a full fine-tune of a 7B model will not fit at all, no amount of accumulation helps: the weights, gradients and optimizer moments are already over budget before a single activation exists. That is a case for a parameter-efficient method or a bigger card, and it is worth establishing which situation you are in before you spend an evening tuning flags.

The practical test: drop batch size to 1 and sequence length to something small. If it now runs, your problem is activations and this article applies. If it still fails, your problem is the first three rows and you want a different training method.

02Gradient accumulation: a bigger batch without the memory

Accumulation splits a batch into smaller pieces. You run a forward and backward pass on each piece, add the gradients into the same buffer, and only step the optimizer once you have been through all of them. Sixteen micro-batches of 2 give you the gradient of a batch of 32, while never holding more than 2 samples worth of activations.

What you pay is wall-clock time, and not always in the way people expect. The arithmetic per sample is unchanged, so in theory the cost is nothing. In practice small micro-batches use the GPU less efficiently, so throughput drops: the same tokens per epoch take longer. How much depends on how small you go. Going from 8 to 4 is often nearly free. Going to 1 can be substantially worse per sample, because the matrix multiplications become too small to keep the GPU busy and per-step overheads stop being amortised.

One correctness note that bites people. If your loss is averaged over the batch, you have to divide each micro-batch loss by the number of accumulation steps, or scale the accumulated gradient afterwards. Get it wrong and your effective learning rate is multiplied by the accumulation count, which usually shows up as a loss curve that diverges early. Most training frameworks handle this for you; if you have written the loop yourself, check it.

The other thing accumulation does not fix is anything that normalises across the batch. Batch normalisation computes statistics per micro-batch, so accumulating 16 micro-batches of 2 is not equivalent to a batch of 32. Transformers generally use layer normalisation, which is per-sample, so this rarely matters for language model fine-tuning. It matters a lot for vision architectures that still use batch norm.

03Gradient checkpointing: recompute instead of remember

The backward pass needs the intermediate activations from the forward pass to compute gradients. Normally every layer keeps its output around for this, which is why activation memory grows with depth. Checkpointing keeps only a subset, and recomputes the rest on demand during the backward pass by re-running the forward pass from the nearest saved point.

The saving is large. Instead of storing activations for every layer you store them at intervals, which turns memory that grows with depth into something closer to memory that grows with the square root of depth, depending on how the checkpoints are placed. On a deep transformer this is often the difference between a sequence length of 2k and 8k at the same batch size.

The cost is one extra forward pass over the recomputed segments. A forward pass is roughly half the work of a backward pass, so the usual figure quoted is around 30 to 40 percent more compute per step. That is a real cost and it is predictable, which makes it easy to reason about: if checkpointing lets you double your sequence length and slows each step by a third, that is almost always a good trade.

Two practical notes. First, checkpointing interacts badly with anything nondeterministic in the forward pass. If you have dropout, the recomputed forward has to use the same random state as the original, otherwise your gradients are computed against a different network than the one that produced the loss. Framework implementations handle this by saving and restoring the RNG state; custom implementations often do not, and the symptom is a run that trains but converges worse than it should for no obvious reason. Second, checkpointing is usually configured per layer block rather than globally, and the default granularity is not always the best one. If you are still short of memory with it on, checkpoint more aggressively before concluding it did not work.

04Mixed precision, and where it actually bites

Mixed precision stores weights and activations in 16-bit while keeping a 32-bit copy of the weights for the optimizer update. Activations roughly halve, which is direct relief, and on hardware with tensor cores the arithmetic is also considerably faster. This is standard practice now rather than an optimisation, and if you are not using it you should start there.

The choice worth understanding is fp16 versus bf16. Both are 16 bits. They divide those bits differently: fp16 has more mantissa and less exponent, bf16 has the same exponent range as fp32 and less mantissa. In practice that means bf16 has worse precision per value but does not overflow or underflow where fp16 does.

This is why fp16 training needs loss scaling. Gradients are small numbers, and small numbers underflow to zero in fp16, so the loss is multiplied by a large factor before the backward pass and the gradients divided by it afterwards. Dynamic loss scaling adjusts that factor when it detects overflow, which works but adds a failure mode: a run that suddenly produces NaN losses and then recovers is usually the scaler hunting for a workable value. bf16 needs none of this because its exponent range matches fp32, which is why it is the default choice on hardware that supports it.

On an A100 both are available and bf16 is generally the right pick for training. On older cards without bf16 support you are on fp16 and loss scaling, and you should expect to look at the loss curve more carefully.

05Sequence length is the lever people forget

Activation memory scales linearly with sequence length, and attention scales worse than that unless your implementation uses a memory-efficient kernel. Halving the sequence length is often the single largest saving available, and it is free if your data does not actually need the length you configured.

Which is worth checking, because the configured maximum is frequently much longer than the real distribution. If the 95th percentile of your training examples is 900 tokens and you have set a maximum of 4096, you are paying for padding on almost every batch. Two fixes: set the maximum near the real upper end of your data, and pack short examples together into a single sequence so batches contain useful tokens rather than padding. Packing needs an attention mask that stops examples attending across the boundary between them, and most training libraries now support it directly.

Sorting examples by length so that each batch contains similarly sized items reduces padding waste too, at the cost of some correlation between batch composition and content. Bucketing by length and shuffling within buckets is the usual compromise.

06The order to try things in

These techniques compose, but they have very different costs, so the sequence matters.

1. Mixed precision, if you are not already using it. Halves activations and speeds up compute. Effectively no downside on modern hardware with bf16.
2. Fix your sequence length and pad waste. Often the largest single saving, and it makes training faster rather than slower.
3. Reduce micro-batch size with accumulation to hold effective batch size. Cheap down to a point, then increasingly expensive per sample.
4. Gradient checkpointing. Large, reliable saving at a known compute cost of roughly a third.
5. An 8-bit optimizer. Cuts optimizer state substantially. Well tested for fine-tuning, and worth trying before anything more exotic.
6. A parameter-efficient method, if you are not already on one. This changes the shape of the problem rather than trimming it, and for most fine-tuning work it is the right answer anyway.

Notice that the first two make training faster and the next three make it slower. That is the real decision point.

07When to stop optimising and take the bigger card

Every technique here converts memory pressure into time. At some point the conversion rate stops being worth it. A useful way to decide: work out what the run costs at your current settings, then what it would cost on a larger instance with the settings you would rather use. If checkpointing and micro-batches of 1 have made your run take twice as long, a card that is less than twice the price and lets you turn both off is straightforwardly cheaper, and you get a less fragile setup as a bonus.

Fragility deserves weight in that calculation. A run configured right at the memory ceiling fails on the one unusually long example in your dataset, and it fails several hours in. Leaving headroom is not waste, it is what stops you babysitting the job.

If you would rather not work this out from a spec sheet, email contact@vijaycloud.com with your model, method and sequence length. We will tell you which instance fits, including when the smaller one is fine. The 14-day guarantee is there so testing this costs you nothing if you guess wrong.