01What happens between start and first token

A model server that takes three minutes to become useful is not doing three minutes of thinking. It is doing a sequence of distinct things, most of which are data movement, and knowing which is which tells you where the time is available to recover.

The middle two rows usually dominate, which is the useful thing to internalise: cold start is a bandwidth problem. A 14B model in bf16 is roughly 28 GB of weights that has to be read off a disk and pushed across PCIe before any arithmetic happens. On local NVMe at several gigabytes per second that is a matter of seconds. From network storage at a few hundred megabytes per second it is minutes. Same model, same GPU, an order of magnitude difference in startup, entirely decided by where the file lives.

02Weight loading is a storage problem

Because loading is bandwidth-bound, the levers are all about reducing bytes or moving them faster.

Where the weights live. Local NVMe beats network-attached storage substantially, and pulling from object storage over the network is worse again. If your serving instance downloads weights from a remote registry at every start, that download is your cold start, and caching them on local disk removes most of it.

The file format. Formats designed for memory-mapped loading let the OS map the file into the address space and page it in, rather than reading, deserialising and copying. That avoids a full extra copy in host memory. It also avoids the security problem of formats that execute code on deserialisation, which is a separate and good reason to prefer them.

How many bytes there are. A 4-bit quantised model is roughly a quarter the size of the same model in bf16, so it loads roughly four times faster. If you were considering quantisation for memory or throughput reasons, faster startup comes along with it.

Sharding across GPUs. On a multi-GPU instance, shards can load in parallel, so wall-clock loading time falls with the number of devices reading at once, provided the storage can supply them all. If your storage saturates at 500 MB/s, four GPUs reading in parallel do not help.

03Compilation, autotuning and the first forward pass

Modern serving stacks do work on first use that they do not repeat. Attention kernels get selected or autotuned for your specific shapes. Graph compilers trace and optimise the model. Some quantisation paths build lookup tables at load. All of this is genuine compute and it can add tens of seconds.

Most of it is cacheable, and often not cached by default. Compilation caches usually live in a directory that a container throws away on restart, which is why a stack that starts in 40 seconds on your workstation takes three minutes in production: identical work, done from scratch every time. Mounting the cache directory on persistent storage is a small change with a large effect.

The part that is not cacheable is shape-dependent. If the compiler specialises on batch size and sequence length, a request with a new shape pays compilation the first time it is seen. This produces the confusing situation where the service has been up for an hour, feels fast, and then one unusual request takes ten seconds. Stacks that pad to bucketed shapes trade a little waste for not having this happen.

Which is the argument for an explicit warmup step. After loading, send a handful of synthetic requests covering the shapes you expect before you mark the instance healthy and let real traffic in. Otherwise your first real user pays for the warmup, and your health check reported ready while the service was not.

04Page cache is why the second start is fast

Restart a server that has just been running and it often comes up dramatically faster. Nothing was optimised. The weight file is still in the operating system page cache in host RAM, so the read never touches the disk.

This explains a lot of confusing measurements. It is why cold start times in your testing look better than in production, because you tested by restarting repeatedly. It is why the first start after a reboot is slow. And it is why a machine with enough RAM to hold the weight files behaves differently from one without, even with the same GPU and the same disk.

You can use this deliberately: reading the weight file once at boot, before the server needs it, populates the cache so the real load is served from memory. It is a crude trick and it works. What it needs is host RAM comfortably larger than your weights, which is worth checking against your instance spec, since it is a number people skip when the GPU is the thing they are shopping for.

The corollary is that any benchmark of cold start needs to say whether the cache was cold. A figure measured with warm cache is not a cold start figure, and the two can differ by a factor of ten.

05Scale-to-zero has a price, and this is it

Serverless GPU platforms bill only while you are running, which is genuinely attractive for sporadic workloads. The mechanism is that they stop your instance when idle, and every stop means the next request pays a cold start. Everything above is the cost of that model.

So the trade is legible. Sporadic, latency-tolerant work such as batch jobs, internal tools or a demo used a few times a day: scale-to-zero is a good fit and the idle savings are real. Anything user-facing where a request might arrive at any time: a first user occasionally waiting minutes is usually unacceptable, and the mitigations all involve keeping something warm, which is the thing you were trying to avoid paying for.

Providers offer middle grounds, and they cost money because they must: keeping a snapshot of GPU memory ready, keeping the container warm without the GPU attached, or maintaining a floor of one always-on replica. That last one is worth pricing out honestly, because a floor of one replica plus burst capacity is often close in cost to just running the instance, at which point the simpler architecture wins.

A dedicated instance sits at the other end of this: it stays up, so there is exactly one cold start, at deploy time. You pay for hours nobody is using, and you never explain to a user why the first request of the morning took two minutes. Which is right depends entirely on your traffic shape, and it is worth working out rather than assuming.

06What to measure

Four numbers, kept separate, because collapsing them into one hides where the time is.

• Time to weights loaded. From process start to weights resident in GPU memory. Your storage bandwidth ceiling, and the biggest single lever.
• Time to ready. Loaded plus compilation, allocation and warmup. This is what your health check should actually gate on.
• First-token latency on the first real request. If this is much worse than the steady-state figure, your warmup is incomplete.
• Steady-state first-token latency. The number you quote to users, and the only one that should appear in an SLA.

Measure the first two from genuinely cold: fresh instance, cold page cache, empty compilation cache. Anything else flatters the result. And if you are comparing serving stacks, compare startup as well as throughput, because the fastest stack at steady state is not always the one that gets there quickest, and which matters depends on how often you deploy.

The wider point is that most of what makes a cold start slow is decided by the instance rather than the model: storage speed, host RAM, whether caches survive a restart. Those are choices made when you pick the machine. If you want to talk through the startup profile of a specific model on specific hardware, email contact@vijaycloud.com and we will go through the numbers with you.