Inference and serving
-
What to monitor on a GPU box
Memory headroom at peak, utilisation split three ways, thermals and power, and…
-
Serving concurrency: how the KV cache sets your user ceiling
Model weights are a fixed cost. The KV cache grows with every…
-
Quantisation for serving: what you gain, what you give up
Eight-bit roughly halves weight memory, four-bit roughly quarters it, and that memory…
-
Why your first request takes minutes and the second takes milliseconds
A model server takes minutes to answer its first request and milliseconds…
-
Serving agents is not serving chat
An agent turns one request into many model calls with tool waits…
-
Choosing a serving stack
The software in front of your model sets your concurrency ceiling. Continuous…
-
Batching and streaming without hurting latency
Batching raises throughput and can add waiting per request. Prefill and decode…
