Writing
Blog
Practical notes on GPU infrastructure, model training and the compliance questions that come with hosting data in India. No launch announcements.
-
What to publish while you are still broken
A status page decides whether an outage costs you trust as well as uptime. What to publish, when, and why your uptime percentage is less reassuring than you think.
-
Most incidents are something you shipped
Most production incidents follow a change somebody made. In AI systems the change is often a model version or a config value, which is exactly what nobody is tracking.
-
Alert storms: grouping matters more than thresholds
One failure rarely produces one page. Why tuning thresholds does not fix alert storms, and what grouping by cause looks like on a GPU fleet.
-
What an AI ops agent needs to see, and where that data ends up
An incident-investigating agent is only as good as what it can read. The access list it needs is also the most sensitive export you will ever approve.
-
What to monitor on a GPU box
Memory headroom at peak, utilisation split three ways, thermals and power, and latency percentiles. Four signals explain most incidents, and none is model quality.
-
Serving concurrency: how the KV cache sets your user ceiling
Model weights are a fixed cost. The KV cache grows with every token and every concurrent request, and it is what actually limits how many users you can serve.
-
Quantisation for serving: what you gain, what you give up
Eight-bit roughly halves weight memory, four-bit roughly quarters it, and that memory becomes concurrency. Quality loss is small on average and concentrated in reasoning and rare vocabulary.
-
LoRA, QLoRA or a full fine-tune: how to choose
Full fine-tuning costs about 16 bytes per parameter, LoRA about 2, QLoRA about 0.6. A decision procedure for choosing between them.
-
Data residency and data sovereignty are not the same thing
Residency is where the bytes sit. Sovereignty is whose law reaches them. A provider can hold your data in India and still be reachable by a foreign order.
-
Seven questions to ask a GPU provider before you commit
Tenancy, memory, billing unit, metering, location, support hours and exit terms. Six of the seven should be answerable from a provider published pages, including ours.
-
Where your prompts actually go, and what self-hosting does not fix
Self-hosting removes a third party from the path your prompts travel. It does not remove your obligations as a data fiduciary, and it does not delete your logs.
-
Sizing VRAM: how batch size and context length decide whether your job runs
Weights, gradients and optimizer state you can calculate exactly. Activations depend on your batch size and sequence length. A working method for training and for serving, with the KV cache maths.
-
What a fine-tune actually costs when the GPU is billed by the month
A dedicated GPU billed monthly changes the arithmetic: the marginal cost of another experiment is zero, but idle time is pure waste. The break-even table, with our own prices.
-
The utilisation problem: why most GPU spend is wasted
Surveys keep finding GPU utilisation in the single digits. The bottleneck is rarely the GPU: it is data loading, human review cycles, and work nobody scheduled.
-
Why your first request takes minutes and the second takes milliseconds
A model server takes minutes to answer its first request and milliseconds for the next. Where the time goes, and which parts you can actually remove.
-
Serving agents is not serving chat
An agent turns one request into many model calls with tool waits between them. Concurrency ceilings computed for chat overstate what agent traffic can sustain.
-
Why fine-tunes fail in production, and it is rarely the model
A model that scored well can still fail in front of users. The causes are almost always upstream of the weights: the dataset, the evaluation, the input distribution and the latency budget.
-
The line items people miss on a GPU bill
The hourly rate is the part everyone checks. Storage, egress, idle time, snapshots and support are where two similar quotes stop resembling each other.
-
Logging AI workloads without creating a liability
Prompt logs are the best debugging artefact you can have and the easiest personal data store to create by accident. Separate telemetry from content and set retention first.
-
What belongs in a data processing agreement with a GPU provider
Most DPAs are SaaS templates. GPU compute inverts the assumptions: you hold root, the provider cannot see your data, and four clauses that matter are usually missing.
-
Evaluating a fine-tune before you ship it
Training loss tells you the model fit your data, not that it got better at your task. What to build before you train, and what to check before you ship.
-
The specs that throttle training and are not the GPU
A GPU at thirty per cent during training is almost never a GPU problem. Storage, host RAM and vCPU each starve it in a different way, with a different signature.
-
Reading a GPU spec sheet: which numbers actually matter
Capacity decides whether a job runs, bandwidth usually decides how fast, and TFLOPS is meaningless without a precision. The four numbers that matter, in order.
-
Fitting a bigger training run on the GPU you already have
Out of memory does not always mean you need a bigger card. Gradient accumulation, checkpointing and mixed precision each buy back VRAM at a different cost.
-
When one GPU stops being enough
Three different problems get answered with the same request for more hardware. Only a memory ceiling is a hard reason, and even then software levers usually move it.
-
Building a dataset worth the GPU hours
The dataset sets the ceiling; compute only decides how fast you reach it. Coverage, deduplication and formatting consistency matter more than volume.
-
Choosing a serving stack
The software in front of your model sets your concurrency ceiling. Continuous batching, paged cache, quantisation support and metrics are what differentiate.
-
Batching and streaming without hurting latency
Batching raises throughput and can add waiting per request. Prefill and decode have opposite bottlenecks, so a setting that helps one hurts the other.
-
Keeping training data in India: what the DPDP Act actually asks of you
The DPDP Act does not ban cross-border transfer outright. Here is what it does require, and where hosting location genuinely helps.
-
What “dedicated GPU” actually means, and how to check
Dedicated, time-sliced, MIG and oversubscribed all get marketed the same way. Here is the difference, and how to verify what you are on.
Vijay, trading as VijayCloud · Mumbai, India