01Loss is not the metric you care about

Training loss measures how well the model predicts the next token in your training data. Validation loss measures the same thing on data it has not seen. Both are useful for detecting that something has gone wrong, and neither answers the question you actually have, which is whether the model does your task better than what you had before.

The gap is easy to demonstrate. A model can reach excellent validation loss on a dataset of support tickets by learning your house style, your greeting, and the average length of a reply, while getting no better at the part that matters, which is diagnosing the customer problem correctly. Loss rewards fluency in your distribution. It does not distinguish confident and wrong from confident and right, because both are single tokens with a probability attached.

So use loss for what it is good at. A validation loss that rises while training loss falls means you are overfitting and should stop. A training loss that will not move means your learning rate or data pipeline is broken. Beyond those diagnostics, do not use it to decide whether to ship.

02Build the evaluation set before you train

This is the single most useful discipline in the whole process, and it is almost always skipped because it feels like a detour when you are keen to start a run.

The reason to do it first is not process hygiene, it is bias. If you train the model and then sit down to write test cases, you will write test cases informed by what you have already seen the model do. You will unconsciously pick examples in the shape of your training data, because that is the shape now in your head. The set you produce will be one your model passes, and you will have learned nothing.

Write it from the task instead. What does a correct answer look like? What are the cases you know are hard? What has gone wrong with the current system? Those questions have answers that do not depend on any model, which is exactly what makes the resulting set worth something.

Size matters less than people fear. A hundred cases you have thought about carefully will tell you more than two thousand sampled at random, because random sampling of real traffic gives you a set dominated by the easy majority. You want the set weighted toward the decisions that are actually difficult. Twenty to fifty cases is enough to catch gross regressions; a couple of hundred is enough to compare two candidate models with some confidence.

And keep it out of the training data. If you built the eval set by pulling examples from the same export you trained on, deduplicate properly, including near-duplicates, or your numbers are meaningless. This is the most common way evaluation gets quietly invalidated.

03Three kinds of case, all three needed

An evaluation set that is only representative traffic will miss the ways fine-tuning fails. Keep three groups and score them separately, because a single blended number hides the thing you most need to see.

The regression group is the one teams leave out, and it is the one that catches the expensive surprises. Fine-tuning on a narrow distribution reliably degrades performance outside it. If you train exclusively on short factual answers, the model gets worse at long-form explanation even though you never asked it to. If every training example is in English, multilingual ability degrades. If your data has no examples of the model declining a request, it becomes less willing to decline.

Twenty cases sampled from general capability, kept fixed and run every time, will tell you whether the specialisation cost you something you were relying on.

04Compare head to head instead of scoring in isolation

Asking a grader to give an answer a score out of ten produces numbers that look precise and are not. The scale drifts between cases, between graders, and between runs of the same grader. A 7.4 average tells you almost nothing on its own.

Pairwise comparison is far more stable. Take the same input, generate an answer from the base model and one from the fine-tune, present them without saying which is which, and ask which is better and why. Human judges do this consistently. So, largely, do model graders. What you get out is a win rate, which is directly interpretable: if the fine-tune wins 62 percent of the time, ties 20 percent, and loses 18 percent, you know what you have.

Two details make it trustworthy. Randomise which answer appears first, because both human and model graders have a position bias. And keep the losses, not just the count of them: the cases where the base model won are the most informative output of the whole exercise, because they tell you what your training data is missing.

If you are using a model as the grader, spot-check it against your own judgement on a couple of dozen cases before you trust the aggregate. Model graders have consistent preferences that may not match yours, notably for longer and more hedged answers. Once you know the grader agrees with you often enough, it is a reasonable way to scale.

05The four things fine-tuning quietly breaks

Beyond task quality, check these specifically. Each has a habit of degrading without showing up in a loss curve or an average score.

Instruction following. Base instruction-tuned models follow directions in the prompt. Fine-tuning on input-output pairs with no instructions can teach the model to ignore them, because in every example it saw, the instruction was absent and the output followed anyway. If your production prompts include instructions, test that they still take effect.

Output format. If you need valid JSON, test for valid JSON on every eval case and count the failures. Format compliance often improves with fine-tuning and sometimes gets worse in a specific way, such as correct structure with a stray explanatory sentence before it. Either is easy to detect automatically and easy to miss by eye.

Refusal behaviour. A dataset with no examples of declining anything shifts the model toward answering everything, including things it should not or cannot. If your application has boundaries, put cases for them in the eval set.

Length and verbosity. Models adopt the length distribution of their training data quite strongly. If your examples are all terse, expect terse answers even where detail was wanted. Track median output length as a metric; a large shift is a signal worth looking at even when quality scores hold.

06What to do before and after you ship

Before: run the full set against the base model and the candidate, keep the raw outputs, and write down the win rate per group. If the fine-tune does not clearly beat the base model on representative cases, do not ship it, however good the loss curve looked. If it wins on representative cases but loses on regression cases, decide explicitly whether that trade is acceptable rather than discovering it later.

After: keep the eval set and the outputs. Its second job is regression testing. When you retrain on more data in two months, the only way to know the new version is better than the current one is to run both through the same set, and the only way to do that is to have kept it. Version it alongside the model.

Then sample real traffic. An eval set built before launch reflects what you expected users to do, and users will do things you did not expect. Reviewing a small number of real interactions each week is how the adversarial group grows into something that reflects reality. That is also where the next round of training data comes from.

The evaluation work is not GPU-heavy, but it does want the model kept resident so you can iterate without reloading weights each time. A dedicated instance is well suited to this, because comparison runs on shared or time-sliced hardware give you inconsistent latency and make it harder to keep the run stable. If you want a second opinion on an eval plan before you commit a month of GPU time, email contact@vijaycloud.com.