01The dataset is the ceiling

A fine-tune reproduces the patterns in its examples. It cannot learn a behaviour you never demonstrated, and it will faithfully learn behaviours you demonstrated by accident.

That makes the dataset the binding constraint on outcome quality, and it makes dataset work the highest-leverage part of the project. It is also the part most often rushed, because it is unglamorous and there is a GPU sitting idle.

Our post on why fine-tunes fail in production covers what goes wrong downstream. This one is about getting the input right in the first place.

02How many examples

The honest answer is that it depends on what you are teaching, and the dependency is strong enough that any single number would mislead.

What is reliable is the shape of the relationship. Teaching a format or a style takes remarkably few examples, because the pattern is consistent and shallow. Teaching a task with structure — classification into your categories, extraction of your fields — takes more, roughly in proportion to the number of distinct cases. Teaching domain knowledge takes far more, and often should not be a fine-tune at all: retrieval is usually the better tool for facts.

The common failure is the opposite of scarcity: tens of thousands of near-identical examples that teach one narrow pattern very thoroughly and nothing else.

03What quality means concretely

“High-quality data” is advice nobody can act on. Four specific properties are checkable.

The output is what you actually want. Not approximately right, not right-with-caveats. If your examples hedge, the model hedges. If half your examples answer in one style and half in another, you have taught inconsistency.

The input contains everything needed. An answer that relied on context not present in the prompt teaches the model to produce confident content it cannot ground. This is the single most common way fine-tuning increases hallucination.

Consistency between similar examples. Two near-identical inputs with contradictory outputs teach the model that the task is arbitrary. Contradictions are worse than gaps.

The difficulty distribution matches reality. Data that is uniformly easy teaches a model that performs well on your evaluation and poorly on your hard cases.

04Coverage and diversity

Coverage is about the space of inputs; diversity is about not repeating yourself within it. Both are more valuable than volume.

Enumerate the dimensions your inputs vary along — length, formality, language, domain, whether required information is missing, whether the request is in scope — then check you have examples across each. This exercise reliably reveals whole regions with no coverage, and those regions are where production failures come from.

Two categories teams systematically under-collect. Negative and out-of-scope cases: what should the model do when it cannot or should not answer? If you never demonstrate refusal, it will not refuse. And edge cases at the boundary: the examples that are nearly one category and nearly another are the ones that teach the distinction.

05Deduplication and leakage

Two failures with the same root cause, and both inflate your metrics while degrading your model.

Near-duplicates within the training set effectively raise the weight of one pattern without adding information. Exact-match deduplication is easy and insufficient; the damaging duplicates are the ones that differ by a name or a date. Fuzzy matching or embedding similarity finds those.

Leakage into the evaluation set is the more serious problem. If a near-duplicate of a training example sits in your held-out set, your score measures memorisation. This is how a fine-tune reports a large improvement and delivers none.

Split before you augment, and split on something meaningful rather than randomly. If your data has natural groups — per customer, per document, per session — split on the group. A random split puts sentences from the same document on both sides and quietly leaks.

06Formatting consistency

Unglamorous and consequential. The model learns your formatting exactly, including the parts you did not intend as signal.

Keep the template identical across every example: same field order, same delimiters, same whitespace conventions, same handling of optional fields. Decide once whether an absent field is omitted or present-and-empty, and be consistent, because the model will treat that difference as meaningful.

Then verify that the format you train on is byte-identical to the format you serve. A mismatch between training and inference templates is among the most common causes of a fine-tune performing far worse in production than in evaluation, and it is invisible unless you check the exact strings.

07A build procedure

In order. The first five steps involve no GPU at all.

1. Write down the behaviour you want, concretely enough that two people would grade an output the same way. If you cannot, the dataset will encode the ambiguity.
2. Build the evaluation set first, by hand, from real inputs, including hard and out-of-scope cases. Fifty deliberate items beat five thousand sampled ones.
3. Enumerate coverage dimensions and check for empty regions.
4. Collect or write training examples, validating each against the four quality properties above.
5. Deduplicate, then split on natural groups. In that order.
6. Verify the serving template matches the training template character for character.
7. Train on half, then all, to learn whether volume is your constraint before collecting more.

Whether that dataset then fits on a 16 GB or a 40 GB card is a separate question, covered in choosing between LoRA, QLoRA and a full fine-tune.

If you want a second opinion on a dataset plan before spending GPU hours on it: contact@vijaycloud.com.