LLM evaluation: how to know whether your AI feature is actually getting better

Build the golden dataset first

Collect real inputs from actual usage, not invented examples, and record the output you would consider correct for each. Fifty to two hundred cases is enough to be useful, and the composition matters more than the count: include the ambiguous inputs, the very long ones, the malicious ones and the ones that should be refused.

Getting agreement on the correct output is often the hardest part, and that difficulty is itself informative. If your own team cannot agree what a good answer looks like, the model has no chance, and the specification needs work before the engineering does.

Choose metrics that fit the task

For extraction and classification, exact or field-level match against expected output is objective and cheap. For structured generation, schema validity plus field accuracy separates "parseable" from "correct", which are genuinely different failure modes with different fixes.

For open-ended generation there is no single correct string, so score against a rubric: is it factually supported by the provided context, does it follow the format, does it stay in scope, does it decline appropriately when it should. Rubric scoring is more work to set up and far more informative than a similarity number.

Evaluate retrieval separately from generation

In any RAG system, measure whether the correct passage was retrieved before measuring whether the answer was good. Combining them hides the most common failure, which is that the answer was poor because the right context never arrived.

Standard retrieval measures apply: whether the relevant document appeared in the top results at all, and how highly it ranked. Teams that instrument this separately fix problems in days rather than cycling through prompt rewrites for weeks.

An end-to-end score tells you something is wrong. Layered scores tell you what to fix.

Using a model as a judge, carefully

Having a strong model grade outputs against a rubric scales evaluation far beyond what human review can cover, and it works well enough to be genuinely useful. It also has real biases: a tendency to prefer longer answers, and leniency toward outputs that merely sound confident.

Use it as a screening layer rather than an oracle. Calibrate it against a human-scored subset, check that agreement periodically, and keep humans reviewing a sample. Model-graded evaluation is a force multiplier for human judgement, not a replacement for it.

Make it continuous

Wire evaluation into CI so any change to a prompt, model version, retrieval configuration or chunking strategy triggers a run against the golden set, with results recorded over time. A quality trend line is worth more than any individual score because it shows direction.

Then close the loop with production monitoring: sample real traffic, watch refusal and error rates, and feed interesting failures back into the golden set so it grows toward the problems you actually have. Our AI observability guide covers the runtime half, and testing AI features covers the QA framing.

Frequently asked questions

How large does a golden dataset need to be?

Fifty to two hundred well-chosen cases covering your real distribution of inputs, including edge cases, is enough for meaningful signal. Coverage of failure modes matters far more than raw size.

Can I use an LLM to evaluate another LLM?

Yes, with calibration. Model-graded evaluation scales well but carries known biases, so validate it against human scoring on a subset and keep sampling manually.

How is this different from normal software testing?

The outputs are not deterministic, so you score against criteria rather than assert equality, and you track distributions and trends rather than a binary pass or fail.

Shipping AI features without a way to tell whether changes help? A free 30-minute consultation will get you a practical evaluation plan for your specific feature.