Public benchmarks tell you very little about whether a language model will work for your use case. They are useful for comparing models on a small number of general capabilities, in controlled conditions, on tasks that are rarely the tasks you actually need done. The gap between benchmark performance and production behaviour is where most AI programmes lose their credibility, and it is where evaluation has to earn its keep.
This piece is about building evaluation harnesses that actually predict production behaviour. It is written for teams who have shipped at least one LLM system and are now trying to ship the second one with less pain.
## Start from the decision, not the response
Evaluation should measure the thing that has consequences. For a chat assistant, the thing that has consequences is usually not the fluency of the response. It is whether the response led the user to the right action, or contained a claim the user could rely on, or safely refused when the request was out of scope. Evaluations that grade responses in isolation, without reference to the downstream action or the ground truth the response should have expressed, systematically overestimate quality.
The first evaluation to build is therefore not a scoring rubric for text quality. It is a set of end to end scenarios that describe what the user was trying to accomplish, what the correct outcome looks like, and how a graded response contributes to that outcome. This is more work than a rubric. It is also the only evaluation whose results correlate with what your users will experience.
## Use three layers of evaluation
Mature harnesses run at three layers, and each layer catches different failure modes.
**Unit level.** Small, targeted tests for specific behaviours: refusal in defined categories, correct extraction of named entities, correct handling of ambiguous inputs, correct citation of retrieved sources. These run on every change and are cheap.
**Scenario level.** End to end scenarios of the kind described above, with graded outcomes. These run on every material change and on a regular cadence, and they are the primary signal for promotion decisions.
**Population level.** Sampled evaluation over real production traffic, with human review of a stratified sample and automated review of the rest. These run continuously in production and are the primary signal for drift.
Missing any of the three layers leaves a class of failure invisible. Unit only catches known bugs and misses emergent behaviour. Scenario only catches designed cases and misses the long tail. Population only catches what production reveals and misses everything before launch.
## Grading: humans, models, and where each is honest
The grading problem is where most harnesses go wrong. Human graders are expensive and slow but honest about what they cannot judge. Model graders are cheap and fast but confidently wrong in predictable ways. The right posture is to use each where it is honest.
Use human graders for outcomes that require judgement about tone, safety, appropriateness or subject matter expertise, and for the population level sample that ties the whole harness back to reality. Use model graders for outcomes that can be reduced to checkable properties: presence of a required element, absence of a forbidden element, correct format, correct citation of a source that is available to the grader. Do not use model graders as a general quality signal, because they will happily grade fluent nonsense as high quality when the ground truth is missing.
## Version everything, correlate everything
An evaluation result that cannot be tied to the exact model version, prompt version, retrieval configuration and grading version in force at the time is a number without a home. Store all four with every result. When performance changes, the first question is always which of the four changed, and if the answer is not obvious from the record, the harness is not yet doing its job.
The second question is always whether the change in evaluation performance correlates with a change in production behaviour. If it does not, the evaluation set is not representative and needs to be extended. If it does, the harness has earned another quarter of trust.
## Refresh the evaluation set on purpose
Evaluation sets go stale. New user behaviours appear. New content is added to the retrieved corpora. New failure modes emerge from adversarial users and from the model provider's own updates. A harness whose test sets have not been extended in six months is a harness that is measuring last year's system.
The discipline that works is to allocate a small, predictable amount of effort each sprint to extending the evaluation set from production incidents, from user feedback, and from adversarial probing. Treat it as maintenance, not as a project. Programmes that try to refresh the set only when something breaks refresh it only after users have already been affected.
## What good looks like
A team running a mature LLM system in production can, on any given day, answer four questions in under an hour. What is the current performance of the system against the evaluation set that matters. What changed since the last measurement. What does production sampling say about whether the evaluation set still represents reality. And what is the plan to extend the evaluation set this sprint.
If your team cannot answer those four questions, the evaluation harness is not yet load bearing. The leaderboard number for the underlying model is, at that point, the least interesting number in the room.
