An eval looks like a measurement. Underneath, it's a definition of quality.
May 2026
Suppose you are handed a stack of student essays and asked to grade them. You start with a rubric someone gave you: a point for a clear thesis, points for evidence, a point for a conclusion that follows.
The first few essays sort themselves. Then the rubric starts to strain. One essay hits every item and says nothing; it is correct and empty. Another answers a slightly different question than the one assigned, and answers it well. A third makes an argument the rubric never anticipated, one that is better than the model answer. You find yourself deciding, case by case, what a good essay actually is. The rubric was not measuring quality. It was standing in for a definition of quality you are now writing as you go.
This is the shape of an eval, the thing teams build to decide whether an AI system's output is any good. An eval looks like a measurement, and it reports a number that behaves like one. Underneath, most of it is a definition of quality that someone had to write, and the definition is a design object.
An eval has three parts, and none of them are as settled as the number at the end suggests. There is a set of examples to run. There is a definition of what counts as a good response. And there is a judge, a person or a model standing in for one, applying that definition. The middle part is where the work is. Good does not sit in the data waiting to be measured. Someone decides what it means, writes it down clearly enough that a stranger or a model could apply it the same way twice, and accounts for the cases where the honest answer is that the question cannot be answered.
Three of those decisions are places a designer shapes what the eval measures, and each one can be seen more clearly by returning to the essays.
What counts as an answer at all
Go back to the essay that answered a different question. Under a rubric that only knows how to score answers to the assigned question, it has no valid place. The grader either forces it onto the scale or throws it out.
A research agent built to answer questions about call transcripts had the same gap. Early on it always answered. Asked what objection a customer raised on a call, it produced one, whether or not the call contained an objection. The answers read as fluent and specific, and many were wrong. The eval had no slot for "the call does not say," so every incentive, for the model and for the people grading it, pushed toward producing an answer. The system was being graded on confidence and credited with correctness.
Making abstention a valid response changed what good meant. "The call does not contain this" became a correct output, in the product and in the eval that judged it. Allowing the system to decline did more for the hallucination rate than most of the prompt work around it. A good answer was no longer a filled field. It could be a true statement that the field is empty.
The design decision underneath: an interface that has a shape only for answers will collect answers, real or not. The shape for "nothing here" has to exist before the eval can reward honesty.
What good is, written down
Return to the grader deciding, essay by essay, what quality is. Those decisions are not noise on the way to a score. They are the definition forming.
The same thing happens when domain experts review an AI system's output and mark the poor ones. The marks matter less than the reasons. This answer invented a detail. This one answered a near-miss of the real question. This one was true and useless. Collected and organized, those reasons become guidelines for what a good answer looks like. The guidelines become the rubric. The rubric becomes the eval, and can seed the prompt tuning that follows.
The path runs from a messy human reaction to a written standard to a repeatable measurement, and every step is design work. It is the work of turning fuzzy judgment into a specification precise enough to apply the same way twice, which is the same skill as writing acceptance criteria a whole team can agree on. The output happens to be an eval rather than a feature.
For a designer, this means the review step is not data collection to hand off. The categories the experts reach for when they explain a bad answer are the raw material of the definition of good.
How the judgment gets collected
The last decision is the one most easily missed, because it hides in the tool rather than the rubric.
The interface an expert uses to review outputs is itself an instrument, and its design decides what signal comes out. Consider a review tool where approving an answer takes one click and overriding it requires typing a reason. The intent is reasonable; a rejection should carry an explanation. But the arrangement makes agreeing cheap and disagreeing expensive, and the override rate falls, not because the answers improved but because correcting them became tiring. A low override rate then reads as evidence the system works, and the eval has reported a comforting result for the wrong reason.
Call this asymmetric review cost: whenever one response is easier to record than another, the collected data tilts toward the easy one. It has nothing to do with the model's quality and everything to do with the friction in the tool. The correction is to make disagreement at least as cheap as agreement, capture the reason in a way that takes a tap rather than a paragraph, and treat a suspiciously clean approval rate as a symptom rather than a result.
The three decisions, and reading them backward
Compressed, the designer's surface in an eval is three choices upstream of the number: the set of outputs that count as valid, the definition of what counts as good, and the method by which human judgment is collected. Expanded, they are three questions worth answering before trusting a score.
Can the system return "I don't know," and does the eval count that as correct when it is?
Is the definition of good written down well enough that two reviewers apply it the same way?
Does the review interface make every response equally cheap to record?
When an eval's number stops matching reality, the same three decisions explain most of the failures.
If the system produces confident, fluent errors, the valid-output set may have no room for abstention, so the model is rewarded for guessing over declining.
If two reviewers score the same output differently, the definition of good is not yet shared, which is a problem in how the rubric is written and read rather than in the model.
If the override or failure rate looks reassuringly low, asymmetric review cost may be suppressing disagreement before it is recorded.
If the metric improves while users complain, the examples being evaluated may cover a narrow slice of real use, and the eval is precise about the wrong thing.
What this leaves
A better model moves a system faster toward whatever the eval points at. If the definition of good is wrong, it arrives faster at the wrong place. The definition is built, not found, and building it is design work whether or not a designer is in the room.
Sources and further reading
Practitioner source: the author's work building AI Analyst, a research agent that answers questions over call transcripts.Where the abstention, expert-review and review-interface findings described here come from.
