// ARTIFICIAL INTELLIGENCE
Measuring hallucination: how to build an eval set
7 min readnijitech
“The model makes things up sometimes” is an observation, not a measurement. You cannot claim to have fixed a problem you cannot measure. A practical way to build an evaluation set.
Full post
One of the most common sentences in AI projects is this: “the model makes things up sometimes”. That is an observation, not a measurement. It does not say how often it makes things up, on which kinds of question, or whether yesterday’s change made it better or worse.
And you cannot claim to have fixed a problem you cannot measure. An evaluation set — an eval for short — closes that gap: a test suite of known questions and expected answers, run after every change.
1. Examples must be real, not representative
The first mistake in an eval set is usually made here: the questions get invented at a desk. What belongs in the set is the input the system actually meets — real questions users wrote, real documents that arrived, real edge cases that happened.
What a good set contains
- Common, normal cases — most of the volume is here
- Known edge cases: missing field, unexpected format, very long input
- Examples known to have failed before — every fixed bug joins the set
- Questions whose answer should be “I do not know”
That last item is the most often skipped and the most useful. A model knowing when to stay quiet matters as much as it giving the right answer.
2. The expected answer has to be written down
For each example the correct answer is written in advance. In systems producing free text this looks hard, but in most cases what can be defined is not the whole answer but the elements it must and must not contain: “must give this figure”, “must cite this source”, “must not make a pricing commitment”.
3. Scoring should not collapse to one number
A single success percentage is comforting, and misleading. Ninety per cent does not tell you which kinds of question the remaining ten per cent gathered in. Split the set into categories and score each separately, and where the drop happened becomes visible.
Hallucination needs a measure of its own: was information produced without a source? That is not the same as “the answer was wrong”. A wrong answer is an accuracy problem; a sourceless answer is a trust problem.
4. The set must run after every change
The real value of an eval set appears not on the first run but on the second. When the model changes, the prompt is updated or the context is trimmed, the set runs again and tells you which category regressed.
This is the same thing as regression testing in software, with one difference: in software the same input gives the same output, and with a model it may not. So the threshold should look at the average of repeated runs, not a single one.
Where to start
Doable in the first week
- Collect thirty real inputs from the past month
- For each, write what the expected answer must and must not contain
- Split the examples into three or four categories
- Run it once; that number is neither good nor bad, it is your starting point
Thirty examples looks small, but it is infinitely better than zero. The real gain starts on the second run.
Products mentioned in this post
From the glossary: Hallucination · Evaluation set (eval)