Evaluations, Explained From Zero

Every other lever in this series — grounding, choosing an approach, picking a model, tuning cost — is a change you make hoping to improve things. Evaluation is what tells you whether it actually worked. This guide starts from nothing and builds up the discipline that turns AI quality from a vague feeling into a number you can watch and steer.

1 · “It seems to work” isn't enough

Here's how most AI features get judged early on: someone types a few questions, the answers look good, and everyone says “great, ship it.” This is testing by vibes, and it's dangerously unreliable. You tried maybe five inputs; your users will try five thousand, including the awkward, edge-case, and adversarial ones you never imagined. “It seemed fine when I tried it” tells you almost nothing about performance at scale. Worse, when you later change a prompt or swap the model to improve one thing, you have no way to know whether you accidentally broke five others — because you're not measuring. The discipline that fixes this is called evaluation, or “evals,” and it separates a hobby demo from a feature you can confidently improve and rely on. If you can't measure quality, you can't manage it.

2 · An eval is a repeatable test

So what is an evaluation, concretely? It's a repeatable test for your AI feature — the same idea as a test suite in ordinary software. You assemble a fixed collection of example inputs, run them all through your system, and score the results against what a good answer looks like. Because the set is fixed and the scoring is defined, you get a number — say, “82 out of 100 answers were good” — that you can track over time. The power is in the repeatability: run the same eval today and next week and you can see whether quality went up, down, or sideways. That turns quality from a feeling into something you can steer. An eval doesn't have to be fancy to be transformative; even a modest, consistent test beats endless one-off vibe checks, because it gives you the one thing vibes never can — a stable yardstick.

3 · Start with a test set

The foundation of any eval is the test set: the collection of examples you'll run through your system. Building a good one is the highest-value work here, and it's mostly judgement, not code. Start with real inputs — actual or realistic questions your users ask — not just the easy cases you know work. Deliberately include the tricky ones: ambiguous questions, edge cases, things just outside what the system should handle, and inputs that have burned you before. For each example, note what a good answer looks like — sometimes an exact expected answer, sometimes just the key facts it must contain or avoid. And you don't need thousands; a few dozen well-chosen, representative examples covering your real range of use teach you far more than a huge pile of random ones. The test set is your definition of quality made concrete — so choose examples that genuinely reflect what “good” means for your feature.

4 · How do you score an answer?

The trickiest part of evals is scoring: how do you decide if an answer was good? It depends on the task. When there's a clearly correct answer — a classification, a number, a yes/no — you can score by exact match: did it get it right? Simple and automatic. But most interesting AI outputs are open-ended — a summary, an explanation, a written reply — where there's no single right string. There, one option is human rating: a person reads each answer and scores it on the qualities you care about — accuracy, helpfulness, tone. That's the gold standard but slow and expensive. So a popular middle path is the AI judge: you use another model, given a careful rubric, to score the answers — far faster and cheaper than humans, and surprisingly decent, though it needs its own checking against human judgement to trust it. Match the scoring method to the task: exact match where answers are crisp, human or AI judging where quality is a matter of degree.

5 · Catch regressions before users do

Here's where evals earn their keep day to day: catching regressions. AI systems are frustratingly interconnected — you tweak the prompt to fix one annoying behaviour and may silently break three others, because the same prompt drives every answer. Without an eval, you'd never know until users complained. With one, the workflow becomes safe and fast: make your change, re-run the eval, compare the score to before. Went up? The change helped. Went down? You just caught a regression before it shipped — roll it back or dig in. This transforms how you improve an AI feature. Instead of nervously guessing whether a change is a net win, you measure it, the same way software teams run their test suite before every release. It's the difference between improving with confidence and changing things in the dark and hoping. The eval is your safety net for every “improvement.”

6 · Evals evolve with your product

A test set isn't something you build once and freeze — it's a living thing that grows with your product, and this habit compounds. Every time a real failure slips through to users — a bad answer, an embarrassing mistake, a case you didn't anticipate — you don't just fix it; you add that example to your test set. Now that exact failure is checked forever, so it can never quietly come back. Over time, your eval accumulates the hard-won lessons of everything that ever went wrong, becoming a richer and richer definition of quality. This is precisely how mature software test suites grow — each bug spawns a test guarding against its return — and it works just as well for AI. It also keeps your eval tracking what actually matters to your users, since it's built from their real problems. So treat every production failure as a gift: a new, permanent test that makes your system a little harder to break each time.

7 · You can't improve what you don't measure

Zoom out to why this sits at the heart of AI quality. Every other lever — grounding to cut hallucinations, choosing between prompting, RAG, and fine-tuning, picking a model, tuning cost — is a change you make hoping to improve things. But without evaluation you're flying blind: you can't tell whether a change helped, hurt, or did nothing, so you're guessing. Evals give you the number that makes all those decisions real. Should you switch to a cheaper model? Run the eval on both and see if quality holds. Does adding RAG actually help? Measure it. The old management saying fits AI exactly: you can't improve what you don't measure. Evals turn “I hope this is better” into “I can see this is better,” and that shift — from hoping to knowing — is what lets you improve an AI feature steadily and confidently instead of thrashing in the dark.

8 · A simple test you can run this week

9 · Glossary — every term, spelled out

Evaluation (eval)

A repeatable, scored test of an AI feature that produces a number you can track.

Test set

The fixed collection of example inputs, with a notion of what a good answer looks like for each.

Exact match

Scoring by whether the answer equals the known-correct one — for crisp, closed answers.

Human rating

A person scoring open-ended answers on qualities like accuracy and helpfulness — the gold standard.

AI judge

Using another model with a rubric to score answers at scale, checked against human judgement.

Regression

A change that quietly makes some answers worse; evals catch it before users do.