Evals: Testing Model and Agent Behavior
A unit test answers whether the code does what it says. An eval answers whether a model, an agent or a prompt behaves well often enough to trust. AIDLC needs both, and most teams only have the first. This page is the practical guide: what to eval, how to build the set, how to grade it, how to gate a release on it, and how to keep the whole thing honest.
What an Eval Is, and Is Not
A test runs deterministic code and expects one answer. An eval runs a system whose output varies, scores each result against a stated standard, and reports how often it meets it. The unit is a case, the output is a score, and the question is "is this version at least as good as the last one?" rather than "is this exactly right?".
That difference is why evals do not replace the tests described in Testing Strategy. The code around a model still gets tests. The model, the agent and the prompt, which change behavior without changing a line of code, get evals. The two sit side by side in the same pipeline, and a release needs both green.
Test
Deterministic. One input, one expected output, pass or fail. Fails the build the moment it breaks. Answers: did the code change behavior?
Eval
Statistical. Many inputs, a graded score per case, compared against a baseline. Fails the build when a blocking case fails or the score regresses. Answers: did the behavior get worse?
The Three Things to Eval
AIDLC has three different behaviors that can regress silently. Each gets its own suite, because each fails in a different way.
Product model behavior
When the feature you ship calls a model, the model's answers are part of the product. Eval the answers a user would see: correct, safe, in the right format, and honest about what it does not know. This is the track described in AIDLC vs the ML Lifecycle.
Coding and review agents
The agents that write, test and review your code are also systems whose behavior shifts with a model upgrade or a new instruction file. Eval them on tasks with known good outcomes, before they touch the real repository.
Prompts and review rules
A prompt, a skill or a review rule is code that nobody compiles. Change one word and behavior moves. Every edit to these files runs the suite for the agent that reads them, exactly as a code change runs its tests.
The pinning and fixture practices behind this are summarized in Engineering Practices; this page is the long form.
Building an Eval Set
The eval set matters more than the tool. Start with a few dozen well-chosen cases, not thousands of generated ones, and mark each case blocking or non-blocking from the first day.
Blocking and non-blocking cases
- Blocking: a failure stops the release. These are the "must never" cases from the problem definition and every acceptance criterion the model is responsible for. They are pass or fail, and one miss is enough.
- Non-blocking: quality cases such as tone, completeness and ranking. They produce a score that is compared with the current version, and a drop opens a review rather than stopping the build.
Where cases come from
- Acceptance criteria. Every criterion the model owns becomes at least one case, which keeps evals traceable to the Analyze output.
- Production failures. Each bad output a user or reviewer reports is sanitized and added the same week, the learn loop applied to behavior.
- Red-team probes. Cases written to break the system on purpose: prompt injection, off-topic requests, missing context and requests that should be refused.
Keep the set in the repository, versioned, with the same review as code. Include the easy cases too: a set made only of hard cases cannot tell you when something ordinary broke.
Grading Methods
Use the cheapest grader that can judge the case, and keep the expensive ones for what the cheap ones cannot see.
| Method | Best for | Watch out for |
|---|---|---|
| Exact match or code check | Structured output, schema validity, a refusal, a tool call with the right arguments, tests passing | Too strict for free text; a correct answer in different words fails |
| Rubric | Free text with clear properties: cites a source, stays in scope, includes the required warning | A vague rubric gives vague scores; write each line so two graders agree |
| Model as judge | Applying a rubric at scale to many outputs | See the cautions below |
| Human grading | Calibrating the judge, high-risk cases, anything subjective | Slow and costly; use a sample, not the whole set |
Cautions for model-as-judge
- Judges tend to prefer longer answers and answers that resemble their own style. Write the rubric to penalize padding.
- Do not let a model grade its own output when a different model family is available.
- Calibrate first: have a person grade a sample, compare, and fix the rubric until the judge and the person agree on most cases.
- Pin the judge model and its prompt like any other dependency. A changed judge changes every score.
- Never use a judge for a blocking case that code can check.
The Eval Gate in CI
The eval suite runs as a gate in the pipeline, alongside the tests. It is the same idea as gates as code: the rule is enforced by the workflow, not by someone remembering.
What triggers the suite
- A change to a prompt, skill, instruction file or review rule.
- A change of model, version or tool configuration.
- A change to the retrieval data or the code that builds the context.
- A schedule, so drift in a hosted model is caught even when nothing of yours changed.
What passes
- Every blocking case passes, with no exceptions.
- No non-blocking score falls below the current production version by more than the agreed tolerance.
- Every model and prompt version is pinned, so the result can be reproduced. See Model and Tool Pinning.
- A failure names the case, the output and the grader's reason, so a person can act on it.
The comparison that matters is against the version in production, not against a perfect score. A model upgrade is a change like a dependency bump: run the suite on the candidate and the current version, compare, then promote.
Evals for Coding Agents
An agent that works inside a repository cannot be judged from one reply. Grade what it did: the changes, the tests, the tool calls and whether it stayed in bounds. Agents like Claude Code, Codex, Copilot and Cursor can all be evaluated this way, because the method needs only a task, a sandbox and a check.
Fixture repositories
A small, frozen repository with a known state: a failing test, a missing endpoint, a seeded defect. Reset it before every run so one run cannot affect the next.
Task suites
A list of tickets with a hidden check for each: the target tests pass, the old tests still pass, no file outside the allowed paths changed, and no test was weakened to get a pass.
Review-rule fixtures
Sanitized pull requests with known issues. Run the review agent on each and track, per rule, how often it flags the real issue and how often it flags something harmless.
Run these in the sandbox and under the budget set in Agents in the Pipeline. Record the transcript of every run, because a pass reached by a strange route is a result worth reading.
Keeping Evals Honest
An eval that always passes protects nothing. Three things quietly erode a suite.
Contamination
If eval cases leak into prompts, examples or training data, the system learns the answers instead of the skill. Keep the cases out of prompts and examples, and keep a held-out set that nobody tunes against.
Drift
Users change what they ask, and hosted models change underneath you. Refresh the set from recent production failures, retire cases that no longer matter, and run the suite on a schedule.
Flaky graders
If the same output scores differently on two runs, the grader is the problem. Run it several times on a fixed sample, fix the rubric, and quarantine a flaky case rather than letting people learn to re-run until green.
A useful self-check is to break something on purpose: weaken a prompt or remove a rule, and confirm the suite goes red. If it does not, the suite is not testing that behavior.
An Eval Case Template
One case as a JSON record. The example is an invented illustration, not a real suite.
{
"id": "support-draft-017",
"target": "product-model",
"blocking": true,
"source": "production-failure",
"criterion": "AC-3",
"input": {
"user_message": "I was charged twice. Refund me now.",
"context": ["billing-policy-v4"]
},
"expected": {
"must": ["routes to a person", "does not promise a refund"],
"must_not": ["states a refund amount"]
},
"grader": { "type": "rubric", "rubric_id": "billing-escalation-v2" },
"added": "2026-10-30",
"owner": "support-platform"
}An Eval Plan Template
One page per model-backed feature or agent, written in Analyze and kept next to the requirements brief.
EVAL PLAN: [feature or agent]
TARGET
Product model / coding agent / prompt or rule (pick one per plan):
MUST NEVER (becomes blocking cases)
-
CASE SOURCES
Criteria covered:
Production failures to seed:
Red-team probes:
GRADERS
Code checks:
Rubrics:
Judge model and pin (if any):
Human sample size and who grades:
GATE
Triggers:
Blocking rule:
Tolerance for non-blocking scores:
Compared against: current production version
HONESTY
Held-out set:
Refresh cadence:
Scheduled run:
OWNER: REVIEWED BY:Eval Anti-Patterns
Vibe checks
Trying five prompts by hand and calling it tested. Nothing is recorded, so nothing can be compared.
One big score
A single average hides a blocking failure. Report blocking cases separately from quality scores.
Unpinned everything
A moving model, prompt and judge make every result unrepeatable and every regression unexplainable.
A judge for everything
Using a model to grade what code could check adds cost and noise to the cases that matter most.
Fixing the eval to pass
Loosening a case when it fails. Change a case only with a reason in review, never to turn a build green.
A set that never grows
If production failures never become cases, the suite measures last quarter's risks.
Frequently Asked Questions
A test runs deterministic code and expects one exact answer, so it passes or fails. An eval runs a system whose output varies, grades many cases against a stated standard, and compares the score with the current version. Tests cover the code around a model; evals cover the model, the agent and the prompt, which change behavior without changing code.
A few dozen well-chosen cases are enough to start. Cover every acceptance criterion the model owns, the must-never cases, a handful of red-team probes and some ordinary easy cases. Then add every production failure to the set within the same week it happens, so the set grows with real risk instead of guesses.
Only with care. Judges can favor longer answers and their own style, so write the rubric to penalize padding, avoid grading a model with itself, calibrate against a human-graded sample, and pin the judge like any dependency. Never use a judge for a blocking case that code can check directly.
Test the Behavior, Not Just the Code
Write the eval plan for your next model-backed feature before the first prompt is written.