Your first agent evals: from zero to a release gate in two weeks
The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one, baseline with repeated trials and error bars, and wire the result into CI as a gate. Two weeks of part-time work; the output is the suite every other reliability practice hangs from.
Why this wins its question: A dated, ordered bootstrap plan with a stop condition (a working CI gate) — the existing literature explains why evals matter or how one framework works, not the shortest path from nothing to a gate.
Key takeaways
- Twenty log-harvested tasks, one grader style, one adopted framework, one CI job — resist enlargement until the gate exists.
- Define done as end state; a task whose outcome cannot be asserted gets replaced, not judge-graded.
- Baseline with 4-8 trials per task and a standard error, or noise will read as regressions.
- Invented tasks stay green forever and predict nothing — logs or it does not count.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Start simple is vendor-stated best practice: add agent complexity only when it demonstrably improves outcomes, with extensive sandboxed testing and guardrails before autonomy grows.
Framework adoption beats framework building on day one: open eval frameworks ship reusable templates for basic and model-graded evals plus private-eval support, and Inspect adds sandboxed execution, agent primitives and 200+ prebuilt evaluations.
Grade end state, not narration: comparing the final state of the acted-on system against a goal state is how agent benchmarks catch wrong actions hidden behind plausible transcripts.
Baselines need repeated trials and uncertainty: consistency across k runs diverges sharply from single-run success, and comparisons without standard errors mistake noise for findings.
The shape of the problem
Teams without evals do not lack conviction — they lack a first suite small enough to finish. The answer is twenty tasks, one grader style, one adopted framework, one CI job. Resist every enlargement until the gate exists.
Week 1: tasks and graders
1. Harvest 20 real tasks from logs. Not invented scenarios: pull the last 20 things users actually asked the agent to do, keep the inputs verbatim, strip secrets. Ten routine, five hard, five that previously failed (those five are your future regression tests). 2. Define done as end state. For each task, write down what must be true of the world afterwards — records changed, files present, API responses matching — never what the transcript should say (claim c3). If a task's outcome cannot be stated as a checkable condition, replace the task; do not reach for a judge yet. 3. Write the graders as plain assertions. State diffs, exact checks, executable verification. Boring by design: a grader you can falsify in one minute is worth ten clever ones.
Week 2: framework, baseline, gate
4. Adopt, do not build (claim c2). Pick an existing framework — the choice matters less than not writing your own runner. Fixtures and graders encode your domain; everything else is commodity. 5. Baseline with repeated trials. Run each task 4-8 times against the current agent; record per-task pass rates and a standard error for the suite (claim c4). This number panel — not any single score — is what "the agent got worse" will mean from now on. 6. Wire the CI gate. The suite runs on every prompt change, tool change and model change; per-task thresholds on the critical five, suite-level trend for the rest. Keep the agent's autonomy matched to what the gate proves (claim c1).
After the gate exists
- Falsify it: plant a known regression, watch red, restore (see
falsify-your-first-guardian).
- Feed it: every new incident adds one task with a grader (see
from-incident-to-check).
- Grow deliberately: more tasks on the paths users actually hit, a
calibrated judge only where no assertion can reach.
Two traps
- The invented-task suite. Twenty scenarios nobody asked for,
green forever, predictive of nothing. Logs or it does not count.
- The framework project. Three sprints into a bespoke runner, no
gate yet. The runner was never the hard part; your fixtures are.