Deterministic harnesses vs LLM-as-judge: when each wins
A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended quality — at the cost of documented biases that must be measured before the judge's verdicts mean anything.
Why this wins its question: Existing writeups advocate for one side; this one gives the routing rule — computable outcome, use the harness; judgment call, use a calibrated judge; never let the judge grade what a diff can — with each cell of the table sourced.
Key takeaways
- Computable outcome means harness; judgment call means calibrated judge; never let a judge grade what a diff can.
- Judge agreement with humans tops out around human-to-human agreement — that is the ceiling, not a defect.
- Deterministic gates decide ship/no-ship; judge scores ride along as telemetry until calibrated on your domain.
- Harness details are part of the measurement — formatting alone has moved scores about 5 points.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Deterministic grading scales to real agent tasks: tau-bench compares the final database state against an annotated goal state, so the verdict is reproducible and independent of transcript plausibility.
LLM judges reach over 80% agreement with human preferences on open-ended chat, while carrying position, verbosity and self-enhancement biases and limited reasoning on hard grading.
Mature eval frameworks treat both as templates: OpenAI's evals ships basic (mechanical) evals and model-graded evals side by side, selected per task.
Mechanical grading is not automatically objective: formatting inconsistencies alone shifted MMLU scores by about 5%, so harness details are part of the measurement.
The routing rule
If the outcome can be computed — a test passes, a database state matches a goal, a file compiles — grade it deterministically. Reach for an LLM judge only when the property you care about is a judgment call (tone, helpfulness, reasoning quality) that no diff can express. Never let a judge grade something a diff could have graded: you would be adding noise and bias to a solved measurement.
Head to head
| Dimension | Deterministic harness | LLM-as-judge |
|---|---|---|
| Reproducibility | Same input, same verdict (claim c1) | Verdicts vary with prompt, position, judge model (claim c2) |
| Coverage | Only computable outcomes | Open-ended quality, style, safety judgment |
| Known biases | Harness bugs, format sensitivity (claim c4) | Position, verbosity, self-enhancement (claim c2) |
| Cost per verdict | Near zero after fixtures exist | One strong-model call per item |
| Gaming surface | Overfit to fixtures | Please-the-judge outputs, verbosity inflation |
| Failure visibility | Loud (test crashes, diff mismatch) | Silent (plausible score, wrong reason) |
Where each wins
The harness wins for tool-using agents acting on systems of record: grade the final state, not the conversation (claim c1). It also wins for anything regression-gated in CI, where verdict stability is the point.
The judge wins for open-ended output where human preference is the ground truth and you can afford calibration: measured agreement with humans above 80% is achievable (claim c2), which is human-level — and also the ceiling.
Both lose when treated as free. Harness details are part of the measurement (claim c4); a judge is an instrument that needs a calibration sheet before its verdicts count.
The production pattern
Layer them, as mature frameworks already do (claim c3): deterministic gates decide ship/no-ship; judge scores ride along as trend telemetry until their agreement with a human-audited sample has been measured on your own domain. Promote a judge to gatekeeper only after that calibration — and re-falsify it when the judge model changes.