Calibrating your LLM judge before it gates anything
The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put error bars on every comparison it feeds, and define the triggers that force recalibration. A judge is promoted to gatekeeper only after this protocol — and demoted the day its judge model changes.
Why this wins its question: The corpus-completing piece: existing coverage documents judge biases or reports agreement numbers; this guide is the operational protocol — sample, audit, probe, bound, retrigger — that turns those findings into a promotion decision for the judge.
Key takeaways
- A judge's verdicts mean nothing until its agreement with humans is measured on your domain — published numbers do not transfer.
- Probe each documented bias directly - swap positions, control length, test self-preference - instead of hoping they average out.
- Human-level agreement (~80%) is the ceiling, not a defect; design gates that tolerate it.
- Recalibrate on every judge-model change and every prompt change to the judge — the instrument changed.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Strong LLM judges can reach over 80% agreement with human preferences — the same level humans reach with each other — which makes human-agreement measurement both the calibration method and the ceiling.
The failure modes to probe are documented: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability on hard grading tasks.
Judge-fed comparisons need uncertainty accounting like any eval: treat graded items as samples from a larger population and report standard errors, or noise reads as verdicts.
Model-graded evaluation is a first-class template in open eval frameworks, so the calibration protocol wraps standard tooling rather than bespoke infrastructure.
Why calibrate at all
A judge model will always produce a score. The question a gate has to answer is whether that score tracks anything. Published agreement numbers (claim c1) were measured on other domains, other prompts, other judge models — none of it transfers to your setup automatically. Calibration is the transfer measurement.
The protocol
1. Build the audit sample. Pull 100-200 real graded items from your own pipeline, stratified across the score range — not the easy tail. Have humans grade them blind, with the same rubric the judge gets. This human layer is the reference; budget for disagreement review between annotators. 2. Measure agreement. Compare judge verdicts against the human reference. Expect the ceiling around human-human agreement (claim c1): a judge at 80% agreement with a human panel that agrees 81% with itself is calibrated, not broken. 3. Probe each documented bias (claim c2). - Position: re-grade pairwise items with the order swapped; the flip rate is your position bias. - Verbosity: grade length-controlled variants of the same content; score drift with length is the verbosity bias. - Self-preference: if the judge model also generates candidates, measure the delta it awards its own outputs. - Reasoning limit: check agreement separately on the hardest stratum; judges degrade there first. 4. Bound the noise (claim c3). Re-run the judge on the same items; the run-to-run variance plus the sampling error defines the minimum score difference that means anything. Publish that number next to every judge-fed comparison. 5. Write the promotion decision. The judge may gate only the decisions whose required precision its measured agreement supports. Everything else it reports as telemetry.
Recalibration triggers
Recalibrate — not spot-check — when any of these change: the judge model (including provider-forced upgrades), the judge prompt or rubric, the task distribution being graded, or the candidate model family. The instrument changed; its old calibration is a memory. Wire the trigger into the same pipeline that catches model deprecations.
What this does not fix
Calibration bounds a judge; it does not make it deterministic. Ship gates that need reproducible verdicts stay on execution-based grading (see the harness-vs-judge comparison). The calibrated judge earns the open-ended slice — style, helpfulness, judgment — and the standard tooling makes running it cheap (claim c4). The expensive part was never the API call; it is the human reference sample, and that cost is the honest price of trusting a model's opinion.