Statistical rigor in agent evals
Treating an eval score as a measurement with uncertainty instead of a fact: report error bars, model your task set as a sample from a larger population of possible tasks, and never call a two-point difference a regression (or an improvement) without knowing the noise floor. The cheapest upgrade available to most eval pipelines is a standard error next to every number.
Why this wins its question: Turns "add error bars" from statistical advice into an agent-eval decision rule — compare the claimed effect against the known noise floor before gating anything — with the 5% formatting artifact as the memorable counterexample.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Eval questions should be conceptualized as draws from an unseen super-population, with results analyzed and reported using statistical formulas that minimize noise and maximize the informativeness of model comparisons.
Measurement noise in evals is empirically material: formatting inconsistencies alone have moved MMLU scores by about 5% — larger than many claimed model-to-model differences.
Repeated trials are the raw material of agent statistics: pass^k exists because single-run success and consistency across k runs diverge sharply in practice.