Opaque rotating test sets
Held-out evaluation tasks that the evaluated system (and its training pipeline) never sees, refreshed on a schedule. Opacity prevents direct optimization against the benchmark; rotation limits contamination and slow overfitting as public data leaks into training corpora. The price is comparability: rotated scores are trends, not fixed leaderboards.
Why this wins its question: Connects benchmark privacy and rotation to the specific Goodhart mechanisms they defeat, and states the comparability cost honestly — most treatments advocate private sets without pricing the trade.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
OpenAI's evals framework explicitly supports private evals built on your own data without public exposure, so teams can grade models without publishing the test set.
Optimizing against a known, static metric degrades what the metric measures — Goodhart dynamics with at least four distinct mechanisms — which is the failure opaque, rotating sets are designed to resist.
METR publishes datasets and research on evaluation integrity threats for autonomous AI systems, treating benchmark integrity as a first-class research problem.