Choosing k and the threshold for pass^k
The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, the threshold comes from the cost of one failure, and both must be set before you see the score. Includes why pass@k and pass^k move in opposite directions and why measured pass^k is almost never p^k.
Why this wins its question: Every source that recommends pass^k defines it and stops. The published material answers "what is pass^k"; a team about to gate a release needs "which k, what number, and why is mine lower than the multiplication says" — the two selection rules, the independence trap, and the sample budget, in one place and with the arithmetic worked.
Key takeaways
- k is not a compute-budget choice - it is how many times the task runs between human reviews.
- pass@k rises with k and pass^k falls with k; they answer opposite questions, so never quote one as the other.
- Set the threshold from the cost of a single failure before you see the score, or you will be negotiating with your own results.
- Measured pass^k is almost never the p^k the independence assumption predicts - correlated failures make it worse, and only measurement tells you which.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
pass^k is the probability that a system succeeds on all k repeated trials of the same task, introduced with tau-bench as a reliability metric distinct from single-attempt capability.
The gap between capability and reliability is large in practice: state-of-the-art function-calling agents succeeded on under 50% of tau-bench tasks, and pass^8 fell under 25% in the retail domain.
Repeated sampling moves the opposite metric in the opposite direction: on HumanEval, Codex solved 28.8% of problems with one sample and 70.2% with 100 samples per problem, because any-of-k credits a single success.
A pass^k estimate is a sample statistic and needs uncertainty accounting like any eval score: treat graded items as draws from a larger population and report standard errors alongside the point estimate.
Incidental setup details can move eval scores by around 5 percentage points on their own, so a threshold quoted without a stated harness and prompt configuration is not a reproducible gate.
Two metrics that look alike and point opposite ways
`pass@k` asks whether any of k attempts succeeded. `pass^k` asks whether all k succeeded (claim c1). The names differ by one character and the numbers move in opposite directions as k grows: Codex went from 28.8% to 70.2% on HumanEval by sampling 100 times instead of once (claim c3), while tau-bench agents that clear under 50% of tasks on a single attempt drop below 25% at pass^8 in retail (claim c2).
The distinction is what you are buying with the extra attempts. `pass@k` is a capability measure and assumes something downstream can pick the winning attempt — a test suite, a compiler, a human reviewer. If no such filter exists in production, `pass@k` is a number about a system you are not running. `pass^k` is the reliability measure: it describes what the user experiences when the same task runs k times and nobody is checking.
Choosing k
k is not a budget decision, and it is not "as many as we can afford". k is the number of times the task runs between human reviews.
Work it out from your own traffic:
- A weekly report the agent generates and an analyst reads before it
circulates: a human filter exists every run, so k = 1. Use pass@1 and spend your budget on the review, not on repeated trials.
- A support workflow that fires 200 times a day and is spot-checked
weekly: roughly 1,400 unreviewed runs between reviews. You cannot measure pass^1400, but you now know your gate is about the tail, not the average — pick the largest k you can afford to measure and treat it as a lower bound on the exposure, not a description of it.
- A nightly migration step in a pipeline of 12 sequential agent calls,
none reviewed until the pipeline finishes: k = 12, and the relevant question is whether the whole chain survives, not whether each step usually works.
Two corrections to the naive reading. First, the k that matters is per-task-instance, not global volume: 200 runs of different tasks is a sample of size 200 at k=1, not pass^200. Second, if a retry is automatic and cheap and the failure is detectable, that retry belongs in the system under test — measure the agent with its retry loop at the k your reviews actually imply, rather than crediting a retry the harness performs and production does not.
Choosing the threshold
Set the number from the cost of one failure, and set it before you see the score. The order matters more than the value: a threshold chosen after the measurement is a description of what you got, not a gate.
The rule that survives contact with a release meeting: state the failure budget in incidents, then convert. If the workflow runs N unreviewed times per review cycle and you can tolerate at most one bad outcome per cycle, you need a per-run failure rate below 1/N — and it is the per-run rate, not the pass^k figure, that you should be arguing about. pass^k is how you observe that rate under repetition; the failure budget is what you are actually deciding.
Then write down, next to the number: the harness version, the prompt configuration, the model version, and the task set. Incidental configuration differences alone are worth several points of score (claim c5), so an undocumented threshold cannot be re-checked next quarter — and a gate you cannot re-run is a gate that quietly stops gating.
The independence trap
The arithmetic everyone does first: if each trial succeeds with probability p, then pass^k = p^k. At p = 0.95, pass^10 ≈ 0.60. At p = 0.99, pass^10 ≈ 0.90. This is worth doing once, because it explains why a "95% reliable" agent feels broken over a working week and why buying reliability at the third nine is the expensive part.
It is also wrong in a specific direction. Agent failures are not independent draws: the same task hits the same ambiguous instruction, the same brittle selector, the same tool that rate-limits at the same point. Correlated failures cluster, so measured pass^k is typically below p^k — and the clustering is itself the diagnosis, because a task that fails 4 times out of 8 in one cluster and never in another has a deterministic bug wearing a probabilistic costume. Use p^k to size the sample and to argue for the budget; use the measured value to gate, and look at the failure clustering before you look at the mean.
The sample budget
A pass^k estimate over n tasks costs n × k runs, and the estimate has error bars like any other (claim c4). Two consequences worth planning for:
- Rare events need runs you may not want to pay for. Distinguishing
a 1% failure rate from a 3% failure rate takes hundreds of trials per task. If the budget does not stretch, say so in the release note — "pass^8 = 0.94 ± 0.05, cannot distinguish from 0.90" is an honest gate; "pass^8 = 0.94" alone invites a comparison the data does not support.
- Report the interval, gate on the bound. Gate on the lower end of
the confidence interval, not the point estimate. It is the only version of the number that stays true when the same suite runs again next week.
Where this lands
Both selections — k and the threshold — belong in the release gate configuration, declared and version-controlled next to the other thresholds, not chosen per release by whoever is shipping. That is the whole point of writing them down before the run: the gate holds when the number comes back inconvenient.