Is your LLM performing @k? A discussion of common metrics.

You might think your LLM is performing OK on various @k metrics. But what, exactly, does that number tell you? That the model can generate a correct solution? That your system can identify one? Or that its attempts are often nearly correct?

These are different questions. Confusing them can make an evaluation look more informative than it actually is.

Several metrics for evaluating black-box systems, typically LLMs, share a common structure:

a) There is a test set of $p$ problems.

b) For each problem, the system generates $k$ candidate solutions.

c) Some or all of these candidates are assessed, either automatically, by humans, or through a combination of the two.

d) The results are aggregated into a single number.

I will collectively call these @k metrics. The notation is convenient, but it can hide important choices: how candidates are generated, which ones are selected, what counts as success, and how results are aggregated. A metric can be mathematically well-defined while a reported evaluation remains underspecified.

I will start with binary assessments: each candidate is accepted or rejected. Write $a_{ij}=1$ if candidate $j$ for problem $i$ is accepted, and $a_{ij}=0$ otherwise. Here, “correct” means accepted under the stated assessment procedure. Passing a finite test suite, for example, does not establish that a program is correct on every possible input.

We can treat the assessor as a black box when defining the aggregation rule. That does not remove the need to describe its acceptance criteria when reporting an evaluation.

This distinction is separate from whether an assessment is execution-based or execution-free, as discussed in the ODEX paper. A continuous similarity score such as BLEU does not directly supply a binary verdict. To use it inside a pass/fail metric, one would first need to specify a per-candidate scoring procedure and a threshold; the resulting metric would measure success against that threshold.

n@k: generation followed by selection.

The AlphaCode paper, Section 2.2, defines $n@k$ as the percentage of problems solved using $n$ submissions chosen from $k$ generated candidates, where $1\leq n\leq k$. A problem counts as solved if at least one submitted candidate passes the hidden tests. Selection may use information available to competitors, but not those hidden tests.

The important detail is that the system chooses which $n$ candidates to submit. The paper explicitly says: “The filtering method is up to the system itself”.

Consequently, $n@k$ measures the performance of a generation-and-selection system. Two systems can generate exactly the same candidates and achieve different scores because they select different subsets.

Suppose a model generates ten candidates for a problem, exactly one of which is correct. A selector that chooses that candidate achieves success at $1@10$; a selector that chooses any other candidate fails. The generator has not changed.

To make this dependency visible, I suggest writing $f@n@k$, where $f$ denotes the selection procedure. If $S_i^f$ is the set of $n$ candidate indices selected for problem $i$, the observed score is

Multiply by 100 to express it as a percentage. The hat indicates a score calculated from a particular set of generated candidates; repeated generation and selection may produce different results.

For example, one could write logprobrank@n@k when selecting the $n$ candidates with the highest model log-probabilities, or M@n@k when ranking them with a reward model $M$. These are proposed labels, not established standards, and they still require details such as length normalization and tie-breaking.

Calling $n@k$ inherently ill-defined would go too far. It is well-defined once the system and evaluation protocol are specified. The problem arises when a bare number is presented as if it characterized the generator alone. You do not need access to proprietary model weights to interpret the score, but you do need to know what procedure produced it.

pass@k: did any candidate succeed?

For each problem, generate $k$ candidates and count the problem as solved if at least one is accepted. The observed score is

Since every candidate is eligible, no selection procedure is needed: $\mathrm{pass@}k=k@k$.

For any selector that submits a subset of the same $k$ candidates, its success rate cannot exceed pass@k. If none of the candidates is correct, selection cannot rescue the problem. If at least one is correct, selection can still fail to find it. The bound applies to selection from a fixed pool; a system that revises candidates is changing the pool.

This also resolves a tempting alternative definition. Suppose we define $n@@k$ by asking whether there exists a subset of $n$ candidates containing a correct solution. This gives an oracle upper bound, without requiring an actual selection mechanism. But for every $1\leq n\leq k$,

The reason is simple: whenever the pool contains a correct candidate, we can include it in a subset of any permitted size. If the pool contains none, no subset helps. The existence-based definition therefore collapses to pass@k even when $n<k$.

This is a useful way to understand pass@k: it measures whether the candidate pool contains a successful answer. It does not tell us whether the system can recognize that answer without access to the final assessment.

An earlier related measure appears in SPoC, Section 6, as “success rate at B”: the fraction of problems solved within a budget of $B$ trials. Its trials involve search and compiler feedback. The Codex paper subsequently uses the name pass@k and gives an estimator for it. The shared idea is success within a budget; the procedures generating the attempts need not be identical.

What does k count, and are the attempts independent?

The event “at least one of these $k$ candidates succeeds” is meaningful whether or not the candidates were generated independently. Independence is an assumption about the sampling procedure, not something guaranteed by the symbol @k.

For independent, identically distributed attempts on problem $i$, let $q_i$ be the probability that one attempt is accepted. Then

This formula does not generally apply to attempts that depend on previous outputs or feedback. Ten fresh samples, ten steps of an interactive debugging session, and ten refinements of one proof are different protocols. Each can have a success rate, but the protocol must be stated. In particular, distinguish feedback available during generation from assessment used only to calculate the final score.

There is also a difference between the budget $k$ being evaluated and the number of samples used to estimate performance. The Codex paper generates a larger pool of $N\geq k$ samples per problem. If $c_i$ of those samples are accepted, its estimator is

where $\binom{N-c_i}{k}=0$ when $N-c_i<k$. The term in brackets is the probability that a uniformly chosen subset of $k$ candidates from the observed pool contains a success. Under independent, identically distributed sampling, it is an unbiased estimator of success with $k$ fresh attempts. Taking arbitrary subsets of an adaptive search trajectory does not generally estimate the success of running that search for $k$ steps.

Finally, $k$ counts candidates, not computational cost. Comparisons should also report relevant token, time, tool-use, and selection budgets.

How many candidates succeeded?

For a fixed pool of $k$ candidates, pass@k treats one accepted candidate and $k$ accepted candidates identically: both solve the problem. To distinguish them, report the fraction of accepted candidates:

Under the same independent sampling protocol, this estimates pass@1, using multiple samples per problem. Generating more samples improves the estimate; it does not turn the quantity into a best-of-$k$ success rate.

There is an important qualification to the claim that pass@k ignores the number of successes. A single observed pool contributes only a yes/no outcome. Expected pass@k nevertheless depends on the underlying success probabilities, and the larger-pool estimator above uses the counts $c_i$. What pass@k does not do is directly report the fraction of candidates that are correct.

Reporting pass@1 alongside pass@k helps distinguish a model that succeeds reliably on individual attempts from one whose success depends on having many chances.

Partial credit asks another question.

Counting fully correct candidates is different from measuring how close an incorrect candidate comes to passing. A program might fail only one test, or it might fail every test. A binary verdict treats both as failures.

Yeo and colleagues (2024) address this with pass-ratio@n. Their metric averages the squared fractions of test cases passed by the generated programs. Using $k$ for their number of generated candidates, and writing $r_{ij}\in[0,1]$ for the fraction of tests passed by candidate $j$ on problem $i$, the per-problem score is

For example, a program passing 8 of 10 tests contributes $0.8^2=0.64$. To aggregate across problems, one can average these per-problem scores, giving each problem equal weight. Pooling all tests instead would give more weight to problems with larger test suites.

This measures partial test success across candidates. It is neither the fraction of candidates that pass every test nor the probability that at least one candidate does so. Squaring is also a substantive choice: it penalizes intermediate pass fractions compared with an unsquared average.

Partial credit requires a defensible scoring rule. Passing 90% of a test suite does not necessarily mean that a program is 90% correct: the result depends on the tests and their weighting. Mathematical proofs make this difficulty particularly clear. One invalid step can undermine an entire argument, so counting valid-looking steps is not a satisfactory measure of proof correctness. Partial-credit rubrics are possible, but the rubric itself needs justification.

When reporting an @k result, I would therefore state:

• Generation: the model, prompts, sampling settings, and whether candidates are independent attempts or part of an adaptive process.
• Budgets: the number of candidates generated and submitted, together with relevant computational costs.
• Selection: how submitted candidates are chosen and what information the selector can access.
• Assessment: what counts as acceptance, or how partial credit is assigned.
• Aggregation and estimation: how problems are weighted, whether the result comes from one sampled pool or an estimator, and how variability is measured.

Generating a correct solution, finding it among other candidates, and making partial progress are all worth measuring. Choose the quantity that answers your question, and describe the procedure well enough that the reported number can be interpreted.