Repeated sampling needs a verifier to pay off

Inference-time scaling is often summarised as "sample more, score higher". The published result is more specific, and the difference between the two readings is a production decision, not a semantic one.

What is repeated sampling in language models?

Repeated sampling is an inference-time scaling strategy that generates multiple candidate answers for one problem. The model receives a task, produces a sample collection, and then a separate method chooses what to return. More samples expand the candidate set; they do not establish which candidate is correct.

Brown et al. define coverage as "the fraction of problems that are solved by any generated sample." The paper reports that coverage "scales with the number of samples over four orders of magnitude" and that the relationship "is often log-linear and can be modelled with an exponentiated power law, suggesting the existence of inference-time scaling laws." The paper measures the opportunity created by sampling, not a guarantee that the system collects every correct candidate.

What does sampling 250 times mean for accuracy?

A larger sample collection can contain solutions for more problems, but its coverage is not ordinary accuracy. The reported result becomes delivered accuracy only if the system selects the candidate that solves each problem. The sample count measures generated opportunities, not the fraction of answers returned correctly.

On SWE-bench Lite with DeepSeek-Coder-V2-Instruct, Brown et al. report that the fraction of issues solved rises from 15.9% with one sample to 56% with 250 samples, "outperforming the single-sample state-of-the-art of 43%." The result is a coverage measurement under the stated benchmark and model conditions, not a claim that 250-sample inference returns 56% accuracy.

Read carelessly, that sentence promises a system three and a half times better. Read correctly, it promises that a correct patch was somewhere in the pile.

Why does coverage not equal delivered accuracy?

Coverage answers an existence question: did any generated sample solve the problem? Delivered accuracy asks a different question: did the system identify and return that sample? Repeated sampling raises the first quantity, while a verifier, scorer, or other selector determines the second.

A collection can contain valid and invalid candidates at the same time. Majority voting searches for agreement, while a learned reward model estimates quality. Neither operation supplies an oracle label. A model that asks itself to judge its own candidates is also using a selection procedure, not an automatic correctness certificate. The discussion of LLM self-correction covers the related problem of unaided repair.

Coverage is therefore an upper bound on attainable delivered accuracy for a fixed sample budget. The gap between the two quantities is the selection problem, and repeated sampling does not remove it.

What does a verifier add to inference-time scaling?

A verifier tests each candidate against conditions that define success, then lets the system select a candidate that passes. Verification converts a collection-level possibility into an actionable decision. Exact checks work best when the task exposes a machine-testable contract, such as executable behavior or validity in a formal proof system.

The paper states that "in domains like coding and formal proofs, where answers can be automatically verified, these increases in coverage directly translate into improved performance." The conversion from coverage to performance is tied to automatic verification.

The first two rows match the paper's coding and formal-proof examples. The latter rows are general engineering patterns, not additional measurements from the paper. A structured-output validator can reject malformed data while accepting a semantically wrong value. The verifier must test the property the system promises to deliver.

Why do majority voting and reward models plateau?

Majority voting and reward models are selection methods, not automatic proof of correctness. They help only when agreement or learned scores continue to distinguish valid candidates from plausible failures. In domains without a correctness test, the selection signal can stop extracting value from a growing sample collection even while coverage rises.

The abstract reports: "In domains without automatic verifiers, we find that common methods for picking from a sample collection (majority voting and reward models) plateau beyond several hundred samples and fail to fully scale with the sample budget." The source paper separates generating more candidates from selecting the right candidate.

A majority vote is useful when equivalent correct answers converge on the same observable form. Free-form answers do not always have that property. A reward model can rank candidates without possessing ground-truth access. More candidates cannot repair a selector whose signal no longer distinguishes correctness.

Is repeated sampling the same as evaluating reliability?

Repeated sampling for inference and repeated runs for reliability answer different questions. Inference-time scaling generates candidates for one problem and selects one to return. Reliability evaluation repeats a system run to measure outcome variation across runs. One practice improves an answer attempt; the other measures the distribution of attempts.

The post on AI agent reliability treats repeated runs as evaluation. This one treats repeated sampling as a search over candidates before returning an answer. Keeping the objectives separate prevents a coverage result from being reported as ordinary accuracy or as a reliability estimate.

How should engineers decide whether to sample more?

Increase the sample budget only when additional candidates address a measured failure mode, a verifier can identify successful candidates, and the quality gain justifies the added inference work. Repeated sampling multiplies generation cost roughly linearly with the sample count. A larger budget is a cost and latency decision, not a free accuracy setting.

Each candidate requires another model generation, and verification or ranking adds its own work. The cost of running an AI agent is relevant even when sampling improves the attainable result. You do not need a dollar estimate to establish the tradeoff: more generations consume more resources, while the benefit depends on coverage and selection quality.

Track three quantities separately. Coverage shows the available upside. The selector's passing-candidate rate shows how much of that upside the system captures. Total inference and verification cost decides whether the captured improvement belongs in production.

Repeated sampling is a conditional scaling axis. It expands the set of possible solutions. A verifier is the mechanism that turns that expanded set into a returned answer.