AskpertWhy a benchmark score does not transfer to your workload
GSM1k rebuilt a famous benchmark with new problems and matched difficulty. Some model families dropped up to 13%. The gap is the part of the score that was never generalization.
On this page
Terms, definedthe jargon, decoded
- Data contamination
- The risk that benchmark items, close variants, or their solution patterns appeared in a model's training data.
- Held-out set
- Evaluation data kept out of training, prompts, demonstrations, and public circulation, so a score on it still means something.
- Overfitting to a benchmark
- Scoring higher on known benchmark items than on matched unseen items from the same task.
- Spearman correlation
- A rank-based measure of association between two variables, which describes a relationship rather than establishing a cause.
- Generalization gap
- The performance difference between a familiar evaluation set and a matched set of new items.
A benchmark result belongs to a model, a prompt, an evaluator, and a particular collection of items. Your application has a different distribution, different failure costs, and different opportunities for a model to have encountered similar material during training.
The transfer question is narrower than "how good is this model?" It is: how much of this measured performance survives when the items change to match the work that matters? GSM1k gives a clean example, because it changes the items while preserving the properties that make GSM8k recognizable.
What does GSM1k show about benchmark contamination?
That a model can perform worse on new problems even when those problems preserve the original benchmark's style and difficulty. GSM1k consists of 1250 problems requiring only elementary mathematical reasoning. Its creators commissioned it to mirror GSM8k's style and complexity, matching human solve rates, number of steps in solution, answer magnitude, and more.
A model does not face an unrelated task on GSM1k. The mathematical domain stays familiar, the expected reasoning stays comparable, and the answers stay within the same general scale. Only the items are different.
Evaluating leading open- and closed-source models on GSM1k instead of GSM8k, the paper observes accuracy drops of up to 13%. That figure is not a general contamination measure for every model. It is the observed gap under this paired evaluation, with the paper's stated construction and model set.
Why does the GSM8k-to-GSM1k gap matter?
Because it separates performance on familiar items from performance on matched but different ones. A model's GSM8k accuracy includes whatever it learned about those particular problems, their wording, or their solution patterns. GSM1k tests whether the capability persists once that item-level familiarity is removed.
A model does not need to reproduce a stored answer for training exposure to affect its score. Familiar phrasing, recurring templates, or remembered solution structures can make a benchmark easier than a comparable private workload. The paper frames its findings as evidence about overfitting and possible memorization, not as proof that every high-scoring model copied every item.
The gap is informative even when its exact cause cannot be identified from the score alone. A result that changes substantially on a matched set deserves a different level of trust from one that stays stable, which is the same distinction reliability is a distribution, not a score draws at the system level.
Which models overfit to GSM8k?
Phi and Mistral show evidence of systematic overfitting across almost all model sizes examined, while Gemini, GPT, and Claude show minimal signs of it. Figure 1's caption puts the drops for the Phi and Mistral families at almost 10% under the reported comparison.
That split is the part most summaries lose. The experiment does not establish that every benchmark score is inflated by contamination. Frontier models in the paper held up; other model families showed a recurring decline. The variation between families is itself the useful information, because it means the question "does this score transfer?" has a different answer depending on which model you are considering.
So a benchmark delta supports a conditional statement: this model family performed differently when matched items replaced the public items. It cannot support the universal statement that benchmarks are useless, or that a particular model definitely memorized the benchmark.
Does the correlation prove that models memorized GSM8k?
No. The paper reports a positive relationship, Spearman's r² = 0.32, between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k. That is evidence consistent with partial memorization, not proof of copying by individual models or on individual items.
The relationship connects two observations: how likely a model was to generate a GSM8k example, and how much its performance differed across the paired benchmarks. A positive relationship supports the interpretation that exposure may be related to the gap. It does not identify the mechanism for every model, establish that memorization caused every error difference, or show that a model stored complete solutions.
Treat the correlation as a diagnostic signal rather than a forensic finding. Engineering teams do not need to prove a model's training history before asking whether a public score survives on private items drawn from the intended workload.
What should an engineering team measure instead?
Build a held-out evaluation set from your own input distribution and keep its items out of public circulation. The set should represent the task, output format, edge cases, and failure costs that determine whether the system is useful. A public benchmark can remain a reference point, but the held-out result should govern the deployment decision.
| Evaluation evidence | What it supports | What it does not support |
|---|---|---|
| Public benchmark score | Performance on published items | Transfer to a private workload |
| Matched replacement set | Stability when items change but task properties hold | A complete explanation for a score gap |
| Held-out workload set | Expected behavior on the intended distribution | General performance outside that distribution |
| Private or post-cutoff items | Lower exposure to known public examples | Proof that no related training material existed |
The test set has to be protected from accidental inclusion in prompts, demonstrations, fine-tuning data, and regression fixtures that later enter training. Its value depends on remaining a genuine test rather than becoming another familiar target.
Evaluation also needs consistent execution conditions. Prompt formatting can move accuracy on its own, so an evaluation that changes both the items and the template measures more than item generalization. Formatting alone can swing accuracy, which makes prompt versioning part of the benchmark record.
Finally, use public benchmark deltas as a warning signal when the margin is small. Prefer benchmarks whose items were created after a model's training cutoff or kept private, while recognizing that neither property proves the absence of related training material. The strongest evidence is convergence between a relevant public result and a protected held-out one.
Are benchmarks still useful after GSM1k?
Yes, when their scope is stated precisely and their limits are tested against the target workload. GSM1k does not invalidate GSM8k. It demonstrates that a GSM8k score can contain a component that does not persist on matched, unseen problems, and that the size of that component differs across model families.
A benchmark is a measurement, not a capability certificate. Its transfer depends on item similarity, model exposure, prompt conditions, and the distribution of the work being predicted. No single score answers all of those questions. Scoring design is yet another axis: a benchmark that rewards guessing over abstention shapes model behavior as well as mismeasuring it.
Use public benchmarks to form hypotheses, then use held-out workload data to test them. When the public score is high but the matched private score falls, trust the evidence from the distribution the system actually has to serve.
What is benchmark contamination in large language models?
Benchmark contamination is the risk that evaluation items, close variants, or their solution patterns appeared in a model's training data. It can make a public benchmark score look stronger than performance on new items from the same task distribution. GSM1k tests this concern by replacing GSM8k items while preserving key properties of the original task.
What is the difference between GSM8k and GSM1k?
GSM8k is the public grade school arithmetic benchmark, while GSM1k is a matched replacement set containing 1250 new problems. GSM1k was commissioned to mirror GSM8k's style and complexity, including human solve rates, solution steps, and answer magnitude. Only the item content differs.
Does a lower GSM1k score prove memorization?
No. The paper reports a positive relationship between the probability of generating a GSM8k example and the GSM8k-to-GSM1k performance gap, with Spearman's r² = 0.32. That relationship supports a possible exposure explanation but does not establish copying for any individual model.
Do all models overfit to public benchmarks?
Not in this study. Phi and Mistral showed systematic overfitting across almost all model sizes, while Gemini, GPT, and Claude showed minimal signs of it. The variation between model families is part of the finding, so "benchmarks are meaningless" overstates what the experiment established.
How can teams test whether a benchmark transfers to their workload?
Create a protected held-out set from the distribution the system will serve, then evaluate the model under the same prompt and execution conditions used in production. Public benchmarks remain reference measurements; the held-out result provides stronger evidence about workload performance because its items represent the real task and are not public targets.