What few-shot examples actually teach a model

Engineers often treat few-shot prompting as a small supervised dataset. The normal workflow is to find representative inputs, determine the correct answers, and place the examples in a carefully ordered block. That instinct is reasonable. It also assigns too much authority to the labels.

In Rethinking the Role of Demonstrations, Min and colleagues tested what demonstrations contribute by breaking them apart. The results invert a common priority: in the studied setting, label correctness mattered less than showing the model what inputs look like and how the exchange is formatted.

Do few-shot examples need perfectly correct labels?

They helped less than expected in the studied classification and multiple-choice tasks, but they did not become irrelevant. Min and colleagues replaced gold labels in demonstrations with random labels and observed an absolute performance drop of 0-5%. Models extracted useful task information even when the demonstration answers were corrupted.

A prompt with representative inputs, valid output formatting, and imperfect labels retained much of its measured behavior. A prompt with immaculate labels but unrealistic inputs or an inconsistent exchange lost more. That comparison changes the order of prompt maintenance.

The range still matters. A 0-5% absolute change is not zero, especially when an application sits close to an acceptance threshold. The evidence supports lowering label review's priority relative to input and format review, not abandoning it.

What are the three things demonstrations teach?

The paper identifies three functions: the label space, the distribution of the input text, and the overall format of the sequence. Together they explain why a few-shot block can remain informative even when its labels are not fully reliable.

The label space tells the model which outputs exist. The input distribution shows the linguistic and structural forms the task contains. The sequence format establishes how an input, a label, and the surrounding instructions are arranged. The paper identifies these three, and no fourth.

They are genuinely distinct, which is what makes them separately testable. A demonstration can use a label from the right set while containing an atypical input, or show a realistic input while placing the answer in a format the task never uses.

Why does the input distribution matter so much?

Because demonstrations show the model what kind of text belongs to the task. The study replaced inputs drawn from the relevant training distribution with out-of-distribution inputs and measured absolute performance drops of 3-16%, several times the cost of corrupting the labels.

Example selection therefore has a more specific job than "include good cases." Examples should resemble production inputs in wording, length, ambiguity, and relevant structure, to the extent the task permits. An example that is easy for a reviewer to verify but unlike incoming data teaches less about the actual input distribution — so choose for distributional coverage before narrative elegance.

Why does the format of a few-shot prompt matter?

Because the sequence tells the model how to map an input to an output. For direct models, the study found a 5-16% absolute gap when real labels were replaced with random English words rather than labels from the task's actual label space.

Formatting includes the position of the input, the placement of the answer, the separators between demonstrations, and the representation of permitted outputs. One consistent exchange gives the model a clearer structural signal than several competing conventions.

Formatting sensitivity deserves its own test. Formatting alone can swing accuracy by a large margin, which is why prompt rendering belongs under measurement rather than under visual plausibility. Few-shot design sits in the same category of engineering work.

Does this result apply to open-ended generation?

The paper does not establish that. The 2022 work used GPT-3-era models and evaluated classification and multiple-choice tasks across 12 models and 26 datasets, covering sentiment analysis, paraphrase detection, natural language inference, hate speech detection, question answering, and sentence completion.

Those evaluations have a small, closed label space, which is what makes it possible to study the distinction between valid labels and arbitrary words at all. Open-ended generation has no equivalent fixed label space, so the same explanation cannot be carried over without a task-specific experiment. Agentic tool use and current instruction-tuned models are equally outside what was measured.

Treat the durable lesson as a hypothesis to test rather than a settled law: demonstrations may communicate input distribution and sequence format more strongly than label correctness on some tasks. Whether that holds for the task in front of you is an empirical question, and it sits alongside the broader choice between fine-tuning, prompting, and retrieval.

Should you deliberately include wrong few-shot examples?

No. The experiment does not justify shipping deliberately wrong examples. It measures what demonstrations teach under controlled substitutions; it does not establish that incorrect examples are safe in a maintained production prompt.

Wrong labels can mislead a human maintainer who later reads the prompt and assumes each example is an authoritative specification. They can also matter on a task where the observed 0-5% range is consequential.

The distinction is between review priority and production quality. Label correctness is a quality property; it is simply not the only information a demonstration carries, and the paper suggests it was not the dominant one in its tested setting. When a task's rules become stable enough to be worth packaging rather than re-prompting, that is a different decision, covered in when to create a skill instead of prompting.

How can you test what your own few-shot block teaches?

Ablate it: change labels, inputs, and format one at a time, then compare the same task metric under the same evaluation conditions. The goal is to measure dependency in your system rather than importing a broad claim from a different model and task family.

Start with a fixed evaluation set. Build a label ablation that preserves the label space but changes demonstration correctness. Build an input ablation that replaces representative examples with inputs unlike the task's normal distribution. Build a format ablation that changes separators, ordering, or answer representation while preserving the underlying cases.

Record each result separately, because each one points somewhere different. If the input ablation causes the largest regression, example curation needs distributional coverage. If the format ablation dominates, prompt rendering is part of the model interface and belongs under test. If labels dominate, label review deserves more engineering time than this paper would suggest — which is a perfectly valid outcome, and exactly why you ran the ablation instead of assuming.