AskpertCan AI agents correct their own mistakes?
Asking a model to review its own answer measurably lowered accuracy in the ICLR 2024 study. The direction flips once an external signal exists, which makes the signal the design decision that matters.
On this page
- What did the research find about LLM self-correction?
- Why do some papers report successful self-correction?
- What counts as an external verification signal?
- How should an agent use a verification signal?
- Is multi-agent debate enough to verify an answer?
- What should buyers ask when an agent reviews its own output?
Terms, definedthe jargon, decoded
- Intrinsic self-correction
- Revision in which the model critiques and rewrites its own answer using nothing but its own judgment, with no test result, label, retrieved fact, or human feedback.
- External verification signal
- Evidence produced outside the model's unaided judgment that can reject an output, such as a compiler, a test suite, a retrieval hit, or a person.
- Oracle label
- Ground truth supplied to a system during evaluation, telling it whether an answer is correct. Improvements that depend on one do not transfer to production, where it is absent.
- Self-consistency
- Sampling several answers to the same question and taking the majority answer, rather than revising a single answer.
- GSM8K
- A benchmark of grade-school math word problems, widely used to measure multi-step reasoning.
An agent can revise an answer. Revision is not error correction.
Intrinsic self-correction means the model produces an answer, evaluates it using only its own judgment, and rewrites it without a label, an executable test, a retrieved fact, or an independent evaluator. The model is both the generator and the source of the verdict. A correct answer can be changed because the model invents a defect. An incorrect answer survives because the model finds its own explanation plausible. Extra reasoning gives the model more text to inspect, not a new fact that settles whether the inspected text is true.
Huang et al. measured this directly in Large Language Models Cannot Self-Correct Reasoning Yet, published at ICLR 2024.
What did the research find about LLM self-correction?
In the intrinsic setting, GPT-4 GSM8K accuracy fell from 95.5% before revision to 91.5% after one round and 89.0% after two. GPT-4 HotpotQA held at 49.0% after round one and fell to 43.0% after round two. CommonSenseQA moved from 82.0% to 79.5% and back to 80.0%.
GPT-3.5 followed the same shape. GSM8K went 75.9%, 75.1%, 74.7%. HotpotQA went 26.0%, 25.0%, 25.0%. Its CommonSenseQA result collapsed from 75.8% to 38.1% before partially recovering to 41.8%, though that collapse belongs to one reported condition and should not be read as a property of language models generally.
These are accuracies, not failure rates. A review prompt reliably changes the output. It does not move the output toward the truth. The same pliability appears when the pushback comes from a person instead of a prompt: challenged on a correct answer, assistants abandoned it up to 86% of the time.
Why do some papers report successful self-correction?
Because many of them quietly supply information the production system will not have.
Huang et al. report that improvements from methods such as Reflexion and RCI come from oracle labels guiding the correction process, and that the improvements vanish once those labels are unavailable. An oracle label tells the system whether its result is correct. That is a different machine from one that rereads its answer and asks whether the answer looks good. With a label, revision is conditioned on an observed error. Without one, the system has to infer the error and the remedy from the same uncertain process that produced the mistake.
This is why a paper and a product page can both say "self-correction" and describe different systems. The paper's feedback may come from a unit test, a reward signal, a reference answer, or a tool result. The product page usually means a second prompt.
Kamoi et al.'s critical survey of self-correction research reaches the sharper version of this conclusion: no prior work in their review demonstrates successful self-correction with feedback from prompted LLMs, outside tasks exceptionally suited to it. What succeeds are tasks with reliable external feedback.
What counts as an external verification signal?
Evidence generated outside the model's unaided judgment that can reject an output. It does not have to be human. It has to have a relationship to correctness that the draft does not already contain.
| Signal | What it can establish | What it cannot establish |
|---|---|---|
| Compiler or type checker | Whether code satisfies formal language constraints | Whether the design solves the user's problem |
| Test suite | Whether specified cases pass | Whether unspecified cases are safe |
| Database or retrieval result | Whether a claim matches the consulted record | Whether the record is current or complete |
| Independent source or model | Whether another process disagrees | Which process is right |
| Human review | Whether a person accepts the result under stated criteria | Whether the person was correct |
Every signal has a ceiling. A compiler rejects type violations and certifies nothing about business logic. A retrieval result can contradict a claim without proving the source authoritative. An external evaluator can surface disagreement, and disagreement is not resolution.
The design requirement follows: name the observation that could falsify the output, then make revision conditional on that observation. If no available observation distinguishes a correct answer from a persuasive wrong one, the loop is editorial rather than verifying.
How should an agent use a verification signal?
Separate generation, verification, and revision, and give the verifier the artifact plus the external evidence:
- Generate an artifact and record the assumptions that matter.
- Run the strongest available check against it.
- Classify the result as pass, fail, or unverified.
- Revise only after a fail or a specific contradiction.
- Re-run the check and stop when the acceptance rule is met.
Keeping pass, fail and unverified distinct is the part teams skip. Treating "the reviewer found no problem" as a pass rebuilds the closed loop the signal was meant to open, because a reviewer with no independent evidence cannot produce a pass. A test that did not execute is not a passing test. A search that returned nothing does not confirm a claim is false.
Is multi-agent debate enough to verify an answer?
Not when every participant sits inside the same evidence boundary. More model responses can expose disagreement and add sampling diversity, but a panel does not become ground truth because its members were assigned different roles.
Huang et al. compared the two at an equal budget of six responses on GSM8K: self-consistency reached 85.3%, multi-agent debate 83.2%. That does not show debate never helps. It shows agent interaction is not a substitute for a signal that carries accuracy.
A second model earns its cost when it brings different evidence, a different tool, a formal checker, or a separately defined procedure. Without that independence it is one more judgment about the same artifact, and evaluator agreement can reflect shared preferences rather than correctness.
What should buyers ask when an agent reviews its own output?
Ask what produced the verdict, not how many review passes ran. "The agent reviews its own output" describes a prompt sequence, not an evaluation method.
Ask whether the reviewer saw a test result, a source record, compiler output, a reference answer, or a human decision. Ask whether the reported improvement was compared against a single generation at equal inference cost. Ask whether the evaluation included cases where the first answer was already correct, since that is where revision does its damage, and why reliability is a distribution rather than a score.
A self-review prompt can produce a cleaner explanation. Cleanliness is not verification. An agent becomes correctable at the point where something outside its own judgment can reject its work and tell it what to do next.
Can an LLM reliably correct its own mistakes?
An LLM corrects some mistakes, but intrinsic self-correction did not reliably improve reasoning accuracy in the Huang et al. evaluation, and often lowered it. The model is using its own judgment to identify both the error and the fix. Reliable correction needs an external signal that can tell an acceptable result from a plausible but incorrect one.
What is intrinsic self-correction in an LLM?
Intrinsic self-correction is revision based only on the model's own judgment. The model generates an answer, critiques it, and produces a replacement with no oracle label, test result, retrieved evidence, independent reference, or human feedback. The definition deliberately excludes correction methods whose gains depend on information supplied from outside the original answer.
Why does self-correction sometimes improve LLM performance?
Because in those cases the correction process is receiving an external signal. Oracle labels, executable tests, reference answers, retrieved records, and explicit evaluator feedback can each identify a defect the original answer does not reveal. Results from those settings should not be generalized to unaided self-review.
Does asking an AI agent to review its answer make it more accurate?
Not on its own. Huang et al. measured GPT-4 GSM8K accuracy falling from 95.5% to 91.5% after one intrinsic revision round and to 89.0% after two. An answer can become clearer while becoming less correct, so accuracy needs a separate check.
What is the best external signal for an AI agent?
The strongest check that directly tests the output's acceptance condition. A compiler tests language constraints, a test suite tests specified behavior, retrieval tests agreement with a source, and a human tests stated criteria. No signal proves every property, so the agent should record what each check actually covered.