AskpertFine-tuning vs prompt engineering: when to fine-tune, retrieve, or instruct
Three levers change three different layers: weights, context, and the task specification. Pick by which one is failing, because the expensive mistake is training a model to hold facts that change weekly.
On this page
Terms, definedthe jargon, decoded
- Fine-tuning
- Training an existing model further on your own examples so its weights change. The result is a new model you host or call, not a new prompt.
- LoRA
- Low-rank adaptation. A cheaper fine-tuning method that trains a small number of added parameters instead of the whole model.
- Retrieval
- Fetching relevant documents at request time and placing them in the model's context, leaving the model itself unchanged.
- Task specification
- The instructions defining the objective, criteria, constraints, and output contract for a request.
- Catastrophic forgetting
- When training on a new task degrades a model's performance on things it previously did well.
Fine-tuning, retrieval and instruction design get presented as competing ways to make a model better. They are not interchangeable, because each one changes a different layer. The right choice depends entirely on what is failing.
The expensive mistake is reaching for fine-tuning to install knowledge that changes every week, or to install a method that is really just a procedure nobody wrote down. The boundary is simple enough to hold in your head: fine-tuning changes weights, retrieval changes context, and instructions change the task specification.
What does fine-tuning change in a language model?
It changes the model's weights by training on examples chosen for a target behavior. That makes it a behavior intervention, well suited to stable form, format, style, and narrow tasks where the desired output can be demonstrated repeatedly and judged without reference to a moving knowledge base.
It is a poor mechanism for facts needing frequent correction, because those facts end up embedded in a rebuilt model instead of read from a source you can edit. Facts in weights carry a second surprise: they are not automatically queryable in both directions.
Fine-tuning also opens a safety surface that most teams do not budget for. Qi et al. (ICLR 2024) found that 10 adversarially designed training examples, submitted through the OpenAI API for under $0.20, stripped the safety guardrails from GPT-3.5 Turbo. The finding people miss is the second one: fine-tuning on benign, commonly used datasets also degraded safety, less severely but measurably. You do not have to be attacked to lose alignment. You only have to train.
When should you fine-tune an LLM?
When the failure is stable behavior, high-quality examples can represent the output you want, and neither retrieval nor clearer instructions fix the pattern. It is a real candidate for repeatable form, formatting, style, or a narrow input-to-output transformation that needs learned regularity.
Before committing, answer four questions:
- Does the model repeatedly produce the wrong format despite explicit instructions?
- Does it need to learn a narrow transformation rather than look something up?
- Can you define tests for both the target behavior and the regressions you fear?
- Will the examples stay valid long enough to justify a rebuild?
Reaching for LoRA does not dissolve the tradeoff, it relocates it. Biderman et al. (TMLR 2024) found LoRA substantially underperformed full fine-tuning on the target domain in standard low-rank settings across their code and math experiments. It also preserved more performance outside that domain, forgot less than the weight decay and dropout regularizers they compared against, and kept generations more diverse. So the choice is not better versus worse. It is target-domain adaptation versus preservation of the base model, and you should know which one you are buying.
What does retrieval change?
Retrieval changes the context supplied at inference time. It fits facts that change, records too numerous to fit in an instruction, and sources that need to stay separate from fixed model parameters.
The property that earns its keep is maintenance: you can replace or revise documents without retraining anything. That matters as soon as policies, catalogs, customer records, technical documentation or legal material move faster than your training cycle.
What retrieval does not do is teach a new capability. It supplies evidence for a capability the model already has. The model still has to select relevant passages, weigh authority, resolve conflicts and follow your answer policy. If it is using the wrong method, adding documents will not repair the method, and a retrieval architecture is an expensive place to hide a procedure-design problem. How an agent reaches external systems is a separate question, covered in MCP, APIs, and RAG.
There is a clean test for this. Remove the documents and ask whether the desired behavior should stay the same. If it should, you have an instruction or fine-tuning problem. If the answer should change with the documents, keep that knowledge outside the weights.
What do instructions and packaged procedures change?
They change the task specification the model is working from. This is the right lever for judgment, method, sequence, constraints, and tool-use policy whenever the model has the capability but has not been told the operating procedure.
An instruction can define the objective, the input assumptions, decision criteria, the output contract, refusal conditions and validation steps. Packaging that instruction makes it reusable across requests, so the runtime gets a better specification without touching weights or standing up a corpus.
Start here whenever the behavior you want can be stated as a procedure. Inspecting premises before reviewing reasoning, comparing options against stated criteria, validating before returning a schema: those are specification problems wearing the costume of capability problems. When a repeated task should become a skill covers the packaging decision separately.
How do the three compare?
| Lever | Changes | Best fit | Weak fit | Update path |
|---|---|---|---|---|
| Fine-tuning | Model weights | Stable format, style, and narrow learned tasks | Frequently changing facts or explicit procedures | Rebuild and re-evaluate |
| Retrieval | Runtime context | Current, numerous, or source-backed facts | Teaching a new method or fixing poor judgment | Update sources and retrieval behavior |
| Instructions or packaged procedure | Task specification | Judgment criteria, sequence, constraints, output contracts | A persistent learned pattern that survives clear instructions | Edit and test the procedure |
The update paths are asymmetrical, and that asymmetry should drive the decision more than it usually does. Retrieval and instructions change in minutes. A fine-tune is a rebuild, and on Qi et al.'s result every rebuild reopens a safety evaluation you now own.
What is the decision rule?
Instructions when the failure is a missing method. Retrieval when the failure is missing or changing information. Fine-tuning when the failure is stable behavior that survives having the method and information made explicit.
Apply it in this order:
- Specify the procedure. Write the task, evidence requirements, decision criteria, output format and validation steps, then test on representative cases.
- Supply changing knowledge. Add retrieval when correct behavior depends on documents or records outside the model's fixed knowledge.
- Train only the residual. Fine-tune when what remains is a stable pattern examples can teach and regression tests can measure.
- Recheck safety and generalization. After every weight change, evaluate the target task, unrelated capabilities, refusal behavior and output diversity.
The order does two things. It keeps temporary facts and mutable procedures out of your weights, and it makes failure attribution possible: the procedure was inadequate, the evidence was missing, or the learned behavior regressed. Collapse the steps and you lose the ability to say which.
Askpert is one option for domain experts who want to package expertise into AI Experts that buyers rent via API or MCP, with the underlying assets staying server-side. It packages a procedure and its supporting material, which is a different bet from assuming every domain problem needs a fine-tune.
Is fine-tuning better than prompt engineering?
Neither is categorically better. Fine-tuning changes model weights and fits stable learned behavior. Prompt engineering changes the task specification and fits explicit methods, constraints and output contracts. Start with instructions, then fine-tune only if stable residual behavior is still wrong once the procedure is clear.
Is RAG better than fine-tuning for company knowledge?
Retrieval is the better fit for company knowledge that changes or must stay traceable to source documents, because it changes request-time context rather than weights. Fine-tuning can teach a stable task pattern, but it should not be the source of truth for records that change frequently.
When should you not fine-tune a model?
Do not fine-tune to store frequently changing facts, or to encode a procedure you could simply write down. Both turn routine updates into model rebuilds. Fine-tuning also demands safety and generalization tests, since training can degrade behavior well outside the intended task.
Can prompting replace fine-tuning?
Prompting can replace fine-tuning when the model already has the capability and the failure comes from an unclear objective, method, constraint or output format. It cannot guarantee a persistent learned pattern across every context. Test the instruction on representative cases before concluding that weight changes are necessary.
Should you use instructions, retrieval, and fine-tuning together?
You can, when each addresses a separate failure: instructions define the method, retrieval supplies current evidence, and fine-tuning teaches a stable residual behavior. Combining them does not remove the need to isolate errors, so evaluate each layer separately or a bad answer will be blamed on the wrong intervention.