Write a better prompt first, always. If prompting hits a hard ceiling, use RAG to add knowledge. If RAG hits a ceiling on behavior and style, fine-tune. Each step costs more and reverses less cleanly than the one before it. Most teams reach for fine-tuning before exhausting what better prompting could have achieved for free.
Analysis Briefing
- Topic: Decision framework for fine-tuning versus RAG versus prompt engineering
- Analyst: Mike D (@MrComputerScience)
- Context: A research sprint initiated by Claude Sonnet 4.6
- Source: Pithy Cyborg
- Key Question: Which approach actually solves your problem, and which one just feels like solving it?
What Each Approach Actually Changes and What It Cannot
Prompt engineering changes what the model does with a given input without changing the model itself. It is free, reversible, and immediately testable. Its ceiling is the model’s existing capability. You cannot prompt a model into knowing information it was never trained on. You cannot prompt away a fundamental capability gap. But you can prompt away a surprising number of quality problems that teams misdiagnose as capability gaps.
RAG adds information retrieval to the inference pipeline. It gives the model access to documents, databases, and knowledge bases that were not in its training data. It does not change the model’s behavior, style, or reasoning patterns. It changes what the model knows at inference time. RAG is the correct solution when the problem is a knowledge gap: outdated information, proprietary data, domain-specific facts the model was not trained on. It is not the correct solution when the problem is a behavior gap.
Fine-tuning modifies the model’s weights based on training examples. It changes the model’s behavior, style, and default patterns permanently for the fine-tuned version. It is expensive, time-consuming, and the results are not cleanly reversible. Fine-tuning is the correct solution when prompting and RAG have genuinely failed and the problem is behavioral: consistent output format, specific writing style, domain-specific reasoning patterns that prompting cannot reliably produce.
The Decision Questions That Route You to the Right Approach
The first question is whether the problem is about knowledge or behavior. If the model gives wrong answers because it does not have access to the right information, RAG is the solution. If the model gives wrong answers because it reasons incorrectly or produces outputs in the wrong format even when it has the right information, fine-tuning or better prompting is the solution.
The second question is whether the behavior you want is already in the model’s capability range. If you can get the output you want through manual prompting on specific examples, the model is capable of the behavior and the problem is prompting reliability rather than capability. More systematic prompting, few-shot examples, and explicit format instructions often close this gap without fine-tuning.
The third question is how dynamic your knowledge requirements are. RAG handles dynamic, frequently updated information well. Fine-tuning encodes a static snapshot of knowledge into weights. A model fine-tuned on your product documentation requires retraining every time the documentation changes. A RAG pipeline that indexes your documentation requires only an index update.
Why Most Teams Fine-Tune Too Early and Pay for It
Fine-tuning feels like a serious, professional solution to a model quality problem. It involves training runs, datasets, compute costs, and model management. It signals investment. It also has a high failure rate for teams that reach for it before exhausting simpler options.
The most common fine-tuning mistake is training on output examples without establishing that prompting cannot produce those outputs. A model fine-tuned on 500 examples of the desired output format frequently performs worse than a model prompted with 5 examples of that format, because fine-tuning on small datasets introduces overfitting and catastrophic forgetting that degrades general capability while improving performance on the training distribution.
The second most common mistake is fine-tuning to add knowledge rather than to change behavior. Fine-tuning on factual information produces models that confidently state training facts while hallucinating on related questions the training data did not cover. RAG retrieves information at inference time. Fine-tuning memorizes it at training time. Memorized facts do not generalize. Retrieved facts do not require memorization.
What This Means For You
- Exhaust prompt engineering before considering RAG or fine-tuning. Systematic prompting with explicit instructions, few-shot examples, and output format specifications resolves most quality problems that teams escalate to RAG or fine-tuning prematurely.
- Use RAG for knowledge gaps, not behavior gaps. If the model does not know your proprietary information, RAG is the correct solution. If the model knows the information but reasons about it incorrectly, RAG will not fix that.
- Fine-tune for behavioral consistency only after RAG and prompting have failed. Fine-tune on behavior examples, not knowledge examples. Fine-tuning for style, tone, and reasoning patterns works. Fine-tuning to add facts produces confident hallucination on adjacent facts.
- Never fine-tune on a small dataset without testing prompting with equivalent examples first. Five well-chosen few-shot examples in a prompt frequently outperform a fine-tune on 500 examples of the same type, without the cost, the training time, or the catastrophic forgetting risk.
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg → AI news made simple without hype.