Fine-tuning tutorials show a training run completing and a model producing better outputs on the target task. They rarely show the catastrophic forgetting that degraded general capability, the overfitting that made the model brittle on inputs slightly different from the training examples, or the data quality problems that caused the model to confidently learn the wrong behavior. The tutorial ends at the finish line. The real problems start there.
Analysis Briefing
- Topic: Fine-tuning failure modes and why results disappoint in practice
- Analyst: Mike D (@MrComputerScience)
- Context: Born from an exchange with Claude Sonnet 4.6 that refused to stay shallow
- Source: Pithy Cyborg
- Key Question: Why does a fine-tuned model that looked good in training fail in production?
What Catastrophic Forgetting Does to Fine-Tuned Models
Pre-trained language models encode a vast amount of general capability in their weights. Fine-tuning modifies those weights to improve performance on your target task. The modification process does not know which weight changes improve target task performance versus which changes degrade general capability. Both happen simultaneously.
Catastrophic forgetting is the name for the degradation of general capability that occurs as fine-tuning optimizes for the target task. A model fine-tuned on legal document summarization may produce noticeably worse outputs on general conversation, coding assistance, and non-legal summarization tasks after fine-tuning than before. The fine-tuning improved the target task at the cost of distributing weight changes that affected capabilities unrelated to it.
The severity of catastrophic forgetting scales with fine-tuning intensity: more epochs, higher learning rates, and smaller datasets all increase forgetting. A model fine-tuned for three epochs on 500 examples exhibits more forgetting than one fine-tuned for one epoch on 5,000 examples, even if both achieve similar target task performance on the training distribution. Minimizing forgetting requires minimizing the weight changes needed to achieve the target performance, which requires more data and fewer epochs rather than fewer data and more epochs.
Why Your Training Set Is Probably the Biggest Problem
Fine-tuning quality is bounded by training data quality in ways that no training technique can compensate for. A model fine-tuned on inconsistent examples learns inconsistency. A model fine-tuned on examples that contain the wrong behavior pattern learns the wrong behavior pattern, confidently and consistently.
The most common training data problem is inconsistency across examples. If your 500 training examples reflect different standards, different formats, and different quality levels because they were collected from multiple sources or generated by multiple people, the model learns an average of those inconsistencies rather than the behavior you intended. The fine-tuned model is more consistent than the training data, but consistent at the wrong behavior.
The second most common problem is insufficient coverage of edge cases. Training examples that cluster around common cases produce models that perform well on common cases and fail on edge cases more severely than the base model. The base model’s pre-training gave it some capability to handle unusual inputs. Fine-tuning on a narrow distribution reduces that capability in the fine-tuned domain while retaining it in unrelated domains.
The Evaluation Gap That Makes Fine-Tuning Look Better Than It Is
Training loss decreases during fine-tuning. Validation loss on a held-out sample of training data decreases. These metrics indicate that the model is learning the training distribution. They do not indicate that the model performs well on production inputs that differ from the training distribution.
Fine-tuning evaluation that measures performance on held-out training examples rather than on genuinely new production inputs produces systematically optimistic accuracy estimates. The held-out examples come from the same distribution as the training examples. Production inputs come from the actual distribution of things users ask, which is broader, more varied, and contains inputs the training set never anticipated.
The evaluation gap is why fine-tuned models that test well in development fail in production. The training distribution and the production distribution are different. Fine-tuning optimizes for the training distribution. The evaluation measures performance on the training distribution. Production exposes the difference.
What This Means For You
- Evaluate fine-tuned models on genuinely new inputs, not on held-out training examples. Build a separate evaluation set from production traffic or from inputs that were not used in training and do not resemble training examples.
- Test general capability after fine-tuning on tasks unrelated to your target domain. If general capability has degraded measurably, reduce training epochs and increase dataset size before accepting the fine-tuned model for production.
- Audit training data for consistency before training, not after. Inconsistent training examples produce inconsistently behaving models. Fixing training data quality is more effective than adjusting training hyperparameters for a model that learned the wrong behavior.
- Try few-shot prompting with your best training examples before fine-tuning. If 10 to 20 representative examples in a prompt produce acceptable results on your target task, the fine-tuning cost, forgetting risk, and maintenance burden are not justified by the marginal improvement fine-tuning would add.
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg → AI news made simple without hype.