Overfitting is what happens when a model fits the training data so well that it memorises noise and idiosyncrasies instead of learning generalisable patterns. The training loss keeps going down, validation loss starts going up, and the production model performs worse than something simpler would have.
Every model with enough capacity will overfit if you let it. A 100-billion-parameter transformer can in principle memorise every sentence in its training corpus; the only reason it generalises is that the corpus is far too large to memorise and the optimisation process discovers structure as a shortcut. For smaller models on smaller datasets, overfitting is the default outcome unless you actively fight it.
The classic signs of overfitting:
- Training accuracy is near-perfect, validation accuracy is mediocre.
- The model performs differently on data it has seen vs data it has not.
- Predictions are highly confident but often wrong on edge cases.
- Adding more training data (without changing the model) closes the gap.
The fixes, in roughly the order to try them:
- More data — almost always the highest-leverage fix. Synthetic data and augmentation count.
- Simpler model — fewer parameters, fewer layers, or a less expressive architecture.
- Regularization — weight decay, dropout, early stopping, label smoothing.
- Better evaluation — a held-out set that actually resembles production traffic; cross-validation for small datasets.
- Feature engineering — removing leaky features that exist in training but not in production.
For LLMs the picture is inverted. Frontier base models are deliberately underfit on their massive corpora — they could memorise more than they do. Fine-tuning, by contrast, very easily overfits because the new dataset is small and the model is huge. The standard defences are short schedules (one or two epochs), small learning rates, low-rank adapters (LoRA) and constant evaluation against a held-out set from the target domain.
A US team that has been burned by overfitting once tends to instinctively split data correctly forever after. The first time you ship a model with 95% validation accuracy and watch it crater in production is the lesson that sticks.
