Catastrophic Forgetting

Catastrophic forgetting happens when you fine-tune a pretrained model on a narrow task and the updates overwrite the weights that held broader skills. The new task gets better. Everyday abilities — following instructions, coding, other languages, simple reasoning — can quietly get worse.

Intuition

Picture a generalist who studies only medical Q&A for weeks and somehow “forgets” how to write Python or speak French. The model did not delete a folder named “French.” The same shared weights store many skills. Push those weights hard toward one narrow dataset, and other skills can fade.

Why it is scary:

• Silent failure — Task metrics go up while general benchmarks fall, unless you measure both.
• Worst with full fine-tuning — Updating all weights at a high learning rate on small, focused data erases prior knowledge fastest.

How it works

Why fine-tuning erases knowledge

Cause	Plain-English idea
Shared weights	Many skills live tangled in the same parameters.
Narrow data	Fine-tune examples cover a tiny slice of what pretraining saw.
Weight drift	The farther you wander from the starting model, the more old skills suffer.
No memory of old tasks	A normal optimizer does not “remember” earlier abilities unless you re-show old data or add a penalty for drifting.

Measure before vs after

Imagine a 7B model fully fine-tuned for a few epochs on legal contracts: contract accuracy jumps, while general benchmarks drop. That pattern is the warning light. Keep one fixed general eval set and compare base vs fine-tune every time.

A canary table makes the trade visible instead of invisible:

Check	Base model	After fine-tune	Verdict
Contract clause extraction	61%	88%	The goal — working
Follows a simple JSON instruction	94%	71%	Instruction-following eroded
Writes a working Python function	78%	52%	Coding damaged
Answers a basic factual question	89%	86%	Acceptable

Without rows 2-4, this run looks like a clear success. With them, it is a model that got much better at one job and noticeably worse at being useful.

Your canary set does not need to be sophisticated. Twenty to fifty fixed prompts covering the abilities you would be upset to lose, scored the same way every time, catches most of this.

Mitigation strategies

Strategy	Plain-English idea	When to use it
Conservative hyperparameters	Smaller learning rate, fewer epochs, early stop on canary evals	Always — your default baseline (almost free)
Rehearsal / data mixing	Mix a little general instruction data (often about 1–10%) into training so old skills keep getting practice	When you have (or can proxy) general data
Regularize toward the start	Penalize big drift from the pretrained weights (L2-SP or EWC-style ideas)	Full fine-tune required and little general replay data
Weight averaging	After training, blend fine-tuned weights with the original model to recover generality	Model already fine-tuned and general quality dropped
Freeze layers	Lock lower layers; tune only the top part closest to the task	Tight compute or very small datasets

A bit more detail (still plain)

• Rehearsal — Interleave domain data with diverse general examples (instructions, code, math, other languages — whatever you fear losing). Target learning may slow a little; that is the trade.
• Regularize toward pretrained weights — Gently pull weights back toward the starting point. Too strong a pull underfits the new task; too weak and you still forget. Tune against canary scores.
• Conservative knobs — Full fine-tune learning rates around 1×10⁻⁵–2×10⁻⁵; often 1–2 epochs is enough. Train less than “feels safe” if canaries dip.
• Weight averaging — Mix “base” and “fine-tuned” with a blend factor; sweep a few blends and pick the best trade-off between task score and general score.
• Freeze layers — Keep embeddings and early blocks fixed; open only the top blocks. Tuning roughly the top quarter to half of blocks often captures most task gains with less forgetting.

Choosing a stack

Start with conservative hyperparameters. Add rehearsal if you can. Use freezing when data or compute is tight. Reach for regularization or averaging when you must fully fine-tune and general skills are slipping.

What goes wrong

• Celebrating a rising task metric while ignoring fallen general scores.
• Five epochs on a tiny niche dataset at a hot learning rate, then wondering why the assistant “got dumb.”
• No canary set — forgetting stays invisible until users complain.

One-line summary

Narrow fine-tuning can overwrite broad skills; measure general canaries, then reduce forgetting with smaller updates, rehearsal, freezing, regularization, or blending back toward the base model.

Key terms

• Catastrophic forgetting — Losing older abilities after narrow new training.
• Canary / anchor eval — Fixed general prompts you always re-score.
• Rehearsal — Mixing general data into fine-tuning.
• Weight averaging — Blending fine-tuned and base weights after training.