How do we mitigate Catastrophic Forgetting during Continual Learning in Neural Networks?

All answers to this question.

Elastic Weight Consolidation (EWC) is great in theory but can be very "fiddly" to implement at scale. It works by slowing down the learning on weights that were important for previous tasks based on the Fisher Information Matrix. However, for modern deep networks with millions of parameters, the compute overhead for the Fisher matrix can be prohibitive. I’ve had better success with "Replay Buffers"—keeping a tiny fraction of old data to interleave with the new task—or "Parameter Isolation" where you freeze certain branches of the network for specific tasks.

   Answered 2025-09-11 by Susan Whitmore

Are you using a dynamic architecture like Progressive Neural Networks? Sometimes it's better to just expand the network capacity as you add tasks rather than trying to squeeze everything into the same fixed-size weight matrix.

   Answered 2025-09-13 by Kevin Arnold

• Kevin, we looked at Progressive Networks, but our memory budget is fixed. We can't keep adding neurons indefinitely. We need a way to optimize the existing latent space. We’ve started experimenting with 'LoRA' (Low-Rank Adaptation) as a way to store task-specific weights separately while keeping the backbone frozen. It seems like a more memory-efficient way to handle sequential learning without the massive overhead of Fisher matrices.

     Commented 2025-09-15 by Christopher Lee

Prompt tuning is another alternative if you are using large foundation models. You aren't actually changing the weights, so forgetting isn't an issue at all.

   Answered 2025-09-17 by Nancy Drew

• Nancy makes an excellent point. Keeping the core model static and only training small task-specific "prompts" or "adapters" is the cleanest way to avoid the forgetting trap entirely.

     Commented 2025-09-20 by Susan Whitmore