“Remembrance of things past is not necessarily the remembrance of things as they were.”
Marcel Proust
Memory. It shapes our identities, informs our decisions, and—if you believe the ancient Greeks—steers our destinies. Yet for all its importance, memory can be a fickle thing. We humans can remember a childhood phone number for decades, yet blank on the name of someone we met ten seconds ago. Large Language Models (LLMs)—the powerhouse AI systems behind chatbots, translation tools, and more—face an even quirkier memory puzzle. They can reel off centuries of literature but may “forget” how to answer a simple question about yesterday’s training data.
Welcome to the world of catastrophic forgetting, retroactive interference, and proactive interference—terms borrowed from both cognitive psychology and deep learning research. In this blog post, we’ll explore how LLMs gain and lose knowledge, why that matters, and how researchers are working to help these digital “minds” hang onto what they learn.
A Quick Trip Down (Human) Memory Lane
Humans may not be perfect recall-machines, but we do have a knack for hanging onto certain skills for life. Nobody wakes up one day and says, “Oops, I forgot how to ride a bike.” But we do occasionally forget where we left our keys or the name of our high school math teacher.
In psychology, two major types of memory interference are:
- Retroactive Interference: New information overwrites or muddles old information (like memorizing a new password and forgetting the old one).
- Proactive Interference: Old habits or knowledge hamper the acquisition of new learning (think of typing your old zip code because it’s so ingrained).
Much as we’d love to believe AIs have pristine, photographic memories, it turns out they’re not so different from us—and might occasionally forget what was once second nature.
Why LLMs Forget: The Neural Network Angle
Modern LLMs—such as GPT, BERT, LLaMA, etc.—are built on deep neural networks. During training, these networks learn a massive number of parameters that encode patterns in language. This process is somewhat akin to us forming mental associations. However, if you then tune or “teach” the model something new without taking precautions, older “memories” can be overwritten. This phenomenon is often called catastrophic forgetting (French, 1999).
When new information (e.g., legal text) floods in, it changes the model’s internal weights to better handle that new domain. But in the process, the model might lose or distort pre-existing knowledge (Kirkpatrick et al., 2017). That’s roughly equivalent to “retroactive interference” in humans. Meanwhile, if the model’s older patterns are so dominant that it fails to learn the new domain well, we’re looking at a version of “proactive interference.” So yes—LLMs can be just as stubborn as your Uncle Bob refusing to switch from Windows XP.
Retroactive Interference: Out with the Old
Retroactive interference in LLMs manifests when a once-fluent model inexplicably fumbles. Suppose your beloved GPT-based assistant excelled at discussing gardening tips. You then fine-tune it on new data for advanced medical diagnoses. Suddenly, it draws a blank on the best fertilizer for tomatoes—even though it aced that question two weeks ago. It’s like cramming all night for a final exam on Shakespeare and forgetting the name of your best friend the next morning. Painful.
Symptoms of Retroactive Interference
- Loss of Specific Knowledge: The model forgets or confuses previously learned facts.
- Distorted Output: Responses that reference old data now produce partial or garbled information.
How to Combat It
- Regularization: Techniques such as Elastic Weight Consolidation (EWC) anchor important parameters that the model should not drastically alter (Kirkpatrick et al., 2017).
- Replay or Rehearsal: The model periodically “revisits” older data to refresh memories, similar to a student reviewing old lecture notes.
- Adapter Tuning: Only a small subset of parameters (the “adapter”) is changed for new tasks, freezing the original model. This way, the new knowledge doesn’t overwrite everything else.
Proactive Interference: Old Habits Die Hard
What about the flip side—when an LLM’s old knowledge blocks the new stuff? Say your model was heavily trained on marketing copy. Then you try to teach it academic writing. The result? Your brand-new “research assistant” can’t resist peppering every paragraph with “unbeatable deals.” That’s proactive interference: the old patterns are hogging all the mental real estate.
Examples of Proactive Interference
- Persistent Style or Tone: The model can’t drop that salesy voice, even though you want cold, hard data.
- Misapplication of Past Grammar or Facts: The old rules overshadow the new rules, so the model’s output remains stuck in old patterns.
Easing the Old Grips
- Prompt Tuning: By adding a carefully engineered prefix or instruction, you can nudge the model toward a new style or task.
- Domain-Specific Modules: Sometimes called “adapters” or “LoRA modules,” these discrete neural blocks handle the new domain without rewriting older configurations.
- Multi-Task or Joint Training: Train the model on multiple tasks together from the get-go, ensuring no single domain becomes too entrenched.
Bridging the Gap: Human Insights into AI Memory
“The palest ink is better than the best memory.”
—Chinese Proverb
In some ways, we handle memory more gracefully than neural networks (once we learn to ride a bike, we rarely forget), but we’re also prone to interference. We know from experience how damaging it is to cram for an exam without reviewing older material. That’s the logic behind rehearsal methods in LLMs: if you feed the model small batches of old data alongside new material, it’s less likely to lose its pre-existing expertise.
Academically, the parallels are striking. French (1999) and others have long noted that in human cognitive systems, interleaving old and new knowledge is crucial for retention. The same principle applies to LLM training pipelines.
Practical Tools & Tactics
If you’re an AI engineer (or simply an enthusiast) wondering how to keep your LLM from going blank on older tasks, here are some handy tips:
- Parameter-Efficient Fine-Tuning (PEFT): Methods like LoRA or prefix tuning let you add new knowledge in small parameter sets without touching the bulk of the model. This can drastically reduce catastrophic forgetting while requiring less compute.
- Retrieval-Augmented Generation (RAG): Instead of jamming all new info into the model, store frequently updated content in an external database. The LLM retrieves relevant facts at runtime. No daily re-training = fewer chances to forget.
- Model Editing: If you only need to fix a small piece of knowledge, consider a surgical editing technique. This localized tweak spares the rest of the network from a risky parameter overhaul.
- Frequent Testing: Keep a “memory test suite” of queries representing older knowledge. Each time you train or edit the model, run those queries. If performance dips, you know some forgetting snuck in—time to break out the replay methods!
The Levity Factor: AI, Don’t You Forget About Me!
Let’s be honest: it’s downright comical to see an all-powerful AI bot draw a blank on something it literally knew yesterday. Consider this a gentle reminder that behind all the futuristic hype, AI is still a toddler learning to walk—albeit an extremely precocious toddler that can churn out Shakespearean sonnets on command. One might even say it’s like Dory from Finding Nemo, except instead of forgetting Nemo’s name every 15 minutes, it forgets how to perform your favorite crocheting instructions after learning complex astrophysics.
Looking to Tomorrow: A Lifelong Learning Future
Researchers in lifelong learning envision AI that can continuously update its knowledge without sacrificing past expertise (Zheng et al., 2025). The dream? Models that gracefully weave new data into their “brains” the way we integrate new experiences daily—minus the fiasco of forgetting where we put our car keys. By combining advanced parameter-efficient tuning, external memory systems, and a healthy dose of old-fashioned regularization, we might just crack the code of building a truly “remembering” AI.
Human memory is a wondrous, if sometimes slippery, phenomenon. Large Language Models share that tension between remembering and forgetting—only they often do so with more dramatic flair. Retroactive interference can wipe out old knowledge in one fell swoop, while proactive interference can block crucial new learning. Fortunately, techniques like regularization, rehearsal, PEFT, and retrieval-augmentation help LLMs hold onto their valuable lessons while still picking up fresh insights.
And so, to paraphrase the wisdom of Proust: just as our memories are shaped by both the new experiences we gather and the old ones we hold onto, LLMs flourish best when carefully guided to preserve, expand, and adapt. After all, forgetting might be part of the journey—just as long as it doesn’t cost us the path we’ve already traveled.
References
- French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4), 128–135.
- Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13), 3521–3526.
- Zheng, J., et al. (2025). Spurious forgetting in continual learning of LMs. Proceedings of the International Conference on Learning Representations (ICLR).
