definition
Mid-Training Combats Catastrophic Forgetting
Mid-training uses the same algorithm as pre-training but is more specialized—e.g. focusing on long-context documents—and is positioned so the quality content is the last thing the model sees, because neural networks suffer catastrophic forgetting where learning something causes other things to be forgotten.
definitionpending
Speaker
Sebastian RaschkaEvidence Quote
“one problem of LLMs is still that it’s a neural network; it has the problem of catastrophic forgetting. So, you teach it something, it forgets other things.”
Created: 6/13/2026, 3:24:41 AM
Your notes
Loading notes...