definition

Mid-Training Combats Catastrophic Forgetting

Mid-training uses the same algorithm as pre-training but is more specialized—e.g. focusing on long-context documents—and is positioned so the quality content is the last thing the model sees, because neural networks suffer catastrophic forgetting where learning something causes other things to be forgotten.

definitionpending

Speaker

Sebastian Raschka

Evidence Quote

“one problem of LLMs is still that it’s a neural network; it has the problem of catastrophic forgetting. So, you teach it something, it forgets other things.”

Source

Ai Sota #2026— Lex Fridman Podcast
Created: 6/13/2026, 3:24:41 AM

Your notes

Loading notes...