causal

Architecture Not the Continual Learning Bottleneck

Architecture does not seem to be the fundamental bottleneck for continual learning, since many working ideas such as sparse attention and KV cache compaction already exist, so the loss function may be the real bottleneck.

causalpending

Speaker

Dwarkesh Patel

Evidence Quote

“Perhaps the bottleneck is the loss function.”

Source

Dwarkesh Patel Podcast - The next big breakthrough will be AIs learning on the job— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:25:50 AM

Your notes

Loading notes...