causal
Masking enabled parallel training of transformers
The key insight enabling transformers was masking: by using masked convolutions (WaveNet) and then attention, you can backpropagate through every input token in parallel rather than sequentially through time, utilizing GPUs far more efficiently — turning what was an eight-day job into a single day — and attention makes even better use of hardware than convolutions because it applies more compute per flop while learning higher-order dependencies.
causalpending
Speaker
Aravind SrinivasEvidence Quote
“whatever job was earlier running in eight days would run in a single day. I think that was the most important insight.”
Created: 6/13/2026, 3:26:05 AM
Your notes
Loading notes...