causal

Masking enabled parallel training of transformers

The key insight enabling transformers was masking: by using masked convolutions (WaveNet) and then attention, you can backpropagate through every input token in parallel rather than sequentially through time, utilizing GPUs far more efficiently — turning what was an eight-day job into a single day — and attention makes even better use of hardware than convolutions because it applies more compute per flop while learning higher-order dependencies.

causalpending

Speaker

Aravind Srinivas

Evidence Quote

“whatever job was earlier running in eight days would run in a single day. I think that was the most important insight.”

Source

Aravind Srinivas— Lex Fridman Podcast
Created: 6/13/2026, 3:26:05 AM

Your notes

Loading notes...