The decoder's first multi-headed attention layer operates differently from the encoder because the decoder is autoregressive and generates sequences word-by-word, requiring it to prevent attending to future tokens that haven't been generated yet.
causalpending
Speaker
Unidentified Speaker — Illustrated Guide to Transformers Neural Network: A step by… [4Bdc55j80l8]Evidence Quote
“this multi-headed attention layer operates slightly different since the decoders autoregressive and generates the sequence word-by-word you need to prevent it from condition into future tokens for example when computing attention scores on the word am you should not have access to the word fine”
Created: 8/13/2026, 9:51:15 AM
Your notes
Loading notes...