The decoder's first multi-headed attention layer operates differently from the encoder because the decoder is autoregressive and generates sequences word-by-word, requiring it to prevent attending to future tokens that haven't been generated yet.

causalpending

Speaker

Unidentified Speaker — Illustrated Guide to Transformers Neural Network: A step by… [4Bdc55j80l8]

Evidence Quote

“this multi-headed attention layer operates slightly different since the decoders autoregressive and generates the sequence word-by-word you need to prevent it from condition into future tokens for example when computing attention scores on the word am you should not have access to the word fine”

Source

Illustrated Guide to Transformers Neural Network: A step by step explanation— The AI Hacker
Created: 8/13/2026, 9:51:15 AM

Your notes

Loading notes...