The decoder in Transformers uses two types of attention: (1) cross-attention where the decoder embedding acts as query and attends to encoder state embeddings, and (2) masked self-attention (causal attention) where the decoder attends to target sequence tokens but only those that have appeared in the past, with future tokens masked away.
factualpending
Speaker
Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]Evidence Quote
“the decoder uses two types of attention much like the decoder architectures of lstm based sequence sequence models the decoder embedding is treated as the query and provides attention to the encoder's state embeddings the decoder also provides attention to the embeddings of the target sequence however only to tokens that have appeared in the past the tokens in the future are masked away”
Source
10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVBCreated: 8/11/2026, 1:00:15 AM
Your notes
Loading notes...