The attention mechanism (introduced in 2015) allows the decoder to selectively focus on specific tokens in the encoder sequence instead of relying on a single encoder output by having the decoder state act as a query vector, encoder states as key vectors, computing attention scores for each query-key pair, and forming context vectors as weighted averages of encoder states based on these scores.

factualpending

Speaker

Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]

Evidence Quote

“the decoder can now selectively focus on specific tokens in the encoder sequence instead of relying on a single encoder output at each step the decoder can form new context vectors by combining relevant hidden States from the input sequence specifically the decoder State at each time step is treated as a query vector and all the encoder states are considered key vectors”

Source

10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVB
Created: 8/11/2026, 1:00:15 AM

Your notes

Loading notes...