Instead of using absolute positional embeddings as in the original Transformer, relative positional embeddings enable the model to learn positional information of tokens relative to each other and allow attention to tokens in only a fixed-length context window.

factualpending

Speaker

Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]

Evidence Quote

“instead of the absolute positional embeddings the original Transformer applied to each token Transformer Excel introduced the idea of relative positional embeddings which enables the model to learn positional information of tokens relative to each other”

Source

10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVB
Created: 8/11/2026, 1:00:15 AM

Your notes

Loading notes...