self-attention
The transformer mechanism (QK^T softmax times V) that lets each token weigh its relationship to all others, capturing higher-order dependencies.
As defined in Aravind Srinivas
Other definitions
Mechanism in transformers where each token weighs every other token in the sequence, with cost growing quadratically with sequence length.
Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]
Where it comes up · 4
- 10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT5 claims
- Illustrated Guide to Transformers Neural Network: A step by step explanation4 claims
- The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap4 claims
- Aravind Srinivas1 claim
Most important claims · 10 of 14Search all claims for “self-attention” →
- The key insight enabling transformers was masking: by using masked convolutions (WaveNet) and then attention, you can backpropagate through every input token in parallel rather than sequentially through time, utilizing GPUs far more efficiently — turning what was an eight-day job into a single day — and attention makes even better use of hardware than convolutions because it applies more compute per flop while learning higher-order dependencies.Aravind Srinivas
- To achieve self-attention, the input is fed into three distinct fully connected layers to create query, key, and value vectors, which follow the same concept as a retrieval system where queries are mapped against keys to find relevant matches.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Self-attention allows a model to associate each individual word in the input to other words in the input, enabling the model to learn relationships like associating the word 'you' with 'how' and 'are' in a question pattern.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Decoder masked self-attention (causal attention) is critical because it ensures that during training the model can only attend to tokens that have already been generated and not to future tokens, preventing the model from cheating by predicting future tokens which would not be possible during inference.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Self-attention alleviates the long-term dependency issues of ordinary RNNs because models get a holistic view of the entire sequence and can attend to any token regardless of its position in the sequence.10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT
- Multi-headed attention involves splitting the query, key, and value vectors into multiple sub-vectors before applying self-attention, where each split undergoes the same self-attention process as a separate 'head' that learns different attention representations.Illustrated Guide to Transformers Neural Network: A step by step explanation
- In self-attention, queries and keys undergo a dot product matrix multiplication to produce a score matrix that determines how much focus should be placed on each word relative to other words, with higher scores indicating more focus.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The self-attention mechanism works by picking out the most important words in a sentence, figuring out how they are connected, and then determining the overall meaning of the sentence—for example, in 'the cat is sitting on the mat, it is comfortable,' AI must understand that 'it' refers to the cat, not the mat.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- The self-attention mechanism in Transformers works by identifying the most important words in a sentence, figuring out how they are connected, and then determining the overall meaning—for example, in 'The cat is sitting on the mat, it is comfortable,' self-attention allows AI to correctly identify that 'it' refers to 'cat' not 'mat' because cat is more closely connected to the word comfortable.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap
- AI training involves four main steps: (1) automatically encoding words as numeric vectors learned through analyzing vast amounts of text; (2) self-supervised learning through fill-in-the-blank games where AI guesses missing words and adjusts vectors when wrong; (3) using self-attention to focus on relevant words; and (4) continuous training on billions of sentences, updating parameters with each round.The AI Shortcut: How DeepSeek Cracked ChatGPT’s Secrets for Cheap