attention score
A numerical value computed by taking the dot product of a query vector with a key vector, representing how much focus or relevance should be placed on a particular element relative to others
As defined in Illustrated Guide to Transformers Neural Network: A step by step explanation
Where it comes up · 1
Most important claims · 7 of 7Search all claims for “attention score” →
- The decoder's first multi-headed attention layer operates differently from the encoder because the decoder is autoregressive and generates sequences word-by-word, requiring it to prevent attending to future tokens that haven't been generated yet.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The attention scores are scaled down by dividing them by the square root of the dimension of the queries and keys in order to allow for more stable gradients, as multiplying values can have exploding effects.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Masking is the only difference in how attention scores are calculated in the decoder's first multi-headed attention layer compared to the encoder, with masks applied before getting concatenated and fed through a linear layer.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The decoder input goes through embedding and positional encoding layers to create positional embeddings before processing through the first multi-headed attention layer which computes attention scores for the decoder's input.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The softmax function is applied to scaled attention scores to produce attention weights with probability values between 0 and 1, where higher scores are heightened and lower scores are depressed, allowing the model to be more confident about which words to attend to.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Look-ahead masking prevents the decoder from attending to future tokens by adding a mask matrix before calculating softmax and after scaling scores, filling the top right triangle with negative infinity so that softmax converts these to zero attention scores for future tokens.Illustrated Guide to Transformers Neural Network: A step by step explanation
- In the decoder, attention scores for a word like 'am' have values for itself and all other words before it, but zero attention scores for future words like 'fine', which tells the model to put no focus on those words.Illustrated Guide to Transformers Neural Network: A step by step explanation