softmax
A mathematical function that converts a vector of numbers into a probability distribution where each element is proportional to its exponential, ensuring outputs sum to 1
As defined in rGgGOccMEiY
Other definitions
A function that converts a vector of scores into a probability distribution; a key component of the attention operation and the cerebellar circuit approximation.
Sholto Douglas & Trenton Bricken — How LLMs actually think
Where it comes up · 2
- Illustrated Guide to Transformers Neural Network: A step by step explanation6 claims
- Sholto Douglas & Trenton Bricken — How LLMs actually think1 claim
Most important claims · 7 of 7Search all claims for “softmax” →
- Pentti Kanerva's 1980s associative memory algorithm, when implemented as an electrical circuit, looks identical to the core cerebellar circuit found in nearly every organism (including the Drosophila mushroom body), and Bricken's work shows the attention operation is a close approximation of this same circuit—a three-way convergence suggesting attention's success is not accidental.Sholto Douglas & Trenton Bricken — How LLMs actually think
- The decoder takes the index of the highest probability score from the softmax layer to identify the predicted word, adds the predicted output to the list of decoder inputs, and continues decoding until an end token is predicted.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The attention weights from softmax are multiplied by the value vector to produce an output vector, where higher softmax scores preserve the value of words the model has learned to prioritize as important, while lower scores drown out irrelevant words.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The softmax function is applied to scaled attention scores to produce attention weights with probability values between 0 and 1, where higher scores are heightened and lower scores are depressed, allowing the model to be more confident about which words to attend to.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The decoder's final point-wise feed-forward layer output goes through a final linear layer that acts as a classifier with classes equal to the vocabulary size, with softmax producing probability scores between 0 and 1 for each class.Illustrated Guide to Transformers Neural Network: A step by step explanation
- The decoder's job is to generate text sequences by taking the encoder's output and previously generated outputs as inputs, using two multi-headed attention layers, a point-wise feed-forward layer, a linear classifier, and softmax to predict output words.Illustrated Guide to Transformers Neural Network: A step by step explanation
- Look-ahead masking prevents the decoder from attending to future tokens by adding a mask matrix before calculating softmax and after scaling scores, filling the top right triangle with negative infinity so that softmax converts these to zero attention scores for future tokens.Illustrated Guide to Transformers Neural Network: A step by step explanation