definition
LLMs output probability distributions over vocabulary
A language model's output is a probability distribution over its vocabulary for the next token, from which a token is sampled; during training the target distribution is one for the correct word and zero elsewhere, and backpropagation over hundreds of billions to a trillion parameters gradually nudges the predicted distribution toward the correct word across every sentence scraped from the internet.
definitionpending
Speaker
Anil AnanthaswamyEvidence Quote
“what the language model is predicting at the very end is a probability distribution over its vocabulary.”
Created: 6/13/2026, 12:20:04 AM
Your notes
Loading notes...