definition

LLMs output probability distributions over vocabulary

A language model's output is a probability distribution over its vocabulary for the next token, from which a token is sampled; during training the target distribution is one for the correct word and zero elsewhere, and backpropagation over hundreds of billions to a trillion parameters gradually nudges the predicted distribution toward the correct word across every sentence scraped from the internet.

definitionpending

Speaker

Anil Ananthaswamy

Evidence Quote

“what the language model is predicting at the very end is a probability distribution over its vocabulary.”

Source

336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI— Sean Carroll's Mindscape
Created: 6/13/2026, 12:20:04 AM

Your notes

Loading notes...