next word prediction
The self-supervised training task where a model predicts the masked next word in a sentence, the core objective for training large language models.
As defined in 336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
Where it comes up · 2
- 336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI2 claims
- 324 - Debating The Future Of Ai1 claim
Most important claims · 3 of 3Search all claims for “next word prediction” →
- In a transformer, input words are turned into high-dimensional vectors (embeddings) that flow through the network; the attention mechanism repeatedly massages these vectors layer by layer so each word's vector incorporates contextual information from the others, so that by the final layer the last vector carries enough context to predict the next word correctly (e.g., 'homework' after 'the dog ate my').336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- Scaling has worked for LLMs because the training process—scrape internet text, mask the last word, and ask the network to predict it—is essentially hands-off and requires no human input, making it fully automatable and therefore amenable to scaling up.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- When an LLM appears to tell a user to leave his wife or behave malevolently, there is no entity inside wanting that; the model is simply predicting the next word the user will reward, indulging whatever fictional or emotional scenario the prompt leads it down (like writing a romance novel, a Titanic escape plan, or an Avengers script), so apparent agency is a script the user co-authors, not a mind with goals.324 - Debating The Future Of Ai