tokens
The basic units of text (words or word-fragments) that AI language models process and learn from.
As defined in Brendan Foody
Other definitions
Units of text that a language model processes, which can be individual words or parts of words (like punctuation or word fragments); the text shown to a model is converted into tokens
The Future of Artificial Intelligence
Where it comes up · 33
- 90 - 5 Mental Models for Web3 | Chris Dixon9 claims
- Feed Your OWN Documents to a Local Large Language Model!4 claims
- Dwarkesh Patel Podcast - The data black hole at the center of AI3 claims
- AI Trends 2025: Mary Meeker Deck Deep Dive Part 12 claims
- China is killing the US on energy. Does that mean they’ll win AGI? — Casey Handmer2 claims
- Dwarkesh Patel Podcast - Dario Amodei — "We are near the end of the exponential"2 claims
- The Future of Artificial Intelligence2 claims
- The Strange Math That Predicts (Almost) Anything2 claims
and 25 more
Most important claims · 10 of 52Search all claims for “tokens” →
- Current LLMs may mainly learn the structure embedded in roughly 3,000 years of human-written tokens rather than learning the world itself, so their capabilities reflect human curation of language rather than understanding of physical reality.Martin Casado - Entering Uncharted AI Territory - [Invest Like the Best, EP.381]
- Modern large language models don't treat all tokens equally like simple Markov chains do; they also use something called attention, which tells the model what to pay attention to, allowing them to use distant context like previous words 'blood' and 'mitochondria' to know that 'cell' in the phrase 'the structure of the cell' refers to biology rather than a prison.The Strange Math That Predicts (Almost) Anything
- Data center power use has decreased 43% over 8 years while per-unit energy efficiency has improved 50,000x, and tokens per megawatt per year have scaled from 1.3 billion to 65 trillion, demonstrating dramatic improvements in compute efficiency that are underreported in energy consumption headlines.AI Trends 2025: Mary Meeker Deck Deep Dive Part 1
- Blind and deaf people who lack a sense still have general intelligence equal to everyone else, and deaf people who communicate only via sign language and reading ingest far fewer than 200 million language tokens yet are fully general intelligences, suggesting sensory tokens are not what makes humans smart.Dwarkesh Patel Podcast - The data black hole at the center of AI
- Model commoditization is the firm's main worry about AI: frontier model companies lack the proprietary data of Google, Amazon, Microsoft and Meta, and cheap Chinese or European open models mean businesses may not pay for OpenAI or Anthropic tokens.Why Now is the Best Time to Buy Public Software Companies
- Models will struggle in law for a long time because much of the right way to approach legal problems is not written down or codified—it exists in the heads of experts—and without those tokens in pre-training data or expert-created post-training data, the model will inevitably struggle.Brendan Foody
- ChatGPT does not process words as letters but instead breaks them down into tokens represented by numbers, which explains why it struggles with character-level tasks like counting Rs in a word.This is not the AI we were promised | The Royal Society
- Specific tokens tied to quirks of training cause AIs to behave bizarrely - on early GPT-3, the string 'petertodd' could not be repeated and would make the model go crazy, declaring nothing is real in a world of madness and speaking in all caps - revealing weird machinery under the helpful-assistant facade.434 - Can We Survive Ai
- Cryptocurrencies grow without marketing spend by creating skin-in-the-game ownership psychology where users evangelize unpaid because they hold tokens, while Web2 requires 50%+ of revenue spent on sales/marketing to achieve similar adoption90 - 5 Mental Models for Web3 | Chris Dixon
- The game for modern language model text prediction is: given a string of tokens (perhaps 30 long), what are the odds that the next token is this one, or that one, or the other one.The Strange Math That Predicts (Almost) Anything