perplexity
A measure of how well a model predicts text; lower perplexity means better next-token prediction.
As defined in Sholto Douglas & Trenton Bricken — How LLMs actually think
Other definitions
AI-powered search application that uses large language models to provide natural language answers to queries with source citations
Doomberg Riffs: Chips, China & Power Plays: The Big Bets on a Multipolar World
Where it comes up · 4
- Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government1 claim
- Sholto Douglas & Trenton Bricken — How LLMs actually think1 claim
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development1 claim
- Doomberg Riffs: Chips, China & Power Plays: The Big Bets on a Multipolar World1 claim
Most important claims · 4 of 4Search all claims for “perplexity” →
- Giving a model millions of tokens of context (e.g. a code base) produces a step-up in intelligence equivalent to large increases in model scale, because it solves the onboarding problem instantly without retraining—throwing in context dramatically improves next-token prediction the way scale normally would.Sholto Douglas & Trenton Bricken — How LLMs actually think
- Benchmark scores are a terrible way to assess the overall rate of AI progress, and smoother internal metrics such as pre-training perplexity and RL compute efficiency give a better read on how fast the field is moving.Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government
- Pretraining progress is extremely easy to verify because loss on a held-out test set is a direct measure, so GPT-7 could be RL-trained to minimize compute needed to reach a fixed loss, and swarms could search for the lowest perplexity.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- Perplexity AI app hallucinates significantly and is tougher to argue against than ChatGPT due to higher hallucination rate, but it's still a powerful tool for research.Doomberg Riffs: Chips, China & Power Plays: The Big Bets on a Multipolar World