Modern large language models don't treat all tokens equally like simple Markov chains do; they also use something called attention, which tells the model what to pay attention to, allowing them to use distant context like previous words 'blood' and 'mitochondria' to know that 'cell' in the phrase 'the structure of the cell' refers to biology rather than a prison.

definitionpending

Speaker

Derek Muller

Evidence Quote

“they also use something called attention, which tells the model what to pay attention to. So in the phrase, 'the structure of the cell,' the model can use previous context like blood and mitochondria to know the cell most likely refers to biology rather than a prison cell.”

Source

The Strange Math That Predicts (Almost) Anything— Veritasium
Created: 8/11/2026, 5:34:40 AM

Your notes

Loading notes...