factual

Subword Tokenization Avoids Out-of-Vocabulary Words

Language models process text as subword tokens rather than whole words (e.g., 'freedom' as 'free' + 'dom') because this empirically helps avoid out-of-vocabulary problems—if a novel word shares a subword unit with known tokens, the model can still process it.

factualpending

Speaker

Raphael Milliere

Evidence Quote

“actually receive what's known as tokens where sometimes a single word can be broken down into different subword tokens.”

Source

230 | Raphaël Millière on How Artificial Intelligence Thinks— Sean Carroll's Mindscape
Created: 6/13/2026, 12:19:51 AM

Your notes

Loading notes...