factual
Subword Tokenization Avoids Out-of-Vocabulary Words
Language models process text as subword tokens rather than whole words (e.g., 'freedom' as 'free' + 'dom') because this empirically helps avoid out-of-vocabulary problems—if a novel word shares a subword unit with known tokens, the model can still process it.
factualpending
Speaker
Raphael MilliereEvidence Quote
“actually receive what's known as tokens where sometimes a single word can be broken down into different subword tokens.”
Created: 6/13/2026, 12:19:51 AM
Your notes
Loading notes...