Tokenization splits raw text into multiple tokens which are then inputted for processing into a machine learning model, with most modern models using subword tokenizers that split rare words into smaller sub-word units while frequently occurring words are inserted as whole units.
factualpending
Speaker
Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]Evidence Quote
“we split the raw text into multiple tokens which are then inputted for processing into an ml model most models today use a subword tokenizer that splits rare words into smaller sub word units while frequently occurring words are inserted as whole for processing”
Source
10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVBCreated: 8/11/2026, 1:00:15 AM
Your notes
Loading notes...