Tokenization splits raw text into multiple tokens which are then inputted for processing into a machine learning model, with most modern models using subword tokenizers that split rare words into smaller sub-word units while frequently occurring words are inserted as whole units.

factualpending

Speaker

Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]

Evidence Quote

“we split the raw text into multiple tokens which are then inputted for processing into an ml model most models today use a subword tokenizer that splits rare words into smaller sub word units while frequently occurring words are inserted as whole for processing”

Source

10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVB
Created: 8/11/2026, 1:00:15 AM

Your notes

Loading notes...