One of BERT's pre-training tasks was masked language modeling, where random tokens are masked from the input sequence and the network is tasked to predict these tokens using bi-directional self-attention on all other tokens in the sequence.
factualpending
Speaker
Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]Evidence Quote
“one of the pre-training tasks bird was trained on was called mask language modeling random tokens are masked from the input sequence and the network is tasked to predict these tokens by providing bi-directional self-attention”
Source
10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVBCreated: 8/11/2026, 1:00:15 AM
Your notes
Loading notes...