One of BERT's pre-training tasks was masked language modeling, where random tokens are masked from the input sequence and the network is tasked to predict these tokens using bi-directional self-attention on all other tokens in the sequence.

factualpending

Speaker

Unidentified Speaker — 10 years of NLP history explained in 50 concepts | From Wor… [uocYQH0cWTs]

Evidence Quote

“one of the pre-training tasks bird was trained on was called mask language modeling random tokens are masked from the input sequence and the network is tasked to predict these tokens by providing bi-directional self-attention”

Source

10 years of NLP history explained in 50 concepts | From Word2Vec, RNNs to GPT— Neural Breakdown with AVB
Created: 8/11/2026, 1:00:15 AM

Your notes

Loading notes...