dropout
A regularization technique that randomly hides parts of the network during training to prevent overfitting; originally developed for vision, underused for text.
As defined in Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Where it comes up · 2
- Machine Intelligence and Risk Management2 claims
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI1 claim
Most important claims · 3 of 3Search all claims for “dropout” →
- Dropout randomly breaks parts of a model during training, forcing it to solve problems in different or suboptimal ways, which yields a more robust, generalizable model than training on perfectly clean data.Machine Intelligence and Risk Management
- Current pre-training likely doesn't extract maximal value from every token, because predicting the next token from previous ones isn't really how people learn; the model should think much harder on some tokens than others (do a lot more work when reaching 'the answer is' than 'the'), learn data forwards and backwards every which way, and use techniques like dropout and many epochs—so the claim that we're running out of text data is doubtful since far more capability can be extracted from existing text.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- A little stress is healthy for learning: it forces people to weigh routes differently, favoring the faster solution over the complete one, much as noise improves machine models.Machine Intelligence and Risk Management