Double descent is a phenomenon where test error exhibits a non-monotonic U-shape as model size increases while data is fixed: performance improves as networks grow, then worsens sharply around the interpolation threshold (where model parameters equal training examples), then improves again as model size continues to grow, and this pattern occurs across virtually all practical deep learning systems without early stopping.

factualpending

Speaker

Ilya Sutskever

Evidence Quote

“when you increase the size of the neural network slowly and if you don't do early stopping that's a pretty important detail then when the neural network is really small you make it larger you get a very rapid increase in performance then you continue to make it large and at some point performance will get worse and it gets and and it gets the worst exactly at the point at which it achieves zero training error precisely zero training loss and then as you make it large it starts to get better again”

Source

Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94— Lex Fridman
Created: 8/10/2026, 10:49:52 PM

Your notes

Loading notes...