Large neural networks do not overfit quickly because stochastic gradient descent finds approximately the point with smallest norm in the high-dimensional subspace of solutions achieving zero training error, and this minimum-norm solution is provably insensitive to small random variations in the training data when dimensionality is high.

causalpending

Speaker

Ilya Sutskever

Evidence Quote

“you got a huge neural network you have a huge number of parameters and now let's pretend everything is linear then there is this big subspace where a neural network achieves zero error and sgd is going to find approximately the point that's right approximately the point with the smallest norm in that subspace okay and that can also be proven to be insensitive to the small randomness in the data when the dimensionality is high”

Source

Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94— Lex Fridman
Created: 8/10/2026, 10:49:52 PM

Your notes

Loading notes...