factual

Non-linearity makes loss landscapes jagged with no guaranteed global minimum

Because modern neurons include non-linearities (e.g., sigmoids) and networks are enormous, the loss function over a trillion parameters is non-convex and jagged rather than smooth; there is no guarantee it has a global minimum, so training relies on gradient descent finding a satisfactory local minimum among many.

factualpending

Speaker

Anil Ananthaswamy

Evidence Quote

“we don't even know for sure that they have a global minimum, but they may have lots and lots of very good optimal local minima. And so the trick is to somehow use gradient descent to find one of these local minima that is satisfactory”

Source

336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI— Sean Carroll's Mindscape
Created: 6/13/2026, 12:20:04 AM

Your notes

Loading notes...