factual
Non-linearity makes loss landscapes jagged with no guaranteed global minimum
Because modern neurons include non-linearities (e.g., sigmoids) and networks are enormous, the loss function over a trillion parameters is non-convex and jagged rather than smooth; there is no guarantee it has a global minimum, so training relies on gradient descent finding a satisfactory local minimum among many.
factualpending
Speaker
Anil AnanthaswamyEvidence Quote
“we don't even know for sure that they have a global minimum, but they may have lots and lots of very good optimal local minima. And so the trick is to somehow use gradient descent to find one of these local minima that is satisfactory”
Created: 6/13/2026, 12:20:04 AM
Your notes
Loading notes...