curse of dimensionality
The phenomenon where, as the number of dimensions grows, all data points become roughly equidistant, breaking distance-based notions of similarity.
As defined in 336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
Where it comes up · 2
Most important claims · 2 of 2Search all claims for “curse of dimensionality” →
- As the number of dimensions (features) grows, distance-based methods like k-nearest-neighbor fail because in high dimensions everything is roughly as far from everything else, so the notion that similar data points are closer than dissimilar ones collapses—this is the curse of dimensionality, predating and orthogonal to neural networks.336 | Anil Ananthaswamy on the Mathematics of Neural Nets and AI
- The space of all possible prompts is so gigantic that fine-tuning can only cover a tiny subset; there is a long tail of prompts the system was never trained on, so an LLM can always be broken into producing nonsense, making it effectively a giant lookup table rather than a reasoning system.Research Pioneer