155 claims in “machine learning”
A dataset-learning strategy is to first over-parameterize and overfit to prove a learning relationship exists, then compress to the smallest parameter count, since compression is what produces generalization.
Continual learning turned out to be harder than solving Navier-Stokes, as shown by labs not yet cracking it, which means expectations about which problem is harder were wrong.
Choosing which objective to optimize is part of research taste, but paradigm-agnostic outer-loop proxies such as Millennium Prize solves, time to rediscover general relativity from pre-1905 data, and speed of mastering unseen games can verify progress across paradigms.
Because models generalize far beyond what they were trained on, models trained on less ambitious verifiable math problems can solve problems no human had cracked, so the same generalization should make models excellent at unverifiable parts of AI research like research taste.
As pretraining improves, smaller models can match larger ones while training faster and cheaper, which speeds experiments and lets labs train models at a scale where thousands of experiments run daily, so small-scale results need not be bet on bigger models.
Current models are solving Millennium Prize problems while still being savant interns without long-term memory, and if they could actually run, given more capability, they would be faster, like a cartwheeling racer still keeping pace.
AI R&D has three broad buckets, pretraining, post-training, and inference efficiency, each with strong verifiable proxies, and since together they comprise all algorithmic progress, AI progress is generally easy to train against.
Pretraining improvements yield very general gains in intelligence because of a deep link between compression and intelligence, so a pretraining-automation loop makes the model broadly better.
All the progress made in deep learning over the last decade and a half has not required much new theory, meaning empirical recipe-finding has driven the field.
The transformer was a natural combination of ideas, such as attention and residual networks, that had independently been effective, so none of its core pieces were surprising to researchers in the field.
The growth of a business and its segments is relatively stationary, so machine learning works well on it, while asset prices are non-stationary, driven by mood and regime, making price prediction with machine learning a poor application.
Using machine learning to predict stock prices is mostly an ill-posed problem, since no known mapping exists from input variables to price, unlike recognizing a cat in a video, so the assumption of such a mapping is misleading.
Models predict the lower half of a quality distribution (bad essays, business deterioration) more accurately than the upper half, because experts agree about what is wrong but what makes something excellent varies widely across observers.
Hinton's restricted Boltzmann machines trained on handwriting could generate digits from random noise, morphing a seven into a nine and a one into a two through smooth transitions they had never seen, which Lowin initially took as intelligence before deciding it was a deterministic outcome of model construction.
Lowin once believed the dreaming Boltzmann machine demonstrated intelligence but now considers it a deterministic outcome of its construction, and he thinks later models far exceed it.
Using simple models such as linear regression, or iteratively solved models with only as many parameters as regressors, regularizes the model and reduces overfitting, which Lowin considers the greatest danger in quantitative finance.
Iterative machine learning models with a well-defined error term can be pushed to tune parameters a billion times, so they can overfit the data badly.
Recurrent neural networks process data as ordered sequences, and because order equates to context, building up context word by word yields much richer representations than mapping a whole sequence in one shot.
Dropout randomly breaks parts of a model during training, forcing it to solve problems in different or suboptimal ways, which yields a more robust, generalizable model than training on perfectly clean data.
Linear regression has a closed-form solution, whereas machine learning is a class of iterative models where no closed-form solution is known, but a well-defined error term is decreased step by step by adjusting parameters.
Brute force search wastes time on the vast majority of a search space, and evolutionary breeding of candidates is random and hard to interpret, so machine learning favors optimization with known derivatives of the error, which lets practitioners know at each step how the model improves.
JEPA (joint embedding predictive architecture) is an architecture that predicts abstract representations of future states rather than reconstructing raw sensory details, eliminating the need to predict unpredictable information by focusing only on predictable, relevant information.
Generative models that attempt to predict every pixel or detail of future video or high-dimensional continuous data produce blurry, averaged predictions and fail because most details in sensory data are fundamentally unpredictable.
Training JEPA architectures faces the 'collapse' problem: if you simply minimize prediction error in representation space, the system can cheat by predicting constant representations, making the prediction trivial and useless.
Sigreg (Sketch anisotropic Gaussian regularization) prevents JEPA collapse by maximizing information content in encoder outputs through a regularization method based on projecting distributions along multiple axes and ensuring marginal distributions match isotropic Gaussians.
DINO (Self-Supervised Vision Transformers for Image Representation) is the state-of-the-art method for extracting generic features from images without labels, now at version DINO V3, and represents the best system globally for image representation learning.
Reinforcement learning should be minimized or abandoned in favor of world models because it is extremely sample-inefficient and impractical for achieving human or animal-level intelligence.
Energy-based models offer a better paradigm than probabilistic models for capturing dependencies between variables in high dimensions because representing exact joint distributions is intractable.
The credit assignment problem is the fundamental challenge that computational systems must solve: determining which parameters to adjust and by how much to achieve a desired output
A neuron's predicted activity is given by the weighted sum of activities of upstream neurons, determined by synaptic weights which are typically followed by a nonlinear activation function (like sigmoid or ReLU), though the speaker simplifies this for pedagogical clarity
If predictive coding networks were allowed to freely adjust every parameter (both neural activities and weights), they would naturally settle to a zero energy state which would be trivial and not perform any meaningful computation, so in practical implementations and likely in the brain, certain neurons are clamped to specific values
Repeating the iterative relaxation process across diverse examples gradually refines the network's internal model of the world, developing compressed representations of data
In real models with nonlinear activation functions, the update rules for opposing synapses are not mathematically identical, but research suggests perfect symmetry may not be essential; approximate symmetry emerging from independent learning is sufficient for effective network function.
Backpropagation with gradient descent is the workhorse algorithm that powers virtually the entire field of machine learning today
Predictive coding's local autonomy makes the algorithm extremely parallelizable and in certain settings more efficient than back propagation, and theoretical considerations suggest that resulting updates may lead to better solutions than back propagation.
The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.
LeJepa can train JEPA-style models without relying on EMA while still avoiding collapse and learning strong representations, achieving competitive or better performance compared to VICReg, Barlow Twins, and SimCLR on standard vision benchmarks, with accuracy levels similar to state-of-the-art DINO on ImageNet.
The idea of a cat sitting on a couch can be captured in a high-dimensional latent space with all kinds of sentences or images that show this idea pointing to that spot in the latent space, demonstrating that JEPA's representation captures semantic invariants across diverse inputs.
The manifold hypothesis states that all natural data lies on a low-dimensional manifold within the high-dimensional space where it is encoded; this is accurate for any perception problem including MNIST digits, human faces, voice sounds, and natural language.
Within a manifold, it is always possible to interpolate between two inputs by morphing one into another via a continuous path along which all points fall on the manifold; the ability to interpolate between samples is the key to understanding generalization in deep learning because it allows making sense of points never seen by relating them to nearby points on the manifold.
The power to generalize in deep learning is much more a consequence of the natural structure of your data than a property of your model; you will only be able to generalize if your data follows the manifold hypothesis, and the more informative and less noisy your data, the better you can generalize.
A listener named Shawn solved the Jane Street puzzle by decomposing it into two parts: first identifying which layers pair together by looking for negative diagonal patterns in weight matrices, then ordering the pairs by sorting blocks by residual contribution size, combined with local swaps for refinement.
Training world models using generative approaches (predicting pixel-level details) fails because it is impossible to predict all plausible futures in video; there are infinite possible continuations and the system learns to predict the average, resulting in blurry outputs
Video is more redundant than text, which is actually a feature rather than a bug for self-supervised learning because redundancy in data is necessary for learning; without redundancy you cannot learn anything
Joint Embedding Predictive Architecture (JEPA) encodes both input X and future observation Y before prediction, allowing the system to eliminate unpredictable information and produce more abstract, accurate representations compared to generative models that reconstruct raw pixels
Generative models (autoencoders) trained on images do not produce good representations for downstream tasks; joint-embedding architectures are superior and all the best self-supervised image and video representation systems use joint-embedding, not reconstruction
Joint-embedding systems can collapse by ignoring inputs and producing constant representations; preventing collapse is the central problem in self-supervised learning for these systems and requires either contrastive methods (pushing up energy of non-data points) or regularized methods (minimizing volume of low-energy regions)
Techniques like MNCR, NCR squared, WMSE, Sigreg, VICReg, and Barlow Twins all implement information maximization for joint-embedding systems, with JEPA gaining popularity (1,700 papers mention it on Google Scholar)
A fundamental challenge with information maximization methods is that proper definitions of information content require knowing the distribution of vectors, which we do not have; empirical measures are all upper bounds, not lower bounds, so methods use good upper bounds and cross their fingers
Information can be maximized by ensuring all rows of a representation matrix are different (making each sample distinct) or all columns are different (making each variable independent), with the latter (dimension contrastive methods) being preferable because independent variables are more informative