
Inference, Diffusion, World Models, and More | YC Paper Club
What this covers
This is the first session of YC Paper Club, held May 20, 2026, at YC's Mountain View office. It runs as a sequence of five short paper presentations, each followed by discussion. A YC partner opens by framing the group as a way for researchers, engineers, and founders to keep up with the literature. He closes on two problems he considers still open: intelligence per watt and intelligence per sample, where current systems are orders of magnitude behind humans.
The first two talks are about inference and control. Tanishk presents work on speculative speculative decoding. He starts from the asymmetry behind speculative decoding: a transformer can verify many tokens in one forward pass but has to generate them one at a time. He argues that inference will come to be seen as a capability lever rather than a cost lever, because if performance scales with thinking time, tokens per second sets the intelligence delivered. That forecast is flagged as contested. Stannis then covers Diffusion-MPC. Its point is that model predictive control splits a problem into a dynamics model and an action selector, so a planner can pursue a new reward at test time without retraining.
Isaac Ward presents LeWorldModeling and sets world model approaches against model-free policies. He covers collapse as the central training hazard, a JEPA-style architecture with a Gaussian regularizer on latent space, a claimed 50x speed advantage, and the safety value of measuring modeling error. He also notes that the idea goes back to Richard Sutton's 1990 work.
The last two talks concern generalization and data limits. Ashe argues that the Pack-Bayes bound explains why overparameterization works. Ku Woo closes with pretraining under infinite compute. Data grows about 3% a year while pretraining compute grows 4–5x. In that regime, heavy weight decay, ensembling, and distillation outperform the standard recipe, and an 8-model ensemble distilled into a 300M-parameter model keeps 83% of the gain.
Modern AI research is advancing through specialized inference optimization, robotics control via world models, and data-efficient pre-training—establishing that algorithmic innovation in computation, control, and learning regimes unlocks new capabilities independent of scaling laws alone.
- Inference speed is becoming a capability lever rather than a cost factor, with speculative decoding parallelization enabling higher reasoning throughput
- World models enable model-based control with explicit uncertainty quantification and adaptation, contrasting with implicit model-free approaches
- Data-constrained pre-training under infinite compute favors regularization, ensembling, and distillation over parameter scaling, achieving 5x efficiency gains
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Inference costs dominate training costs when serving models to billions of users or trillions of tokens, and within training itself, reinforcement learning (which is a wrapper on inference) is starting to exceed the compute requirements of pre-training.
“there is one point I want to make that I don't hear people talk about enough. So things you may have heard are that inference costs are high. They dominate training costs when you're serving a model for billions of users or you know 10 claud code power users. That's trillions of tokens. Um, not only are inference costs dominating training costs, but even within training, RL is starting to exceed the compute requirements of pre-training.”
Inference in one to three years will be seen as a capability lever rather than a cost or convenience lever, because if an algorithm's performance scales with thinking time, then tokens per second directly equals peak intelligence delivered.
“the claim I'm going to make and maybe this is the one thing to take away from the message I'm trying to send in this talk is that inference today is seen as a sort of like cost or convenience lever. But uh in one two or 3 years inference is going to be seen as a capability. And what I mean by that is that if you have a method, an algorithm, a system where its performance scales with the amount of thinking it does, then fundamentally the speed at which you can do inference, the tokens per second is exactly the peak intelligence that you can deliver.”
In speculative decoding, the key asymmetry that makes it work is that verification is easier than generation because the transformer architecture can compute probabilities for many tokens in parallel in one forward pass, while generation must be done autoregressive one token at a time.
“The sort of key asymmetry here, the reason that speculation works is that it is easier to verify than to generate. This is a feature of the transformer architecture where you can get the probabilities for many tokens in a sequence in parallel in one forward pass. Um but you can't generate them in parallel. auto reggressive decoding as uh one at a time.”
Model predictive control (MPC) can adapt to novel reward functions at test time because it factors the control problem into a dynamics model and an action selector mechanism, allowing the planner to optimize for different objectives without retraining the dynamics model.
“So the main advantages of model predictive control is uh it can adapt to normal reward functions at test time. So uh the dynamics model are also easier to learn and generates better than just policies and the action proposal dynamics model factorization also allows easy adaptation to normal dynamics.”
Model free policies and model based policies are two fundamentally different species in terms of whether they maintain explicit representations of future outcomes, with model free policies mapping observations to actions without explicit future prediction and model based policies explicitly predicting future states
“So on the left, model free. The idea is you're taking some observations, you're feeding this into some kind of big neural network potentially with a bunch of interesting learning tricks there, but you're getting some optimal action out. So, it's just mapping between observation and some optimal action. But at no point is there an explicit representation of what the future might look like if you execute that action...And then over on the um other side, model based approaches, right? So now we're saying we're going to train this world model up explicitly and actually use that in our policy to be able to explicitly predict the outcome of potential actions. So yeah, totally like two different species of policies.”
The central challenge in training world models is that the optimization landscape contains many solutions that cause trivial collapse—such as predicting every state is identical—so researchers use various tricks, special methods, or hyperparameter schedules to avoid these minima.
“So you're co-learning representation and dynamics. And there are many solutions in the optimization landscape that will essentially just cause you to do nothing. So for example a a local min minima in the optimization landscape is to say well every state is just the same it's a trivial collapse basically um and there are many techniques in the literature to say how can you avoid these so there are solutions of a variety different kinds that basically say there a way to avoid the collapse associated with training world models”
World models learn the dynamics of a system by predicting how state will change based on current state and action, enabling capabilities including imagined trajectory rollouts, model-based planning, and surprise quantification through modeling error.
“So really it's about learning the dynamics of the world, which is to say we're trying to come up with some model Typically, we're using like a big neural network to predict how a system will change over time based on its inputs. So, you have your current state or scenario using S for notation here. You're playing some action, maybe that's like a movement or a command for a robot, um, or a language command for a robot, and then you're trying to predict like what its outcome is going to be, like what scenario will it end up in once it's executed that action. So, you're really trying to model the system or the environment that the robot is in, modeling the world. It's a world model. Uh, these kinds of models are really cool. They enable a few really interesting capabilities. One of them is generating imagined outcomes.”
Pre-training on OpenAI o1 and Deepseek R1 demonstrated emergence of reasoning capability through reasoning-time compute scaling, with newer large pre-training runs continuing to improve capabilities.
“So in 2020 with GPT3 we had sort of the emergence of incontext learning. In 2022 with Anthropics RHF, we had sort of the advent of alignment. And maybe most notably in 2024 with both 01 from OpenAI and then later Deepseek R1, we had the emergence of reasoning. And in fact, even still today, we see that with these newer and bigger pre-training runs like Mythos and 5.5, the models just continue to keep better.”
Model free approaches show brittleness to out-of-distribution inputs, while model-based approaches can quantify modeling error and adaptation needs, offering a safety advantage for real-world deployment.
“The model free stuff, some of the weaknesses is they show a little bit of brittleleness to out of distribution. Um, model based ones are great because you can kind of quantify modeling error and this is really important when you're deploying things in the real world.”
The 'standard recipe' (scale model size and epoch data until overfitting with early stopping) fails in the compute-unconstrained regime because overfitting increases as models get larger, causing validation loss to increase after a certain point, so higher compute doesn't help.
“And this is sort of the red line which is what we call the standard recipe. And what you'll see with the standard recipe is that even if you are willing to spend more compute, as you train more and more overparameterized models, you start to overfit more quickly and your loss starts to increase after a certain point.”
When pre-training is data-constrained but compute-unconstrained, the core research question becomes: how should you approach pre-training in this regime, which is fundamentally different from the compute-efficient scaling laws (Chinchilla) that have dominated modern pre-training.
“And so this sort of motivates the core question in this paper which is how should you approach pre-training when you're constrained by data but totally unconstrained by compute. And it's worth maybe spending a few seconds to think for yourself if you haven't already seen this paper like what would you do in this situation. This is a very different algorithmic regime from sort of the computer efficient pre-training world that we've sort of lived in for sort of most of uh uh modern time.”
By optimizing for algorithms with lower compute asymptotes (best loss under infinite compute), researchers can develop new data-efficient methods that don't already exist, providing a systematic way to search for algorithmic improvements.
“But like ultimately what we really want to do is we want these asmtotes to help us develop new and better ideas under infinite compute that that don't already exist. And so if you're interested in the details, that's a QR code for the paper.”
Human-generated text on the internet grows at approximately 3% per year while compute spending on pre-training grows at 4-5x per year, creating an asymmetry that means compute per data point will increase approximately 4x year-over-year as the field scales
“if you look at these sort of public projections of the rate of growth of internet data, they suggest that the amount of sort of human generated text on the internet grows by roughly 3% per year. And the amount of compute that we're spending on pre-training is growing by roughly four or 5x per year. And so what this suggests is that as time passes on, the amount of compute that we're willing to spend per data point is going to continue to increase by roughly 4x year-over-year.”
The mystery of overparameterization is resolved by Pack-Bayes theory: as model parameters increase, empirical risk (training loss) decreases because models fit data better, and simultaneously, more compressible solutions are found, so both terms of the Pack-Bayes bound decrease.
“So the packbased framework gives us a pretty useful way to think about the success of over par parameterization. The first is with empirical risk. Empirical risk is basically training loss. When you increase the number of parameters you can fit your data better. Um so the empirical risk the left uh the first term goes down. And Andrew's work also finds that when we increase the model, when we increase the number of parameters, um we also find more compressible solutions.”
LAY world model uses JEPA (Joint Embedding Predictive Architecture) and adds a SIGG regularizer that enforces the learned latent embeddings follow a Gaussian distribution across all directions, ensuring non-collapse and healthy representation learning.
“So jer is joint embedding predictive architecture it's sort of yan lakun's main work and lay world model is a kind of jepper model uh basically the way it works is you're going to take an autoenccoder um or I should say an image encoder uh encode this observation in this case it's of a robot doing a push cube task that's going to turn that image into a latent vector in the latent space of this encoder uh you're going to train an action condition forecasting module this predictor to be able to predict what is the next latent embedding going to look like when I execute this action. So not what the next image is going to look like but what's the next latent going to look like and you can use the decoder attached to that encoder to decode that back out into a useful image. But for the most part all the interesting work is going to be done in the latent space. And basically what they say is over a batch all of those latent embeddings uh should be in a healthy distribution which they describe as a gausian distributed uh distribution in in the latent space and thus enters the sigg regularizer”
When constrained by data but unlimited in compute, aggressive weight decay regularization (30x larger than compute-optimal settings) produces clean power-law scaling with exponent 1 over model parameters and a measurable asymptote representing optimal performance under infinite compute.
“And so what we do is we show that if you optimally tune your weight decay for each total parameter count. So we're going to optimally tune learning rate, weight decay, and epoch count for each one of these purple points. You can show that your loss follows a really clean power law as you increase the number of parameters in your model. And this is really aggressive regularization. So for context, we use weight decays that are something like 30 times larger than the weight decays that people do for compute optimal pre-training. And so on the legend here, you can see the the sort of the form of this power law. And it has a few nice properties. One is that the exponent on the model parameters n is one. And this is actually predicted by sort of the data constraint theory. The second nice property that it has is that the scaling law has an asmtote which is 3.43 in this case.”
Distillation of an 8-model ensemble (2.4B total parameters) into a single 300M-parameter dense model retains 83% of the loss improvement, allowing data-efficient training to be amortized into a small inference model.
“So we first show that what you can do is you can take an eight ensemble which is roughly 2.4 billion total parameters and you can distill it into a single dense 300 million parameter model which is the pink star in the bottom. And you can do this while retaining roughly 83% of the loss improvement. So this shows you that data efficiency is not something that you need a large amount of inference compute for.”
Two major problems remaining in AI are intelligence per watt and intelligence per sample, with current AI systems being orders of magnitude off from human efficiency on both metrics.
“Back to the sample efficiency thing. I think that like the two major problems we have left really to solve in in AI is intelligence per watt um and intelligence per sample. And if you compare that to to where we're at today compared to humans, um I would say that we're still or an order or two magnitude off on intelligence per watt. Uh and we're me like orders of magnitude off on intelligence per sample.”
The concept of learning world models to predict future states given actions dates back to at least 1990 work by Richard Sutton, not a recent idea, but recently repackaged and advertised under new terminology.
“I wanted to communicate to you all that this is not a new idea at all. It's really just kind of new advertising or packaging on an old idea. So I started going back through Google Scholar and this is a paper that I think is older than the average age of this room. Um from Europe's 1990 and of course Richard S. Sutton who we know from reinforcement learning basically describes exactly a modern world model a black box that takes as input its situation and its action that it's going to execute and outputs a prediction of its immediate next situation.”
LAY world model is 50 times faster than competing approaches across benchmarks because it does all computational work in latent space without additional tricks like multiple forward passes or duplicate model copies in memory.
“Um but a good takeway is that it's about 50 times faster than any of the competition across the board because it's doing all this work in the latent space and it doesn't have to have any like additional tricks relating to more forward passes or like having two copies of the model in memory.”
World models as a capability have biological precedent, suggesting there may be evolutionary or developmental reasons why explicit world models are valuable for intelligent agents
“model based ones are great because you can kind of quantify modeling error and this is really important when you're deploying things in the real world. Uh, we'll talk a little bit about this. I have a little asterisk here, some biological precedent which we'll speak to more.”
YC Winter 2016 batch had ~140 companies with 10-15 becoming unicorns, and during that period Andrej Karpathy, Vaj Deremba, and Greg Brockman were starting OpenAI while asking other AI researchers what problems they were solving.
“I went through winter 16 here. Um it was an unbelievable time. I think 140 companies went through that batch. 10 of 15 of them are unicorns. It's an insane number. um WPY, uh Astronis, um Deep Graham, all these companies were in the batch and during that time uh Sam was still running the show and basically sitting right there would be me, Undercarpathy, Vaj Deremba and Greg Brockman because they were starting this thing called OpenAI and it was like the very early stages and there was like not that many AI companies. So they would ask me and Steve from Debb like what are you guys what are you working on? What are the problems you're working on?”
The YC paper club at Pioneer (the YC office in Woodside/Palo Alto) is intentionally located outside San Francisco city to bring together AI talent from the broader Bay Area including researchers at Google DeepMind, Tesla, xAI, and emerging startups that don't trek to the city to engage with YC.
“I also think that 100% of the AI talent or AI people in the Bay Area, probably about half of them are in the city maybe is a good number. There's anthropic, uh there's open AI, there's cursor, there's all this stuff in the city. Then there's a lot that are down here that are not making the trek up to the city to join YC. And so he's like, Yes, emphatically, yes. Um, and so you have Google DeepMind right on the corner. You have um Tesla, you have XAI, you have Thinking Machines, you have all these other people in Palo Alto, you have a lot of startups.”