What this covers

Nathan Lambert and Sebastian Raschka sit down with Lex Fridman to assess the state of AI development in 2026. Their core claim is that proprietary breakthroughs no longer drive competition—talent rotates, ideas circulate, and the decisive factors are now budget, hardware, and execution quality. The conversation tracks how scaling laws persist across the training pipeline, why RLVR and post-training have become the real frontier, and how open-weight models, particularly from China, are reshaping the competitive landscape while US labs retain advantages in output quality. The pair argue that serving costs dwarf training costs, forcing labs to shrink models despite their intelligence penalties, and that tool use has become foundational—emerging almost entirely from reinforcement learning on verifiable tasks rather than base model capabilities.

The discussion spans economics, infrastructure, and architectural choices. RLVR scales without plateau because models can solve progressively harder problems; RLHF, by contrast, reaches diminishing returns as it averages human preferences and faces the Preference Aggregation Knot. Long context involves a no-free-lunch trade-off between memory and compute cost, while text diffusion models promise speed at the cost of quality. The pair examine how data-mix optimization works at scale and why NVIDIA's moat rests on the CUDA ecosystem rather than silicon alone. They note the flywheel effect binding users to ChatGPT through habit, contest whether Claude Opus hype reflects real usage or echo chamber enthusiasm, and argue that proprietary data in law, finance, and pharmaceuticals will eventually pull frontier talent into specialized in-house training. A contentious thread runs through claims about whether scaling laws guarantee victory or merely reward those who extract the most capability per unit of compute.

Sharpest takeaway

Lambert and Raschka argue that AI progress in 2026 is no longer about proprietary breakthroughs but about resources, execution, and ecosystem dynamics: scaling laws still hold across pre/mid/post-training, the real frontier is post-training (RLVR) and tool use, and open-weight models (increasingly Chinese) are reshaping competition while US labs retain quality leadership.

  • No company will have exclusive technology access because researchers rotate; the differentiator is budget and hardware, not ideas.
  • Scaling laws remain alive across pre-training, RL, and inference-time scaling, but the low-hanging fruit has shifted to RLVR and post-training.
  • Open-weight models, especially from China, are proliferating and could threaten US business models, prompting initiatives like the ATOM/Adam Project.

The argument · threads7 threads · 88 claims
0.72

Capability advances come from training algorithms, not architectural changes.

5 pointscentrality 5/5
  • The rapid advancement in AI happens not in architecture but in the training stages—pre-training, mid-training, and post-training—where capability unlocks like supervised fine-tuning and RLHF (which turned GPT-3 into ChatGPT) come from the algorithmic side rather than architectural change.

    “ChatGPT is basically a GPT-3 model. And GPT-3 is the same as GPT-2 in terms of architecture. What was new was adding the supervised fine-tuning and the Reinforcement Learning with Human Feedback. So, it’s more on the algorithmic side rather than the architecture.”

  • RL post-training does not teach the model new knowledge but unlocks the knowledge it already has from pre-training—it is skill learning (how to solve problems) rather than knowledge soaking, which is why RLVR can take a base model from 15% to 50% accuracy on MATH-500 in 50 steps without teaching new math.

    “RL post-training is more like the skill unlock, where pre-training is like soaking up the knowledge essentially.”

  • RLVR can be trained almost unlimitedly with continued benefit because you let the model solve more and more complex problems, whereas RLHF is preference tuning that reaches a point where more budget makes no sense—since people give different but equally correct answers, RLHF only learns an average preferred style that plateaus.

    “RLVR is more unlimited in how much you can train or still get benefit, whereas RLHF, because it’s preference tuning, reaches a certain point where it doesn’t really make sense to spend more budget on it.”

  • Mid-training uses the same algorithm as pre-training but is more specialized—e.g. focusing on long-context documents—and is positioned so the quality content is the last thing the model sees, because neural networks suffer catastrophic forgetting where learning something causes other things to be forgotten.

    “Mid-training is being selective in terms of quality content at the end, so the last thing the LLM has seen is the quality stuff.”

  • Synthetic data in pre-training does not mean purely AI-made-up data; it often means rephrasing existing content (e.g. turning a Wikipedia article into Q&A or a concise summary) into higher-quality, better-structured text, which yields the same model faster because correct grammar and structure mean it learns the right way first.

    “synthetic data doesn’t necessarily mean purely AI-made-up data. It’s also taking something from a Wikipedia article and then rephrasing it as a Q&A question or summarizing it”

0.68

Resource access and implementation, not proprietary ideas, will differentiate companies.

4 pointscentrality 5/5
  • In 2026 no company will have access to a technology that no other company has, because researchers frequently change jobs and rotate between labs; therefore the differentiating factor will not be proprietary ideas but budget and hardware constraints—the resources needed to implement ideas.

    “I don’t think nowadays, in 2026, that there will be any company having access to a technology that no other company has access to. And that is mainly because researchers are frequently changing jobs, changing labs.”

  • The choice between scaling pre-training and inference scaling is an economic one: pre-training is a fixed cost giving the model a permanent capability, while inference scaling costs money per query forever, so whether to spend more on training depends on how many users you have and how long the model stays on the market.

    “If you spend it more on pre-training, it’s a fixed cost. You train the model, and then it has this capability forever... With inference scaling, you don’t spend money during training; you spend money later per query”

  • NVIDIA's moat is not the GPU itself but the CUDA ecosystem built over two decades; at scale, companies choose the established compatible platform over risky alternatives—but with LLMs now able to help write low-level code, it may become easier to replicate something like CUDA, which originally took 15 years because it was hard.

    “the moat of NVIDIA is probably not just the GPU; it’s more like the CUDA ecosystem, and that has evolved over two decades.”

  • Because labs use each other's published ideas (e.g. Kimi using DeepSeek's architecture), there is constant leapfrogging where the most recent model is probably always the best model, reinforcing that there won't be a single clear winner.

    “One person releases something, the other one comes in, and the most recent model is probably always the best model.”

0.51

Tool use and external retrieval reduce hallucinations better than memorization.

3 pointscentrality 4/5
  • Tool use is a huge unlock because one of the best ways to reduce hallucinations is to have the LLM not try to remember or make up information—using a calculator or Python for math, or a web search for facts—so it retrieves reliable information rather than memorizing it.

    “one of the best ways to solve hallucinations is to not try to always remember information or make things up. For math, why not use a calculator app or Python?”

  • LLMs make knowledge more accessible but don't give you anything not already in the source—for learning from scratch, information-dense textbooks remain the proven strategy (with the LLM used to generate infinite exercises)—yet LLMs add genuine value for on-the-fly, customized personalization where no information-dense resource exists, like planning a Disneyland trip from the sparse, ad-slop-filled internet.

    “it won’t give you anything that is not in the textbook. It’s just packaging it differently”

  • Unlocking more LLM capabilities automatically accelerates all other AI fields, because the researchers and engineers in those fields (e.g. robotics) use LLMs for coding, so optimizing coding LLMs pays off across domains—and the world-class transformer training infrastructure built for language models is now elevating robotics learning.

    “if we unlock more LLM capabilities, it also automatically unlocks all the other fields because it makes progress faster. Because, you know, a lot of researchers and engineers use LLMs for coding.”

0.48

Single high-quality or low-quality results create threshold effects that win user loyalty.

2 pointscentrality 4/5
  • People use an LLM until it breaks or produces a dumb result, then switch—just like with text editors, operating systems, or browsers—rarely comparing models in parallel; a single smart or dumb result creates a threshold effect that wins or loses the user's loyalty.

    “You use it until it breaks, until you have a problem, and then you change the LLM. I think it’s the same way we use anything, like our favorite text editor, operating system, or browser.”

  • Much of LLM usage is driven by name recognition, brand, and muscle memory rather than capability—ChatGPT's longevity created a flywheel where users got used to it and recommend it—and the future will involve multiple models for different use cases (e.g. separate personal and work subscriptions for privacy).

    “a lot of the usage patterns are name recognition and brand, but also almost muscle memory, where ChatGPT has been around for a long time.”

0.48

Industries with proprietary data will build specialized in-house models unlocking major gains.

2 pointscentrality 4/5
  • Industries with proprietary data—pharmaceuticals, law, finance—will eventually hire frontier-lab talent to build in-house models on their data, which would be a major pre-training unlock currently unavailable because that data (e.g. clinical trials) can't be accessed; general-purpose LLMs aren't scratching the surface of what a specifically-trained LLM could do.

    “if you think of big industries like pharmaceuticals, law, or finance, I do think they at some point will hire people from other frontier labs to build their in-house models on their proprietary data, which will be another unlock with pre-training”

  • There is a fundamental difference between raw LLM-generated data and LLM-generated data with a human in the loop who verifies it—even if that verification is a small percentage—because the human filtering and use of the LLM correctly is labor that produces higher-quality output, analogous to how labeled data is created for RLHF.

    “There’s still a human layer that verifies something, and that is in a sense also how data is labeled... it goes through phases and then you actually get higher quality data out of it.”

0.44

Long context and token generation face fundamental efficiency trade-offs without perfect solutions.

9 pointscentrality 2/5
  • Long context is fundamentally a no-free-lunch trade-off: an RNN/Mamba single fixed state is cheap but forgets as context grows because it can't compress everything, while transformers remember every token (great for lookup) but are expensive due to the quadratically-growing KV cache and dot product—so progress comes from finding the right Goldilocks ratio of attention layers to compressed states.

    “One extreme to make it cheap is to have, let’s say, an RNN that has a single state where you save everything... the longer the context gets, the more information you forget because you can’t compress everything into one state. Then on the other hand, you have the transformers, which try to remember every token.”

  • Text diffusion models generate all tokens in a completion in parallel by iterative de-noising (like BERT-style masked filling), promising much faster generation, but the trade-off is quality: papers show that to match autoregressive quality you must crank up de-noising steps until you spend the same compute, and tool use mid-generation is tricky—so diffusion will serve quick, cheap, at-scale tasks rather than replace autoregressive LLMs.

    “the more de-noising steps you do, the better the text becomes... if you want to get the same quality, you have to crank up the de-noising steps and then you end up spending the same compute you would spend on an autoregressive model.”

  • A Manhattan Project-style centralization and containment of AI is not remotely possible in 2025-27, because—as with computers, chips (Huawei now makes them), and the internet—you cannot contain knowledge like that in this day and age; the civilizational-risk motivation behind the original Manhattan Project is hard to apply to open-source models, for which there is no civilizational risk.

    “you see how Huawei makes chips now. It took a few years, but… and I don’t think there is a way you can contain knowledge like that. I think in this day and age, it is impossible, like the internet.”

  • The big Chinese open-weight models may actually be good for US companies, because seeing that a model like DeepSeek-V3.2 is excellent and used with no backlash or security incident could encourage US labs to release better, more cutting-edge open models than the typically one-generation-behind ones they currently release for safety reasons.

    “when these companies see that DeepSeek-V3.2 is really awesome and is being used with no backlash or security risk, that could encourage them to release better models. Maybe that is a very positive thing.”

  • Chinese open-weight models are popular partly because their licenses are unrestricted open-source licenses with no strings attached, whereas Llama or Gemma carry restrictions—such as user-count limits requiring financial reporting to the developer—making the Chinese models more freely usable.

    “the appeal of the open weight models from China is that the licenses are even friendlier. I think they are just unrestricted open source licenses, whereas if we use something like Llama or Gemma, there are some strings attached.”

  • The best way to learn LLMs is to start from a small GPT-2 model that fits on one GPU and reverse-engineer production models: read the config file, add components, load the pretrained weights, and unit-test your implementation against the reference Transformers output—using output-matching as a verifiable reward like RLVR.

    “you can then load the pre-trained weights and see if they work in your model. You want to match the same output that you get with a Transformers model, and then you can use that basically as a verifiable reward to make your architecture correct.”

  • From GPT-2 to today's models, the architectures are fundamentally the same; the differences—Mixture of Experts, multi-head latent attention, group query attention, sliding window attention, RMSNorm vs LayerNorm—are tweaks and knobs, not fundamental changes, so you can convert one into another by adding components in a continuous lineage.

    “It’s not really fundamentally that different. It’s still the same architecture. So you can convert one from one… You can go from one into the other by just adding these changes”

  • Building an LLM from scratch is the best way to learn how it works because code, unlike figures or conceptual explanations, cannot lie—it is math, and if it works you know it is correct, providing self-verification that prose and diagrams cannot.

    “if there is code and the code works, you know it’s correct. There’s no misunderstanding; it’s precise. Otherwise, it wouldn’t work.”

  • Llama declined because Meta executives, excited by its community popularity, tried to use open source for a bigger splash—building giant Llama 4 models overfit to benchmarks (via preference-trained variants) rather than the small, trustworthy, modifiable models people actually wanted—and it imploded under internal political fighting and misaligned incentives between researchers and management.

    “I think the goal was to have a model that people can use, trust, modify, and understand. That includes having smaller models; they don’t have to be the best models.”

0.41

AI remains a tool under human direction; the real risk is skill erosion and lost fulfillment.

4 pointscentrality 3/5
  • AI in its current implementation does not take agency from humans because you have to tell it what to do—it is a more powerful tool than a hammer that can figure things out, but it is still you in charge, which is why Raschka doesn't worry about AI taking over, unlike consciousness which makes humans decide what they want to do.

    “AI in its current implementation, I can’t see it changing. You have to tell it what to do. And so you still have the agency. It doesn’t take the agency from you because it becomes a tool.”

  • If you fully use AI for something you love to do, the thing you love is no longer there—e.g. if an LLM does all your coding, you're just managing something that codes for you—which could lead to burnout and loss of fulfillment, raising the question of whether you'd still feel proud to build something.

    “if you just fully use it for something you love to do, the thing you love to do is no longer there. That could potentially lead to burnout.”

  • There is a Goldilocks zone of struggle necessary for learning: you become an expert by trying things yourself, and if LLMs do everything you never take the next step or get the expert's unlock; the solution is dedicated offline study time alongside LLM use, because if you LLM everything you won't develop expertise.

    “How do you become an expert if you never try to do the thing yourself?”

  • LLMs will at some point solve coding the way calculators solved calculating—a tool where you never need a human to do it—but the open question is independence: will there still be a human asking the AI to build the website, or will AI generate things autonomously without a human in the loop?

    “while I have no doubt that LLMs will at some point solve coding in the way calculators solve calculating, right? At some point, humans developed a tool that you never need a human to calculate that number for”