What this covers

Dylan Patel and Nathan Lambert walk through DeepSeek's recent engineering achievements and their implications for the global AI competition. The conversation spans DeepSeek's architectural breakthroughs—primarily mixture of experts and multi-head latent attention—that enabled frontier reasoning performance on constrained compute budgets. Beyond the technical details, the discussion widens to semiconductor supply chains, US export controls, power grid constraints on data center placement, and how inference-heavy reasoning models are reshaping hardware demand and economics. The speakers also address whether open weights pose genuine security risks and what it means for capability distribution when frontier models ship with permissive licenses.

The conversation covers contending views on several fronts. One thread examines The Bitter Lesson—whether scaled compute or architectural cleverness determines long-term AI wins—and applies it to DeepSeek's demonstration that efficiency can compress timelines. Another line probes whether US export controls actually constrain China's progress or merely impose costs that hand Beijing a longer-term advantage. There is substantial attention to power consumption scaling, transmission bottlenecks in regions like Virginia, and the concentration of semiconductor R&D in three geographic nodes (Taiwan, Oregon, South Korea). The speakers also engage questions about Jevons Paradox and hardware pricing, compounding error in AI agent chains, the feasibility of content censorship in open models, and hegemonic stability as a frame for great-power transition risk. Throughout, they distinguish between what is verifiable and what remains forecast, particularly around which domains will see RL breakthroughs next and how software engineering productivity will reshape labor.

Sharpest takeaway

DeepSeek's cheap, open-weight reasoning models prove that frontier AI capabilities can be achieved with constrained compute through architectural innovation, reframing the US-China AI race around compute access, export controls, and whether scaling or efficiency wins.

  • DeepSeek achieved frontier performance with ~2,000 export-restricted H800 GPUs via MoE and multi-head latent attention innovations
  • Export controls only matter if AI transforms the economy within 5-10 years; otherwise they hand China a long-term win
  • Reasoning models shift compute demand toward inference, changing the hardware and economics of AI

The argument · threads6 threads · 47 claims
0.68

Export controls and the US semiconductor monopoly rest on concentrated R&D in three locations; if that R&D disappears, distributed manufacturing stops within years regardless of political will.

3 pointscentrality 5/5
  • If AI does not make major societal changes within 5-10 years, export controls guarantee China wins long-term because they kneecap American companies' revenue and the production cycle; the controls only create a US advantage if super-powerful AI arrives within that window, making the timing bet critical.

    “if you believe we’re in this sort of stage of economic growth and change that we’ve been in for the last 20 years, the export controls are absolutely guaranteeing that China will win long-term.”

  • Leading-edge semiconductor R&D is concentrated in only three locations worldwide — Hsinchu Taiwan, Hillsboro Oregon, and Pyongtaek South Korea — and while manufacturing can be distributed (e.g. TSMC Arizona), those fabs remain dependent on the R&D centers; if Hsinchu disappeared, Arizona would stop producing within a couple years.

    “there are three places in the world that are doing leading-edge R&D. There’s Hsinchu, Taiwan, there’s Hillsboro, Oregon, and there is Pyongyang, South Korea.”

  • China's industrial capacity far exceeds the US, with steel and aluminum mills consuming gigawatts of power; if China had access to chips, it could build the largest data center in the world, since the US bottleneck for large clusters is power generation and transmission, not industrial will.

    “China has the unprecedented ability to build ridiculous sums of power... They’ve got steel mills that individually are the size of the entire US industry.”

0.57

AI training efficiency gains and mega-cluster power demands follow exponential growth curves that will exhaust US power infrastructure and create regional transmission bottlenecks by 2030.

7 pointscentrality 3/5
  • Output tokens are roughly four times more expensive than input tokens because input (prefill/prompt) can be processed in parallel as a batch, whereas output is autoregressive — each generated token requires re-reading the whole model and KV cache before generating the next, a non-parallel operation.

    “I can do that input, i.e. the prefill, i.e. the prompt, simultaneously in a batch nature and therefore it is all FLOP.”

  • DeepSeek's efficiency gains, contrary to causing NVIDIA's stock drop, illustrate Jevons paradox: as efficiency goes up, total resource consumption rises, evidenced by AWS H100 prices going up and H200s selling out after V3's launch because R1 wants the higher-memory chip.

    “Jevons paradox is true. AWS pricing for H100s has gone up over the last couple of weeks... H200s are almost out of stock everywhere because H200 has more memory and therefore R1 wants that chip over H100”

  • During training, GPU power spikes dramatically when computing versus idling during weight synchronization, and these transient spikes can damage power plants; Meta upstreamed a PyTorch operator that makes GPUs compute fake numbers during weight exchange so power doesn't spike, while Elon uses Tesla Megapacks for the same purpose.

    “Meta actually has accidentally upstreamed something to code in PyTorch where they added an operator... what it does is amazing... when you’re exchanging the weights, the GPU will just compute fake numbers so the power doesn’t spike too much, and so then the power plants don’t blow up”

  • AI mega-cluster power consumption has jumped from ~15-20 megawatts for GPT-4's 20,000 A100s in 2022 to ~150 megawatts for 100,000+ GPU clusters in 2024, a roughly 10x increase in two years, with US data center power consumption projected to reach ~10% of total US power by 2030 — and even that may be insufficient for the labs.

    “you’ve jumped from 15 to 20 megawatts to almost 10x that number... to 150 megawatts in two years from 2022 to 2024.”

  • Because the grid is slow to build while generation can be ramped via natural gas, in some US regions like Virginia it now costs more to transmit power than to generate it, creating second-order constraints on where AI data centers can be built.

    “in some parts of the US like in Virginia, it costs more to transmit power than it costs to generate it”

  • Stargate does not have the announced $500 billion or even $100 billion; OpenAI has $0 committed for it (legally obligated to put $19B CapEx into the joint venture), Oracle is funding the first ~$6B section, and the rest depends on uncommitted money from Oracle, MGX (UAE), and SoftBank, which would need to sell down its ARM stake — so the money does not yet exist but Dylan believes it will come.

    “It definitely doesn’t have $500 billion, doesn’t even have $100 billion dollars.”

  • Google has the biggest cluster overall using TPUs, but spreads it across multiple data center super-regions ~30 miles apart connected by high-bandwidth fiber, whereas Elon's 200,000-GPU Memphis cluster is the largest fully connected single-site cluster; Google doesn't sell TPUs because they're optimized for internal workloads and selling would cannibalize its search profit DNA.

    “Elon today has 200,000 H100s... Meta, OpenAI, and Amazon all have on the scale of a hundred thousand”

0.57

As AI becomes superintelligent, risks range from compounding errors in agentic systems to ideological backdoors in weights to techno-fascism via brain-computer interfaces consolidating power.

7 pointscentrality 3/5
  • The world has been most peaceful when there are global or regional hegemons (Rome in the Mediterranean, Chinese dynasties, the US in recent decades); a transition from one hegemon to two competitive superpowers (US and China) historically does not happen peacefully, as falling empires shake rather than slide out gracefully.

    “It’s an objective fact that the world has been the most peaceful it’s ever been when there are global hegemons, or regional hegemons in historical context.”

  • A worrisome future is techno-fascism where, as AGI becomes powerful, the most powerful humans adopt brain-computer interfaces to merge with and leverage AI far better than others, leading to a small group (thousands to millions) effectively ruling everyone else and the economy around them.

    “I do worry of like techno-fascism type stuff arising. As AI becomes more and more prevalent and powerful... every very powerful human is going to want to brain-computer interface so that they can interact with the AGI”

  • AI agents that chain tasks together suffer compounding error like semiconductor yield: even the best LLMs don't get 100% on benchmarks, so multiplying sub-100% reliability across many steps yields very low overall success, which is why robust open-world agents (like self-driving without geofencing and teleoperators) remain hard.

    “Chaining tasks together each time, even the best LLMs in particularly pretty good benchmarks don’t get 100%... how do you get to enough nines... This is the same thing with self-driving.”

  • As AI models become more capable, they can embed not just literal code backdoors but ideological alignment ('cultural backdoors') deep in the weights that proliferate when open-weights are released; because models slot into a familiar social conversational context, superhuman persuasion may arrive before superhuman intelligence, letting model-makers subtly shape opinions.

    “What happens when the models are back doors not just to computer systems but to our minds?”

  • Software engineering will see AI productivity gains and cost collapse fastest because it is a verifiable domain (you can unit-test and compile), with SWE-bench scores jumping from 4% to 60% in a year; this won't be a cliff for jobs but a flatlining of growth, and the programmer's role shifts to supervising and steering increasingly intelligent systems.

    “the software engineering agents I think can be done faster sooner than any other agent because it is a verifiable domain.”

  • TSMC's dominance rests partly on cultural factors: the top 1% of Taiwan's NTU graduates go to TSMC at relatively low pay (~$80K starting), employees do grueling non-remote fab work and even show up unprompted during earthquakes to recalibrate machines, with workers specializing on a single task for their whole careers.

    “the top percent of graduates of students that go to the best school, which is NTU, the top percent of those all go work to TSMC.”

  • The cost to achieve GPT-3-level intelligence has fallen roughly 1200x in about three years (from ~$60 per million tokens to a few cents), driven by architectural innovations, better data, better training techniques, and better hardware; this trend means we will have AGI-level intelligence before it permeates the economy due to inference cost constraints.

    “we’ve had a 1200X reduction in cost to achieve the same level of intelligence as GPT-3.”

0.54

DeepSeek achieved remarkable training efficiency through mixture-of-experts sparsity and latent attention techniques while manually optimizing for constrained H800 GPU interconnect bandwidth.

3 pointscentrality 4/5
  • DeepSeek's training efficiency comes primarily from two techniques: a mixture of experts model (activating only a subset of parameters per token, dramatically reducing training and inference cost) and a new multi-head latent attention (MLA) technique using low-rank approximation to reduce memory usage.

    “There’s two main techniques that they implemented that are probably the majority of their efficiency... one is they went to a mixture of experts model... the other thing is that they invented this new technique called MLA, latent attention.”

  • Because the H800 GPUs shipped to China legally had cut interconnect bandwidth, DeepSeek bypassed NVIDIA's standard NCCL communications library and manually scheduled communications by assigning specific SMs (GPU cores) to tasks using PTX, an assembly-level approach most companies avoid, in order to fully utilize their constrained hardware.

    “instead of just calling the NVIDIA library NCCL, they scheduled their own communications... they scheduled communications by scheduling specific SMs.”

  • DeepSeek uses an unusually high sparsity factor in its mixture of experts — activating 8 of 256 experts (a 32:1 ratio) versus the more common 2-of-8 (4:1) ratio in models like Mixtral — which only leading labs have recently begun doing and which creates major load-balancing and communication-scheduling complexity.

    “it’s not 1/4th of the model... Two out of eight experts activating every time you go through the model, it’s eight out of 256.”

0.54

China circumvents chip export controls through ByteDance GPU rentals, estimated hidden GPU flows via Southeast Asia, and access to far larger industrial power capacity than the US possesses.

3 pointscentrality 4/5
  • Although DeepSeek publicly cited only 2,000 H800 GPUs for V3 pre-training and 10,000 A100s from 2021, SemiAnalysis research estimates they actually have closer to 50,000 GPUs split across the hedge fund, research, ablations, and training, far more than the cited training-run number.

    “We believe they actually have something closer to 50,000 GPUs, right? Now this is split across many tasks”

  • Rather than the Chinese government subsidizing DeepSeek, it is more likely the High-Flyer hedge fund (whose CEO owns over half of DeepSeek) funds it as a loss leader and recruiting tool, since DeepSeek has not historically raised outside money and labs like DeepSeek and Alibaba are not close with the CCP.

    “I actually don’t think they are. I think when you look at the Chinese labs... there’s labs like Alibaba and DeepSeek, which are not close with the government.”

  • ByteDance is arguably the largest 'smuggler' of GPUs for China, with over 500,000 GPUs mostly rented legally from Oracle, Google, and smaller clouds worldwide for serving TikTok; additionally an estimated 200,000-300,000 GPUs were routed to China via Singapore/Malaysia, though large-scale smuggling becomes economically harder to hide as the dollar amounts grow.

    “ByteDance, arguably is the largest smuggler of GPUs for China... ByteDance has over 500,000 GPUs... they’re all rented from companies around the world.”

0.22

Frontier model training relies on unverifiable intuition about which configuration scales, and AI breakthroughs in reasoning and embodied tasks will come from verifiable sandbox domains like robotics and computer use.

2 pointscentrality 2/5
  • Frontier model training involves small-scale experimentation and ablations followed by a high-stakes 'YOLO run' where all resources are committed to a single bet; success depends partly on researcher intuition about which configuration will scale, since what works at small scale doesn't always transfer to large scale.

    “There’s all this experimentation you do at the small scale, research ablations... And then all of a sudden you’re like, “Okay, guys. No more fucking around... Let’s pick what we think will work and just go for it. YOLO.””

  • RL reasoning training only works in verifiable domains (math, code) where answers can be checked; the next 'move 37' breakthrough will likely come from computer use or robotics rather than science, because those provide infinitely verifiable sandbox playgrounds where models can self-play, fail, and learn at scale like a baby learning its body.

    “this is where I think the aha moment of computer use or robotics will come in because now you have a sandbox or a playground that is infinitely verifiable.”