What this covers

This YC Decoded episode, presented by YC General Partner Diana Hu, places DeepSeek's R1 in the context of earlier research. It argues that R1 was assembled from techniques DeepSeek had already published in its February 2024 math paper, its May 2024 V2 paper, and its December 2024 V3 paper. DeepSeek has been releasing weights and research for months, along the lines of Meta's Llama, while OpenAI, Google DeepMind, and Anthropic keep weights closed. The episode also traces reinforcement learning) from AlphaGo's 2016 defeat of Lee Sedol through OpenAI's Rubik's Cube hand and Dota 2 systems, as the lineage R1 belongs to.

The middle of the episode is engineering. It describes FP8) training and DeepSeek's fix of periodically merging results into an fp32 accumulator so small errors don't compound. It also covers why mixture of experts is hard to train efficiently, and the data-movement bottleneck that holds GPU utilization to roughly 35%. US export controls are presented as the pressure that pushed DeepSeek to squeeze more out of its existing cluster. A contested section argues that Nvidia's decade-long integrated systems advantage, meaning InfiniBand, CUDA, and developer experience, remains a moat even as algorithms get more efficient.

The R1 recipe gets the most attention. DeepSeek used GRPO) with problems whose answers can be checked, mainly math and code, and graded only final accuracy and formatting by simple rules. Over thousands of steps, the model learned long chains of thought and had "aha moments" in which it caught its own mistakes and backtracked. Without human examples, its early reasoning was hard to read and switched randomly between English and Chinese. The episode then turns to the public reaction. It attributes much of the hype to misreadings of the $5.5 million figure, which covers only V3's final training run, and to free access to the model. It cites a UC Berkeley lab reproducing the core techniques in a smaller model for $30, and ends by arguing that room remains for new players in GPU optimization and inference software.

Sharpest takeaway

DeepSeek's R1 reasoning model achieved comparable performance to OpenAI's o1 through novel algorithmic optimizations and reinforcement learning techniques that were largely published previously, but the recent public attention stems from accessibility, efficiency gains, and misconceptions about training costs rather than fundamentally new breakthroughs.

  • DeepSeek's key innovations (fp8 training, mixture of experts, multi-head latent attention, multi-token prediction) were published in previous papers from December 2024, May 2024, and February 2024
  • R1's reasoning capability emerged through pure reinforcement learning with simple output-based grading rather than complex human feedback, a technique (GRPO) published in February 2024
  • The $5.5 million training cost misconception obscures that this covers only the final training run, not R&D or hardware costs, and the techniques are reproducible at much lower cost ($30 for smaller models)

The claims · ranked72 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.80

Many of the algorithmic innovations responsible for R1's performance, including those discussed in the V3 paper from December 2024, were previously published in Deep Seek's V2 paper (May 2024) and Deep Seek math paper (February 2024), meaning R1 did not emerge from novel breakthroughs but from combining known techniques.

factualhigh valueestablishednovelty 2/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“many of the algorithmic Innovations responsible for r1's remarkable performance were actually discussed in this past December V3 paper or even before that in deep seeks V2 paper which was published in May 2024 or the Deep seek math paper which came out February 2024”

0.79

Many of the algorithmic innovations responsible for R1's performance were discussed in DeepSeek's December V3 paper, May 2024 V2 paper, or February 2024 math paper, not newly introduced with R1

factualhigh valueestablishednovelty 3/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“many of the algorithmic Innovations responsible for r1's remarkable performance were actually discussed in this past December V3 paper or even before that in deep seeks V2 paper which was published in May 2024 or the Deep seek math paper which came out February 2024”

0.75

Nvidia's integrated ecosystem advantage, built over a decade and including InfiniBand networking, CUDA software, and developer experience, creates a moat that is difficult for competitors to replicate even as algorithmic efficiency improves.

causalhigh valuecontestednovelty 2/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“it is nvidia's key Advantage it is not just about gpus it is about an integrated solution they've been building for over a decade that includes the networking with infinity band software with Cuda and developer experience essentially Nvidia provides a deeply integrated system”

0.75

Deep Seek has been publishing its research and releasing model weights for months, following a path similar to Meta's Llama model, in contrast to OpenAI, Google DeepMind, and Anthropic which keep weights closed and publish more limited technical reports.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“the company has been publishing its research and releasing its model weights for months following a pth similar to meta's Lama model this is in contrast to other major AI Labs like open AI Google deep mine and anthropic that have close weights and publish more limited technical reports”

0.75

Nvidia's competitive advantage is not just about GPUs themselves but an integrated solution built over a decade that includes networking (InfiniBand), software (CUDA), and developer experience, allowing AI researchers to program GPU clusters as a distributed system closer to one giant GPU.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“it is nvidia's key Advantage it is not just about gpus it is about an integrated solution they've been building for over a decade that includes the networking with infinity band software with Cuda and developer experience essentially Nvidia provides a deeply integrated system that lets AI researchers program GPU cluster L is a distributed system and and closer to what Jensen describes is one giant GPU”

0.75

The $5.5 million figure for V3's training cost refers only to the final training run and does not include any training cost of R1 or the associated R&D or hardware operating expenses, which are presumably in the hundreds of millions

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“there's some important fine print here the 5.5 million figure refers only to the cost of the final training run for V3 it doesn't include any of the training cost of R1 or the associated R&D or Hardware operating expenses which are presumably in the hundreds of millions”

0.75

A UC Berkeley lab recently applied R1's key techniques to produce complex reasoning in a smaller model for just $30, demonstrating the reproducibility and accessibility of DeepSeek's approach

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“a UC Berkeley lab recently apply R1 Z's key techniques to produce complex reasoning in a smaller model for just $30”

0.75

Much of the hype cycle around R1 was driven by misconceptions around V3's alleged $5.5 million training cost rather than specific algorithmic improvements

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“another explanation is that a lot of the hype cycle didn't actually have to do with the specific algorithmic improvements that we describ but with misconceptions around v3's alleged $5.5 million in training cost”

0.73

DeepSeek's fp8 accumulation fix periodically merges calculations back into a higher precision fp32 accumulator to prevent small numerical errors from compounding, resulting in far more efficient training across thousands of GPUs while maintaining model quality

factualhigh valueestablishednovelty 2/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“a crucial enhancement is their fp8 accumulation fix which periodically merges calculations back into a higher position fp32 accumulator to prevent small numerical errors from compounding the result far more efficient training across thousands of gpus cutting cost while maintaining model quality”

0.71

Deep Seek's approach to building R1 involved assembling a bunch of problems with verifiable outputs (especially math and coding problems) and designing a training pipeline to get the model to think for extended periods and output correct answers, without providing external examples of how to think from humans or AI.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“they assemble a bunch of problems with verifiable outputs especially in math and coding problems and then design a training pipeline to get the model to think for a bit and output the correct answers but they don't give the model any external examples of how to think whether from humans or Ai”

0.71

Rather than using a complex AI to give the model fine-grained feedback during RL training, Deep Seek uses simple rules to evaluate the model's final output on accuracy and formatting.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“the Grading process was extremely simple rather than using a complex AI to give the model fine grain feedback deep seek uses simple rules to evaluate the model's final output on accuracy and formatting”

0.71

Through reinforcement learning alone with GRPO and simple reward signals, Deep Seek's R1 model developed reasoning behavior that emerged over thousands of RL steps, learning extended chain-of-thought skills and experiencing 'aha moments' where it recognized its own mistakes and backtracked to correct its reasoning.

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“remarkably with this process alone deep seek saw reasoning emerg over thousands of RL steps the model learned skills like extended Chain of Thought and even experience a aha moment where it recognized its own mistakes and backtracked to correct its reasoning”

0.70

The primary bottleneck in GPU-accelerated model training is not computation but data movement between caches and GPUs, which limits GPU utilization to around 35% even with efficient precision formats.

factualhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“the rest of the time these gpus are waiting for data to be moved either between caches or other gpus”

0.69

At FP8 precision, AI training typically achieves only around 35% model floating-point operations utilization (MFU), meaning GPUs are only used at peak potential about a third of the time, with the rest spent waiting for data movement between caches or other GPUs.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“at fp8 it is typical to only see around 35% model flops utilization or mfu meaning gpus are only being utilized at Peak potential about a third of the time the rest of the time these gpus are waiting for data to be moved either between caches or other gpus”

0.69

R1 (one of the first large models to achieve top-tier results purely through reinforcement learning) was developed within the trajectory of prior research including DeepMind's AlphaGo (which used self-play RL to beat Lee Sedol in 2016), OpenAI's robotics hand training (which solved a Rubik's Cube using RL), and OpenAI's Dota 2 victory using RL.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“pure RL has long been a subject of Investigation in Western research Labs such as deep Minds alphao which simulated thousands of random games of self-play to beat Lisa do the world's top go player in 2016 in 2019 open aai achieved notable success using reinforcement learning to train a robotics hand to solve a Rubik's Cube and beat a top human team in competitive Dota 2”

0.69

Deep Seek's approach and results are reproducible: UC Berkeley researchers recently applied R1's key techniques to produce complex reasoning in a smaller model for just $30.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“it is worth noting that this work is reproducible a UC Berkeley lab recently apply R1 Z's key techniques to produce complex reasoning in a smaller model for just $30”

0.69

US export controls on GPU sales to China created hardware constraints that forced DeepSeek to find ways to extract more training capacity and bandwidth from their existing GPU cluster

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“given its Hardware constraints and US exports controls on the sale of gpus to China deep seek needed to find a way to get more training and more bandwidth from their existing cluster of gpus”

0.69

DeepSeek's efficiency improvements enabled R1 to offer near state-of-the-art performance at a fraction of the price of other reasoning models

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“also because of all the efficiency improvements it offers near state-of-the-art performance at a fraction of the price of other reasoning models”

0.69

One explanation for the hype cycle around R1 is the sheer accessibility of DeepSeek's model—R1 is freely accessible through their website and app, free to download, run locally, and customize

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“one explanation is the sheer accessibility of deep seeks model ran is freely accessible through their website and app and it is free to download run locally and customized”

0.69

The cost of intelligence keeps going down as new players demonstrate efficiency improvements, which is fantastic news for AI applications in consumer or B2B sectors

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“ultimately this is fantastic news for AI applications in consumer or B2B since it means the cost of intelligence keeps going down”

0.68

Mixture of experts architecture is not new but has been challenging to train models with this architecture efficiently; DeepSeek introduced novel techniques that stabilize performance and increase GPU utilization in mixture of experts training

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“mixture of experts isn't A New Concept but it's been challenging to train models with this architecture efficiently deep seek introduced novel techniques that stabilize performance and increase GPU utilization”

0.68

DeepSeek uses a simple grading process based on output accuracy and formatting rules rather than complex AI-based feedback to evaluate the model's reasoning and generate reward signals for RL training

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“and Grading process was extremely simple rather than using a complex AI to give the model fine grain feedback deep seek uses simple rules to evaluate the model's final output on accuracy and formatting”

0.68

Unconstrained by human examples, R1's thinking steps initially suffered from poor readability and switched between English and Chinese at random

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“and unconstrained by human examples R1 Z's thinking steps suffer from poor readability switching between English and Chinese at random”

0.68

DeepSeek proves that there is still room for new players on the frontier of AI development, particularly in rebuilding the stack, optimizing GPU workloads, improving software at the inference layer, and developing AI-generated kernels

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“what deep seek really proves is that there is still room for new players on the frontier in particular there's room for rebuilding the stack for optimizing GPU workloads improving software at inference layer tooling and developing AI generated kernels”

0.68

DeepSeek's training pipeline for R1 assembles problems with verifiable outputs (especially in math and coding) and designs a training process where the model thinks through the problem and outputs correct answers, without providing external examples of how to think from humans or AI

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“at a high level they assemble a bunch of problems with verifiable outputs especially in math and coding problems and then design a training pipeline to get the model to think for a bit and output the correct answers but they don't give the model any external examples of how to think whether from humans or Ai”

0.68

Through pure reinforcement learning with simple output grading alone, DeepSeek's model learned reasoning skills like extended chain of thought and experienced 'aha moments' where it recognized its own mistakes and backtracked to correct its reasoning

factualhigh valueestablishednovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“remarkably with this process alone deep seek saw reasoning emerg over thousands of RL steps the model learned skills like extended Chain of Thought and even experience a aha moment where it recognized its own mistakes and backtracked to correct its reasoning”

0.66

Deep Seek V3 was released in December as a general-purpose base model and Deep Seek R1 was released at the end of January as a reasoning model built on top of V3, not as entirely separate models.

factualhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek V3 which was actually released this past December is a general purpose based model...deep seek R1 which is released at the end of January is a reasoning model built on top of deep seek V3”

0.66

DeepSeek optimized V3 training efficiency by using 8-bit floating point format rather than the usual 16-bit or 32-bit format, achieving massive memory savings without sacrificing performance

factualhigh valueestablishednovelty 1/4durability 4/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“One Way de seek optimized for efficiency and got more floating Point operations per second or flops from the gpus was by trading V3 natively in 8bit floating Point format rather than the usual 16bit or 32-bit format this is not a new idea many other labs are doing it too but it was key for getting such massive memory savings without sacrificing performance”

0.65

Deep Seek V3 optimized for efficiency by training natively in 8-bit floating-point format rather than the usual 16-bit or 32-bit format, achieving massive memory savings without sacrificing performance.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“One Way de seek optimized for efficiency and got more floating Point operations per second or flops from the gpus was by trading V3 natively in 8bit floating Point format rather than the usual 16bit or 32-bit format this is not a new idea many other labs are doing it too but it was key for getting such massive memory savings without sacrificing performance”

0.65

Multi-token prediction improves representation planning, allowing the model to pre-plan sequences for smoother and more coherent outputs during inference.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“it also improves representation planning allowing the model to pre-plan sequences for smoother more coherent outputs during inference”

0.65

Most large language models can be improved by being prompted to think step by step, but reasoning models are specifically trained to break down hard problems and think about them for paragraphs at a time.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“most llms can be improved by being prompted to think step by step but what sets reasoning models apart is that they are specifically trained to break down hard problems and think about them for paragraphs at a time”

0.65

Both OpenAI and Deep Seek achieve impressive reasoning results through reinforcement learning, a technique to shape an LLM's behavior based on feedback and reward signals.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“most llms use some variation of reinforcement learning with human feedback AKA RL HF or reinforcement learning from AI feedback AKA RL AI F to improve their models usefulness and Alignment but reasoning models apply RL specifically towards the task of thinking step by step through complex problems”

0.65

R1 zero (trained with pure RL without structured examples) suffered from poor readability, switching between English and Chinese at random, so Deep Seek introduced a cold start phase involving fine-tuning on structured reasoning examples before RL to create R1, which eliminated language mixing issues and made outputs far more comprehensible.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“but unconstrained by human examples R1 Z&'s thinking steps suffer from poor readability switching between English and Chinese at random so deep seek introduced a cold start phase fine-tuning on structured reasoning examples before RL to get our one this eliminated the language mixing issues and made outputs far more comprehensible”

0.64

Deep Seek's R1 reasoning model achieves comparable performance to OpenAI's O1 and Google's Flash 2.0 on certain complex reasoning benchmarks despite the company operating under US export controls on GPU sales to China and hardware constraints.

factualhigh valueestablishednovelty 1/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek R1 which is released at the end of January is a reasoning model built on top of deep seek V3...resulting in R1 a model that's achieved comparable performance to open a01 and Google Flash 2.0 on certain complex reasoning benchmarks”

0.62

Nvidia's key advantage over competitors is not just GPUs themselves but an integrated solution built over a decade including InfiniBand networking, CUDA software, and developer experience that lets AI researchers program GPU clusters as distributed systems close to one giant GPU

factualhigh valuecontestednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“this is nvidia's key Advantage it is not just about gpus it is about an integrated solution they've been building for over a decade that includes the networking with infinity band software with Cuda and developer experience essentially Nvidia provides a deeply integrated system that lets AI researchers program GPU cluster L is a distributed system and and closer to what Jensen describes is one giant GPU”

0.61

Deep Seek's FP8 accumulation fix periodically merges calculations back into a higher precision FP32 accumulator to prevent small numerical errors from compounding, resulting in far more efficient training across thousands of GPUs while cutting cost and maintaining model quality.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“a crucial enhancement is their fp8 accumulation fix which periodically merges calculations back into a higher position fp32 accumulator to prevent small numerical errors from compounding the result far more efficient training across thousands of gpus cutting cost while maintaining model quality”

0.61

Deep Seek V3 uses a mixture of experts architecture with 671 billion total parameters but only activates 37 billion parameters for each token prediction, compared to Llama 3's largest model which uses 405 billion parameters without mixture of experts, meaning V3 activates 11x fewer parameters per forward pass.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek V3 has 671 billion modern parameters but only 37 billion are activated for a given token prediction by contrast the largest and most capable Lama 3 Model doesn't use a mixture of expert architecture so it activates its full 405 billion for each token prediction in other words V3 activates 11x fewer parameters for each forward pass”

0.61

Mixture of experts architecture has not been a new concept, but it has been challenging to train models with this architecture efficiently; Deep Seek introduced novel techniques that stabilize performance and increase GPU utilization when using mixture of experts.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“mixture of experts isn't A New Concept but it's been challenging to train models with this architecture efficiently deep seek introduced novel techniques that stabilize performance and increase GPU utilization”

0.61

Deep Seek V3 uses multi-head latent attention (MLA), first revealed in the V2 paper from May 2024, which compresses key and value matrices into latent representations reconstructed only when needed, reducing KV cache size by 93.3% and boosting maximum generation throughput to 5.76 times compared to traditional approaches.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“V3 makes use of multi head latent attention or MLA which deep seek first revealed with its V2 paper which was published in May 2024 MLA is a solution designed to tackle KV cat storage limitation one of the biggest sources of bam overhead in large models instead of storing full key and value matrices MLA manages to compress press them down into a latent representation reconstructing them only when needed this helped the B2 model reduce his KV cach size by 93.3% and boosted its maximum generation throughput to 5.76 times”

0.61

Unlike traditional models that predict only the next token, Deep Seek V3 uses multi-token prediction (MTP) to anticipate multiple future tokens at each step, densifying training signals and providing more feedback per step for better data efficiency and faster learning.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“unlike traditional models that predict only the next token V3 makes use of multi- token prediction or MTP MTP enables V3 to anticipate multiple future tokens at each each step this densifies training signals providing more feedback per step for better data efficiency and faster learning”

0.61

Multi-token prediction (MTP) modules can be repurposed for speculative decoding, reducing sequential processing steps and significantly speeding up generation during inference.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“MTP modules can be repurposed for speculative decoding reducing sequential processing steps and significantly speeding up generation”

0.61

Deep Seek uses a novel technique published in February 2024 called Group Relative Policy Optimization (GRPO) to update their model based on output scores from simple accuracy/formatting evaluation.

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“they use these output scores to update their model through a novel technique they published in February 2024 called group Rel relative policy optimization or grpo”

0.61

At fp8 precision, AI labs typically achieve only around 35% model FLOPs utilization (MFU), meaning GPUs are utilized at peak potential only about one-third of the time while the rest of the time GPUs wait for data movement between caches or other GPUs

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“at fp8 it is typical to only see around 35% model flops utilization or mfu meaning gpus are only being utilized at Peak potential about a third of the time the rest of the time these gpus are waiting for data to be moved either between caches or other gpus”

0.61

Multi-token prediction modules in DeepSeek V3 can be repurposed for speculative decoding, reducing sequential processing steps and significantly speeding up generation during inference

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“it also improves representation planning allowing the model to pre-plan sequences for smoother more coherent outputs during inference MTP modules can be repurposed for speculative decoding reducing sequential processing steps and significantly speeding up generation”

0.61

DeepSeek V3 uses a mixture of experts architecture with 671 billion total parameters but only 37 billion parameters activated for each token prediction, activating 11x fewer parameters per forward pass than Llama 3's 405 billion non-sparse model

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek V3 has 671 billion modern parameters but only 37 billion are activated for a given token prediction by contrast the largest and most capable Lama 3 Model doesn't use a mixture of expert architecture so it activates its full 405 billion for each token prediction in other words V3 activates 11x fewer parameters for each forward pass saving tons of computation”

0.61

DeepSeek V3 uses multi-head latent attention (MLA) which compresses key and value matrices into a latent representation and reconstructs them only when needed, reducing KV cache size by 93.3% and boosting maximum generation throughput to 5.76x compared to traditional approaches

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“V3 makes use of multi head latent attention or MLA which deep seek first revealed with its V2 paper which was published in May 2024 MLA is a solution designed to tackle KV cat storage limitation one of the biggest sources of bam overhead in large models instead of storing full key and value matrices MLA manages to compress press them down into a latent representation reconstructing them only when needed this helped the B2 model reduce his KV cach size by 93.3% and boosted its maximum generation throughput to 5.76 times”

0.61

Both OpenAI and DeepSeek achieve their impressive reasoning model results through reinforcement learning, a technique to shape an LLM's behavior based on feedback and reward signals

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“with R1 deep seek took a similar approach and published The Secret Sauce open Ai and deep seek achieve their impressive results through reinforcement learning a technique to shape an llms Behavior based on feedback and reward signals”

0.61

DeepSeek applies reinforcement learning through group Relative Policy Optimization (GRPO), a novel technique published in February 2024, to update the model based on output scores

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“they use these output scores to update their model through a novel technique they published in February 2024 called group Rel relative policy optimization or grpo”

0.61

Just 2 weeks after R1 was released, OpenAI released o3 which outperforms both R1 and o1 on key benchmarks, showing the pace of innovation is accelerating

factualhigh valueestablishednovelty 2/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“but the pace of innovation is speeding up just 2 weeks after R1 was released open AI release 03 Mei which outperforms R1 and 01 on key benchmarks”

0.61

Given the extreme algorithmic optimizations DeepSeek achieved, the $5.5 million final training run figure actually seems perfectly possible and is worth noting that this work is reproducible

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“given the extreme algorithmic optimizations here that 5.5 million training run number actually seems perfectly possible and it is worth noting that this work is reproducible”

0.61

Unlike traditional models that predict only the next token, DeepSeek V3 uses multi-token prediction (MTP) which enables the model to anticipate multiple future tokens at each step, densifying training signals and providing more feedback per step for better data efficiency and faster learning

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“finally unlike traditional models that predict only the next token V3 makes use of multi- token prediction or MTP MTP enables V3 to anticipate multiple future tokens at each each step this densifies training signals providing more feedback per step for better data efficiency and faster learning”

0.61

Most large language models can be improved by being prompted to think step by step, but what sets reasoning models apart is that they are specifically trained to break down hard problems and think about them for paragraphs at a time rather than relying on prompting

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“most llms can be improved by being prompted to think step by step but what sets reasoning models apart is that they are specifically trained to break down hard problems and think about them for paragraphs at a time”

0.61

Modern LLMs use some variation of reinforcement learning with human feedback (RLHF) or reinforcement learning from AI feedback (RLAIF) to improve usefulness and alignment, but reasoning models apply RL specifically to the task of thinking step by step through complex problems

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“modern llms use some variation of reinforcement learning with human feedback AKA RL HF or reinforcement learning from AI feedback AKA RL AI F to improve their models usefulness and Alignment but reasoning models apply RL specifically towards the task of thinking step by step through complex problems”

0.61

DeepSeek addressed the readability problem by introducing a cold start phase with fine-tuning on structured reasoning examples before RL, which eliminated language mixing issues and made outputs far more comprehensible

factualhigh valueestablishednovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“so deep seek introduced a cold start phase fine-tuning on structured reasoning examples before RL to get our one this eliminated the language mixing issues and made outputs far more comprehensible”

0.56

Deep Seek's R1 model is freely accessible through their website and app, free to download and run locally, and customizable, making it widely accessible to the broader public.

factualhigh valueestablishednovelty 1/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“ran is freely accessible through their website and app and it is free to download run locally and customized”

0.56

Deep Seek's efficiency improvements allow R1 to offer near state-of-the-art reasoning model performance at a fraction of the price of other reasoning models like OpenAI's O1.

factualhigh valueestablishednovelty 1/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“because of all the efficiency improvements it offers near state-of-the-art performance at a fraction of the price of other reasoning models”

0.55

What Deep Seek really proves is that there is still room for new players on the frontier of AI, particularly in rebuilding the stack for optimizing GPU workloads, improving software at the inference layer, and developing AI-generated kernels.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“what deep seek really proves is that there is still room for new players on the frontier in particular there's room for rebuilding the stack for optimizing GPU workloads improving software at inference layer tooling and developing AI generated kernels”

0.55

Pure reinforcement learning without human examples has long been a subject of investigation in Western research labs, including DeepMind's AlphaGo which simulated thousands of random games of self-play to beat Lee Sedol, the world's top Go player, in 2016

factualhigh valueestablishednovelty 0/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“pure RL has long been a subject of Investigation in Western research Labs such as deep Minds alphao which simulated thousands of random games of self-play to beat Lisa do the world's top go player in 2016”

0.53

R1 achieves comparable performance to OpenAI's O1 on certain math and coding benchmarks, but the pace of innovation in reasoning models is accelerating: just 2 weeks after R1's release, OpenAI released O3 Mini which outperforms R1 and O1 on key benchmarks.

factualhigh valueestablishednovelty 0/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“R1 achieves comparable performance to 01 on certain math and coding benchmarks but the pace of innovation is speeding up just 2 weeks after R1 was released open AI release 03 Mei which outperforms R1 and 01 on key benchmarks”

0.52

Many of the misconceptions around the hype cycle of Deep Seek's announcement did not stem from the specific algorithmic improvements in V3 and R1, but rather from misconceptions around V3's claimed $5.5 million training cost.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“another explanation is that a lot of the hype cycle didn't actually have to do with the specific algorithmic improvements that we describ but with misconceptions around v3's alleged $5.5 million in training cost”

0.51

R1 achieves comparable performance to o1 on certain math and coding benchmarks, demonstrating that DeepSeek's approach yields competitive results

factualhigh valueestablishednovelty 1/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“the results are impressive R1 achieves comparable performance to 01 on certain math and coding benchmarks”

0.51

DeepSeek R1, an open-source reasoning model released at the end of January, achieved comparable performance to OpenAI's o1 and Google Flash 2.0 on certain complex reasoning benchmarks

factualhigh valueestablishednovelty 1/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek R1 which is released at the end of January is a reasoning model built on top of deep seek V3 in other words deeps took B3 and applied various algorithmic improvements to it in order to optimize its reasoning ability resulting in R1 a model that's achieved comparable performance to open a01 and Google Flash 2.0 on certain complex reasoning benchmarks”

0.51

DeepSeek V3, released in December, is a general purpose base model that achieves comparable performance to OpenAI's GPT-4o, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5

factualhigh valueestablishednovelty 1/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“deep seek V3 which was actually released this past December is a general purpose based model that achieves comparable performance to other base models like open ai's GPT 40 anthropics claw 3.5 Sonet and Google's Gemini 1.5”

0.50

OpenAI demonstrated the power of reasoning models in September 2024 with O1, which achieved state-of-the-art results in math, coding, and science benchmarks.

factualhigh valueestablishednovelty 0/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“In September open AI showed the power of this new approach with 01 this achieved state-of-the-art results in math coding and science benchmarks”

0.50

Deep Seek V3, combining multiple efficiency innovations (8-bit training, FP8 accumulation fix, mixture of experts, multi-head latent attention, multi-token prediction), is one of the most impressive base models on the market and has been available for some time (released December 2024).

factualhigh valueestablishednovelty 0/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“taken alog together this makes V3 one of the most impressive based models on the market and it's been out for some time now”

0.48

The $5.5 million figure claimed for Deep Seek V3's training cost refers only to the final training run for V3 and does not include the training cost of R1 or associated R&D and hardware operating expenses, which are presumably in the hundreds of millions.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“the 5.5 million figure refers only to the cost of the final training run for V3 it doesn't include any of the training cost of R1 or the associated R&D or Hardware operating expenses which are presumably in the hundreds of millions”

0.45

OpenAI demonstrated the power of reasoning models in September with o1, achieving state-of-the-art results in math, coding, and science benchmarks

factualhigh valueestablishednovelty 0/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“in September open AI showed the power of this new approach with 01 this achieved state-of-the-art results in math coding and science benchmarks”

0.45

Now is the best possible time to be building an AI startup because the cost of intelligence is declining and accessibility of models is increasing

normativehigh valuespeaker onlynovelty 1/4durability 2/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“so the big takeaway here this is the best possible time to be building a startup”

0.43

The cost of intelligence keeps declining, making this an excellent time to build AI startups because cost reduction is one of the fundamental barriers to AI application that is being continuously lowered.

forecasthigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“ultimately this is fantastic news for AI applications in consumer or B2B since it means the cost of intelligence keeps going down so the big takeaway here this is the best possible time to be building a startup”

0.43

Despite the extreme algorithmic optimizations in Deep Seek V3, the $5.5 million final training run cost figure seems perfectly plausible given the specific efficiency improvements implemented.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“given the extreme algorithmic optimizations here that 5.5 million training run number actually seems perfectly possible”

0.43

Deep Seek's efficiency innovations were not individually novel (8-bit training is used by many labs, mixture of experts is a known architecture, etc.) but the combination and implementation at massive scale represents a meaningful engineering contribution.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“mixture of experts isn't A New Concept but it's been challenging to train models with this architecture efficiently deep seek introduced novel techniques”

0.19

OpenAI achieved notable success in 2019 using reinforcement learning to train a robotics hand to solve a Rubik's Cube and to beat a top human team in competitive Dota 2

factualestablishednovelty 0/4durability 1/4· Unidentified Speaker — The Engineering Unlocks Behind DeepSeek | YC Decoded [4Tmn-XP93m4]

“in 2019 open aai achieved notable success using reinforcement learning to train a robotics hand to solve a Rubik's Cube and beat a top human team in competitive Dota 2”