What this covers

Patrick talks with Etched founders Gavin Uberti and Rob Wachen about a company that sells a full inference rack: the chip, boards, power delivery, interconnects, cold plates, and the manufacturing behind them. The story begins with credibility. Major Valley investors passed on a 30-page memo from two recent Harvard graduates, doubting inference would be a large market. The first Series A needed about $100 million because the cluster had to be built, while the company had roughly $15 million when a $40-50 million physical design payment came due. Patrick frames the gap between internal conviction and outside dismissal as a variant perception opportunity.

The technical middle is concrete. Default EDA tools assume freezing-temperature operation, though AI data centers never run below about 80°C. Bitcoin miners running at under a quarter of GPU voltage inform their low voltage inference bet. They also discuss why GPUs often deliver only 20 to 50 percent of advertised FLOPS, how thermals cap utilization, and why chip-to-chip latency of roughly 4,000 nanoseconds on Blackwell, against a speed-of-light floor of two to three, undercuts tensor parallelism. Gavin's contested argument is that math is cheap and memory loads are expensive, so models should be designed around the machine, as airplanes do not fly like birds.

The forecasts are the most debatable part. Rob expects inference to become the largest market in the world within about a decade, as software acquires real COGS. Lower decode latency could shrink a year-long agent build to a month. Gavin wants token production to reach iPhone-like scale, ending the handcrafted Renaissance-screw era. The conversation also covers hiring, pairing veterans such as Brian Loiler with young builders, and a next generation aimed at gigawatts per month.

Sharpest takeaway

Etched argues that AI inference economics are bottlenecked by general-purpose chip and data center stacks designed before ChatGPT, and that a small team can beat incumbents by relaxing inherited constraints, vertically integrating, and parallelizing the entire build schedule so that production itself becomes the product.

  • Industry tooling and designs carry 'buffer' for general-purpose use cases, such as freezing-temperature timing corners, that do not apply to AI data centers
  • Cluster-scale memory and low-voltage inference target the specific bottlenecks of decode and thermal throttling
  • Vertical integration plus prefetching (building software, racks, and production before the chip returns) compresses bring-up from 10 months to 40 days

The argument · threads11 threads · 97 claims
0.71

Inference demand is becoming the largest market, with token shortages making token capacity extremely valuable.

12 pointscentrality 4/5
  • Rob believes inference will be the biggest market in the world over roughly a decade, because the cost structure of software shifts from near-zero marginal cost to inference-driven COGS and OpEx as agents and coding tools proliferate.

    “it feels like we're on a decade-long march for inference to become the biggest market in the world.”

  • Rob predicts that 2027 will see more AI agents doing knowledge work than humans, and that national energy allocation to inference data centers will govern how large a country's effective workforce can be.

    “I think in 2027 you're going to see there are going to be more agents doing knowledge work than humans”

  • Rob argues that as inference becomes a majority of global GDP, societal productivity will be measured as agents per megawatt or per gigawatt rather than GDP per capita.

    “Right now we measure productivity as a society as GDP per capita, but really it's going to look much more like agents per megawatt”

  • The cost of producing intelligence is far lower than its value, so the world faces a many-year, possibly many-decade shortage of tokens, making any chip or system that produces tokens extremely valuable.

    “the cost of producing intelligence is so dramatically much cheaper than the value of the intelligence, that we are in a many-year, probably many-decade, supply shortage of these tokens.”

  • Colossus charging about 12 dollars an hour for Blackwells and the difficulty of finding 500-megawatt data centers both arise because the buyer can only obtain very large volumes from one seller, so scarcity sets the price.

    “It's because they're the only place you can buy 20,000 of them at once.”

  • In 2022, while building products at the Prod incubator, Rob realized that AI products would cost tens of millions of dollars per year in inference, which would make the cost structure of software companies non-zero marginal cost.

    “all the products I want to build are going to cost tens of millions of dollars a year in inference.”

  • Only a few million people worldwide use paid AI model plans, roughly one one-thousandth of the global population, so the industry is in the early innings and capacity to serve concurrent users is a major future constraint.

    “there's only a few million users in the world using paid plans of AI models. So we're at one one-thousandth of the global population”

  • Because inference will be the largest market and whoever produces the most tokens will be the most valuable company, Etched makes every decision to maximize token capacity online, treating production as the product.

    “We have a saying that production is the product.”

  • Customers should evaluate AI hardware by the number of users or agents they can serve at a guaranteed speed in a given power budget, rather than abstract chip specs, since the speed per user is fixed by the product and the question is how many users fit.

    “The question is, in a given amount of power, how many users can I serve while guaranteeing that speed?”

  • Inference is on a decade-long march to become the biggest market in the world, so the designs for data centers 10 years out have not yet been created and the founders chose to pick a part of that data center to rethink.

    “it feels like we're on a decade-long march for inference to become the biggest market in the world.”

  • Every product the founders wanted to build would cost tens of millions of dollars per year in inference, so software cost structures are changing, with COGS no longer near zero for incremental users and OpEx increasingly tied to inference through coding agents.

    “all the products I want to build are going to cost tens of millions of dollars a year in inference.”

  • Because over $1 billion of revenue exists daily in this category, every day without shipping forfeits opportunity, so willingness to spend should be very high when there is a clear return on investment.

    “every day there's over $1 billion of revenue in this category, and a lot of it's inference. So every day we don't ship, we're just leaving tons of opportunity on the table.”

0.70

Decode latency and concurrency per megawatt, not headline FLOPS, determine inference value.

6 pointscentrality 5/5
  • Lower decode latency shrinks wall clock time for long-horizon agent tasks, so a year-long compute build could take a month and a month-long build could take three days, accelerating scientific and technological progress.

    “So a year-long compute build would now take a month, and that month-long compute build will now take three days”

  • Pre-training researchers care most about wall clock time, meaning the elapsed time from starting a run to getting results, because a six-month run shrunk to a two-month experiment allows many more iterations and architecture changes.

    “If you can shrink this from a six-month run to a two-month experiment, you're going to be able to do many more iterations”

  • Etched claims its hardware delivers roughly an order of magnitude more concurrency at a given interactivity level, which translates directly into tokens per watt and tokens per dollar, meaning a 100-megawatt data center can run many more software agents simultaneously.

    “our hardware is going to generally be able to get you an order of magnitude more concurrency at a given level of interactivity.”

  • Some products, such as coding models, are unusable if they are slow, so providers must hit a minimum tokens per second and decide whether to exclude users or degrade everyone's experience, which creates pressure for new hardware.

    “certain products aren't usable if they're slow. If you want to serve coding models and you want people to actually use them, there's a certain number of tokens per second you need to hit.”

  • Models are moving toward dynamism, meaning per-token or per-user control of computation and memory in attention and mixture-of-experts routing, and current hardware built before these architectures has overhead that makes such dynamic computation inefficient.

    “hardware that really accelerates these types of very dynamic computations is extremely important.”

  • Power availability is the binding limit for AI deployments, and it is much easier to scale from 100 megawatts to a gigawatt than to go from a gigawatt to 10 gigawatts, so operators are trying to scale in by getting more tokens from each megawatt.

    “building new buildings is hard. It's much easier to go from 100 megawatts to a gigawatt than from a gigawatt to 10, and 10 to 100.”

0.56

Purpose-built architectures for modern models beat retrofitted general-purpose chips through compounding gains.

6 pointscentrality 4/5
  • The semiconductor and data center industry is built on buffer because every layer of the stack, from EDA tools to power modules to circuit boards to standard cells, is designed to be general purpose across data center, IoT, and edge uses; when a team designs for one specific use case, it can relax many of those constraints and gain large performance improvements.

    “The reality is the entire semiconductors and data center industry is built on buffer.”

  • Many individual gains such as 20 percent here, 50 percent there, and 2x elsewhere compound across the full system into a design that can be radically better for inference than general-purpose alternatives.

    “there are many more where you get 20% here, 50% there, 2x here — and these compound to a system that can be radically better for inference.”

  • Etched chose not to build an arbitrary graph compiler or support arbitrary PyTorch, CUDA, or ONNX, betting that fewer than 100 models will matter and share similar math, so it built physics-based primitives and gave sophisticated customers direct hardware access, which saved time and yielded more performance.

    “we envisioned a world where there were going to be under 100 models that actually mattered”

  • To run the majority of the world's tokens, a chip must maximize flops per power budget, minimize latency between chips to maximize the scale-up domain, and be produced in as large a volume as possible.

    “if you want to run a majority of tokens in the world, you need to do three things.”

  • Every GPU, TPU, and AI chip serving current models was designed before the modern model era and retrofitted, so there will be an entirely new wave of architectures built for these workloads.

    “the hardware was all designed before ChatGPT. Every GPU, every TPU, every AI chip that was serving these models were fundamentally built before this and was retrofit”

  • Most AI chip architectures were designed before ChatGPT, so modern workloads require chips that differ in how they organize flops, voltage domains, power planes, packaging, board design, and cluster connectivity.

    “that's not that surprising, given all these architectures were built before ChatGPT. If we're trying to build a chip for the modern workloads, it's gonna look very different.”

0.50

Etched's relentless iteration and recruitment of proven veterans and 0-to-1 builders produce seemingly impossible breakthroughs.

9 pointscentrality 3/5
  • Etched operates on a find-a-way mentality: most experiments fail, such as 3 of 30 board experiments working, but only one needs to succeed, so the team assumes each seemingly impossible property is achievable and iterates until it is.

    “A lot of the time when we do experiments, we'll do dozens of experiments and all of them will fail, but we only need one to work.”

  • When Etched asked dozens of semiconductor engineers at many chip companies how to run at lower voltage than GPUs, the answer was that it cannot be done, which the founders found unsatisfying because Bitcoin miners already run at under a quarter of GPU voltage.

    “And the answer we got was, you can't. You can't run at voltages lower than GPUs.”

  • On the first wafer's probe test at 2 to 3 AM, every chip showed red, which a veteran validation engineer met by saying the puzzle begins, and the team saw the first green square within a day, illustrating a find-a-way mentality.

    “He leans back and says, the puzzle begins.”

  • Etched had Codex get GPT-OSS running from scratch using only its documentation, and the model completed the work overnight without human help.

    “we had Codex actually get GPT-OSS running from scratch, just based off of our docs, completely by itself. And it did it, I think, overnight.”

  • The company aims to vertically integrate the entire product, from chips to boards, cold plates, interconnects, and production, because owning each component yields more performance and faster iteration, with the principle that the best vendor is no vendor.

    “I think the best vendor is no vendor. As much as possible, we want to vertically integrate the entire product”

  • Etched's talent model pairs seasoned industry legends such as Brian Loiler, who ran NVIDIA's HGX and DGX systems, with driven young builders who take aggressive risks, because each alone is insufficient to produce breakthroughs.

    “One is not possible without the other, because you need the extremely driven people who just keep asking why and don't know where the bodies are buried”

  • Etched's early believers were either investors who believed in the market and team or deeply technical high-frequency trading firms that audited the microarchitecture, RTL, and schedules, while people in the middle did not understand the bet.

    “if you were anywhere in the middle, you just wouldn't understand it.”

  • Etched maps the hardest technical problems across industries, identifies who did the 0-to-1 work, and recruits them relentlessly, since the share who say yes after the 20th conversation is surprisingly high even though the first-conversation yes rate is low.

    “the number of people who say yes after the first conversation is pretty low, but the number who say yes after the 20th conversation is surprisingly high.”

  • Etched recruited Brian Loiler, who started the HGX and DGX team at NVIDIA, a business representing the majority of NVIDIA's revenue, and his experience of billion-dollar lessons saved the company many development cycles.

    “we brought over Brian Loiler, who built all of NVIDIA's HGX and DGX systems, which is like 80% of their revenue.”

0.43

Low-voltage operation lifts thermal limits, letting chips deliver more FLOPS within the same power budget.

6 pointscentrality 3/5
  • Bitcoin miners run at under a quarter of GPU voltage, which showed that low-voltage chip operation was physically possible and that the question was whether GPU architectures prevented it.

    “Bitcoin miners run at under a quarter of the voltage of GPUs. So this was obviously physically possible.”

  • Utilization cannot reach 100 percent because increasing switching activity draws more power and heat, so the chip self-regulates and lowers clock speed; adding more FLOPS without solving thermals therefore yields no extra performance.

    “If I just add more flops to a GPU today, or another AI chip, I'm not actually going to get more performance, because it's just going to thermal throttle.”

  • Etched's new power delivery mechanism, called low voltage inference, lets chips cram far more FLOPS into the same silicon area without thermal throttling, and Rob predicts all future AI chips will have to be low voltage chips.

    “we think all AI chips in the future are gonna be low voltage chips.”

  • Default EDA sign-off configurations assume chips run at freezing temperatures, but AI data centers never run below about 80 degrees Celsius, so designers can safely drop that assumption and gain clock speed, which is proportional to system throughput.

    “The default configurations for a lot of these EDA tools assume that you're going to be running your chips in freezing temperatures.”

  • Power scales with the square of voltage, so doubling voltage quadruples power and halving voltage cuts power to a quarter, which makes lowering voltage the central lever for fitting more FLOPS into a thermal budget.

    “voltage is quadratically proportional to power. So if I 2x my voltage, my power goes up by 4x.”

  • Model FLOPS utilization (MFU) measures how much of the advertised peak FLOPS a chip actually delivers on real workloads, and GPUs often achieve only 20 to 50 percent depending on the workload, so the relevant metric is achieved rather than headline FLOPS.

    “On GPUs, you often get somewhere between 20% and 50%, depending on the workload.”

0.33

Etched's chip is existential to the company, while incumbents' custom chips are not existential to them.

2 pointscentrality 3/5
  • Etched argues that its chip is existential to the company while labs and hyperscalers are not, because Google, Meta, Microsoft, and OpenAI won't fail if their custom chips fail, so they do not take the same risks and their chips lag in FP8 flop density relative to Blackwell B300.

    “it's completely existential to get as much token capacity online as possible.”

  • Custom AI chips built by Google, Meta, Microsoft, and OpenAI are not existential to those companies, which is why the best chip in the world comes from a company like NVIDIA that builds only that chip, while the founders consider token capacity existential to themselves.

    “it fundamentally is not existential for my company for this product to win.”

0.26

Building the full rack and parallelizing bring-up enables fast, serviceable deployment.

5 pointscentrality 2/5
  • Serving giant models to billions of users requires many chips to communicate quickly with each other, because fast intra-cluster communication alone is insufficient at tens of thousands of chips, and infrastructure must be built across the full stack from wafer to watt.

    “if they can only communicate really quickly with themselves and very slowly with other chips, they're not going to be able to serve giant models at 10,000, 20,000 tokens per second.”

  • Etched built its own rack alongside its own chip, validating cold plates with thermal test chips that mimicked expected hotspots and overpressurizing them before the real chips returned, which has produced no leaks since the chips arrived.

    “Before the chips came back, we made thermal chips that had the exact same hotspots as we expected our chips to have”

  • Etched's decisions are governed by getting as much token capacity online as possible, so it builds a rack instead of just a chip and avoids building data centers because those do not add capacity, since customers handle power and cluster placement themselves.

    “We have a saying that production is the product.”

  • Etched's next generation prioritizes simplicity, fewer parts, quick production cycle times, reliability, and serviceability, with the goal of being producible at gigawatts per month because availability is the best stability for customers.

    “We need to be building a product that is producible at gigawatts per month, in the limit.”

  • Etched parallelized the entire schedule by building software, racks, production lines, and FPGA-based chip emulation before silicon returned, so that bring-up took 40 days, compared with 10 months for another well-known AI chip company to go from silicon to running inference in a rack.

    “There's another very famous AI chip company that took 10 months to go from getting their silicon back to having them running inference in a rack. ... We were able to do it in 40 days.”

0.21

Memory and packaging trade-offs determine which market a chip can serve.

3 pointscentrality 2/5
  • 3D DRAM for decode brings thermal problems, supply chain issues, hybrid bonding challenges, and flops constraints, so adopting it effectively turns a chip into a decode chip with a new set of trade-offs.

    “With 3D DRAM, you have a thermal issue, you have a supply chain issue, you have to figure out hybrid bonding, you have to figure out the flops.”

  • People think about the inference chip market lazily by asking whether a chip is prefill or decode, or HBM, SRAM, or 3D DRAM, but each memory and packaging choice carries trade-offs that Etched's team explored before choosing its architecture.

    “Generally, people think about this market a bit lazily. They say, are you a prefill chip or are you a decode chip?”

  • Etched's first generation uses 4-nanometer silicon and different HBM than Rubin's 3-nanometer chips, so it does not compete for the same wafers and memory, making the market positive-sum since customers often deploy two gigawatts, not a choice between one gigawatt of GPUs and one of Etched.

    “for our first-gen product, we built it on a different supply chain than the Rubins. We're on 4-nanometer, Rubin's on 3-nanometer.”

0.19

Investors and veterans missed AI chip shifts by underweighting changed constraints.

2 pointscentrality 2/5
  • Many traditional semiconductor funds missed the AI chip companies, and coding experts missed the coding companies, because they could not see that the constraints had changed, since EDA tools and FPGA validation have improved in ways that veterans underweight.

    “A lot of the traditional semiconductor funds missed the entire AI chip — like, all the AI chip companies.”

  • Major Valley investors passed immediately on Etched's 30-page memo because the founders were two recent Harvard graduates without a taped-out chip, and investors doubted inference would be a big market since training seemed dominant and models still hallucinated.

    “every major investor in the Valley passed immediately.”

0.05

Early investors' small checks and later doubling down funded Etched's expensive full-stack build.

2 pointscentrality 1/5
  • After weeks of survival-mode outreach, small checks snowballed into soft commitments that summed to about 103 million dollars, which the founders accepted as their Series A, and later rounds were easier because early investors doubled down.

    “we look at it and it's like $103 million. And it's like, these are all soft commits.”

  • Etched's first Series A needed about 100 million dollars because it had to build the full cluster, boards, interconnects, and cold plates, not just the chip, and the company had only about 15 million in the bank when the physical design agreement, costing 40 to 50 million, was due.

    “this was going to cost a lot more than the $15 million we had in the bank.”

0.05

Rob's cancer experience and GPT-4V tumor detection convinced him AI would transform medicine.

2 pointscentrality 1/5
  • Rob describes his parents dropping everything to move to Boston and live with him during months of radiation after a cancer surgery left him below the 98 to 99 percent necrosis threshold, which he calls the kindest thing anyone has done for him.

    “both of my parents decided to move out, drop everything they were doing, and live with me. And I'm eternally grateful.”

  • Rob's realization that GPT-4V could identify a likely tumor from a photo of his back, months before his diagnosis, convinced him that AI would transform medicine and motivated him to pursue faster inference infrastructure.

    “And it immediately says, this could be a tumor, you should get an MRI immediately, go to the doctor.”