Capital, Compute & AI Scaling
What this covers
Modest and Chetan start from reports that pre-training gains have plateaued and work out what follows for the industry's money. They cover the mechanics of the shift to test-time compute, where a model explores candidate solutions in parallel and a verifier scores them. They then turn to infrastructure and venture economics. Chetan makes the capital problem concrete: a one-billion-dollar training run would commit two full funds of a $500 million early-stage vehicle to a single run that may not work.
The infrastructure discussion is Modest's territory. He describes cloud as a logistics business built on multi-year demand forecasting, notes five years of flat data center power demand once efficiency took over, and speculates that inference-heavy networks would favor smaller, lower-latency sites over million-chip clusters. Both speakers cite steep inference cost declines, roughly 100x to 200x in two years, and compare them to internet transit pricing. They tie the decline to application gross margins. Several claims are flagged as contested: that tiny teams have reached the text frontier in about six weeks, that test-time compute may run into limits from search space, verifier quality, or ambiguity, and that new modalities like video and audio could bring another leap.
The last stretch moves to incumbents and definitions. Chetan argues the Innovator's Dilemma bites harder in enterprise software than in consumer, and he is skeptical that Google can defend its position. They also discuss Microsoft's rights to OpenAI's IP lasting only until AGI, and Chetan's expectation that AGI is close or already here. Modest suggests a formal declaration might be met with indifference.
Chetan and Modest argue that the shift from pre-training scaling to test-time compute changes AI economics: it lowers capital barriers for small teams, aligns hyperscaler capex with inference revenue, and favors application-layer startups, while public markets have not yet priced the resulting reshuffling of winners and losers.
- Small teams of two to five people are reaching the frontier with far less capital than the large labs spent, because open-source models like LLaMA can be fine-tuned and distilled.
- Test-time compute ties spending to usage, which makes hyperscaler capex easier to justify than the upfront multi-billion-dollar pre-training bets.
- Public market probability distributions have shifted toward downstream derivative plays, but this is not yet reflected in prices.
Pre-training scaling has plateaued, reviving capital-light venture opportunity at the model layer through test-time compute.
- If pre-training is not scaling, the classic venture model of a small team with a technology breakthrough, capital-light development, and rapid lead over incumbents is viable again at the model layer, after appearing impossible for about two years.
“The venture capital model has been could you get together a team of extraordinary people, have a technology breakthrough, be capital-light, and jump way ahead of incumbents very quickly”
- The industry is shifting to a test-time compute paradigm in which the model explores multiple candidate solutions in parallel, uses a verifier to score and iterate on them, and scales intelligence along a time axis rather than a training-compute axis.
“what test-time compute is in a very basic way is you actually ask the LLM to look at the problem, come up with a set of potential solutions to it, and pursue multiple solutions in parallel”
- All major labs have hit a plateau in pre-training scaling over the last two years, because the world's human-generated text data has been largely tokenized and models trained on synthetic data are not enabling scaling to continue.
“all the labs have hit some kind of plateauing effect on how we perceive scaling for the last two years, which was specifically in the pre-training world”
- Test-time compute may exhaust useful search space quickly, verifier quality may not scale linearly with compute, and some tasks are limited by ambiguity or underlying knowledge rather than compute, so scaling test-time compute will not break all prior data limits.
“It's unclear that that scales linearly with infinite compute.”
- A breakthrough showing that synthetic data enables pre-training scaling would restart 10 to 100 billion dollar superclusters, reversing the current shift to test-time compute.
“if somebody came out with results that pre-training was back on and there was a huge breakthrough on synthetic data”
- All the major labs have hit a plateau in how scaling is perceived for pre-training, since the power laws implied that 10x more pre-training compute yields a step-function gain in model performance, but that has stopped holding in practice.
“all the labs have hit some kind of plateauing effect on how we perceive scaling for the last two years, which was specifically in the pre-training world”
Small teams and open-source models can reach frontier performance for far less than large labs spend.
- For an early-stage fund with 500 million dollars and roughly 30 investments per fund cycle, a one-billion-dollar training run is effectively committing two full funds to a single run that may not work.
“a billion dollar training run is essentially you're committing two funds to do one training run that may or may not work.”
- Teams of two to five people are reaching the frontier of text model performance with spending that is multiple orders of magnitude less than large labs, and this has happened within roughly the last six weeks.
“We're seeing these small teams of quite literally two to five people jumping to the frontier with spend that is not one order, but multiple orders of magnitude less than what these large labs were spending to get there.”
- The proliferation of open-source models, especially Meta's LLaMA family in many sizes, lets small teams download, distill, fine-tune, and modify models to catch up to the frontier with clever techniques focused on a vertical or use case.
“you can take these models, download them, put them on a local machine, you can put them in a cloud, you can put them on a server, and you can use these models to distill, fine-tune, train on top of, modify”
- Because the developer ecosystem assumes the LLaMA transformer architecture as the standard, the stack is standardizing around LLaMA from hardware vendors to hyperscalers, which lowers development friction for new model builders.
“it's sort of standardizing the entire stack towards this Llama way of thinking”
- Mark Zuckerberg said on Meta's earnings call that LLaMA 4 is being trained on a cluster bigger than anything he has seen, reportedly more than 100,000 H100 GPUs, with smaller LLaMA 4 models expected to be ready in early 2025.
“LLaMA 4 is being trained on a bigger cluster than anything he's ever seen out there.”
- Sell-side research has not yet analyzed what the test-time compute paradigm means, private market coverage is thin, and entrepreneurs are matching frontier performance in specific use cases for under a million dollars, which was not possible a year or two earlier.
“just reading sell-side reports that we haven't seen sell-side reports or analysis on what this new paradigm of test-time compute means”
Durable value sits in the application layer, not in rapidly depreciating model weights.
- Benchmark has concentrated investment in applications since the summer of 2022, when developers began building on OpenAI APIs, because applications that could not exist before the current AI wave are where distribution unlock happens.
“our particular approach and strategy here has been to invest heavily in applications, starting when we saw OpenAI APIs start to take off.”
- Model weights depreciate very quickly, and distillation makes the defensibility of any individual model extraordinarily challenging, so durable value must come from the application layer, network effects, and economic capture.
“the depreciation schedule for these models is frightening. Distillation as a technique makes defensibility of these models and these notes of these models extraordinarily challenging.”
- In a reasoning paradigm, application developers can capture performance gains from their own tooling and orchestration that model companies are unlikely to build, so their last-mile edge stays defensible even as reasoning models improve.
“I'm building technology and tooling that model companies are very, very unlikely to build.”
- Once a startup is at the frontier powering a specific use case, it is no longer GPU-constrained for serving a set of enterprise customers, so the hyperscalers become best friends rather than adversaries.
“if you are at the frontier, you're powering a use case, you're not particularly GPU constrained anymore”
Falling inference costs, idle capacity, and bursty usage make application margins strong.
- Some enterprises that pre-bought GPUs now have idle on-prem capacity, so running AI applications on-premises costs them near-zero marginal dollars, which lowers the cost of deploying applications and benefits vendors who sell into that capacity.
“if you have AI applications that run on-prem, it's essentially free. I have all this capacity, I'm not using it for anything.”
- Application companies can route most tasks to small models and reserve frontier models for a few prompts, producing near-zero inference cost and gross margins around 95 percent for some tasks.
“our cost of inference is essentially zero. And our gross margin for this task is 95%.”
- Training is a constant, saturating exercise that uses compute at maximum utilization for months, whereas application inference is extraordinarily bursty, so the hardware, cluster design, and utilization needs for inference differ from those for training.
“Training is just a constant exercise. You're scaling and you're just really hitting all your compute power all the time”
- Reasoning-based test-time compute will likely be used in only a minority of queries, because users are impatient and speed is the most important consideration for application developers, so total compute needs will be bursty rather than constant.
“It's very hard to squint and say it's going to be 90% of queries.”
- The current inference overcapacity will not last, because application developers historically consume all available capacity, so what is now surplus could become under-capacity as AI applications grow more sophisticated.
“Application developers are famous for using all the capacity and pushing the limits”
- The cost of inference for frontier models has dropped roughly 100x to 200x over about two years, a much steeper cost decline than the 15 years of cloud cost curves, and this underpins higher application gross margins.
“the cost of inference of these models is down a hundred X, 200 X.”
- Cerebras reports Llama 3.1 405B inference above 900 tokens per second, roughly 70 to 75 times faster than GPUs for inference, showing order-of-magnitude gains for inference-focused hardware.
“inference on Llama 3.1 405 billion for Cerebras is it can generate 900-plus tokens per second”
Application companies are supply-constrained, with visible demand and quick enterprise sales conversion.
- AI application companies are supply-constrained rather than demand-constrained, and their CEOs see demand as far as they can see, so they segment customers and go where they are first.
“more of these companies are supply-constrained than demand-constrained.”
- AI software spend grew roughly eightfold from 2023 to 2024, from a couple hundred million dollars to over a billion dollars in a year, and application companies appear more supply-constrained than demand-constrained.
“something like it's 8x year-over-year growth between 2023 and 2024 on just pure spend.”
- AI applications sell to enterprise CIOs in dramatically shorter cycles than SaaS, often 15 to 30 minutes to a yes, because demos combined with peer references and a 10x ROI case make the value obvious and pilots convert quickly.
“it's 15 minutes to a yes, 30 minutes to a yes.”
- When technology breakthroughs are dramatic and the future is open-ended, distribution also unlocks and customer acquisition becomes nearly free, making it a great time for early-stage investors to back R&D-driven teams.
“when you have big breakthroughs like this, everything sort of changes again. Distribution is nearly free.”
AI-first startups can now attack large horizontal SaaS markets long closed to new entrants.
- AI automation of document analysis may make law firms more profitable, because rote work that junior staff did and firms could not bill for is compressed from days to hours, freeing lawyers for billable strategic work.
“lawyers end up becoming way more profitable by using AI.”
- AI makes products in large incumbent SaaS markets such as document processing, which Chetan cites as OpenText, roughly ten times better, so startups can again go after large horizontal spend that was thought closed to new entrants.
“AI makes a product here easily 10x better, faster, and all the things that users want when they see it.”
- Benchmark's portfolio includes AI-first startups targeting large horizontal SaaS markets such as sales automation, accounting, financial modeling, circuit board design, game development, ad networks, and document processing against OpenText, implying that markets once considered closed to startups are open again.
“And now we're back to large swaths of horizontal spend again”
- Successful AI applications are not offering a 5 percent improvement over existing SaaS; they aim to eliminate significant amounts of software spend and human capital spend and move that budget into the AI solution.
“what we're presenting is not a 5% improvement over an existing SaaS solution. It's about we can eliminate significant amounts of software spend and human capital spend”
AI already shows extraordinary intelligence, and AGI is arguably already here or very close.
- Chetan expects AI to produce physics proofs and simulations humans could not produce, such as studying black holes or general relativity at new scales, and sees this as a plausible litmus test beyond current capabilities.
“I am confident we're going to start seeing that as it relates to physics proofs.”
- Video and audio data remain far from exhausted, so new modalities could produce capability leaps in human interaction that are as disorienting as image and video generation advances have been.
“we've exhausted data on text, we are not close to exhausting data on video and audio.”
- Humans keep moving the goalposts on AI, so anything a computer can do is relabeled as automation and a new frontier becomes the definition of AI, meaning current systems already exhibit extraordinary intelligence.
“humans are really good at changing the goalposts on expectations.”
- Under an expansive definition of AGI as a highly autonomous system that surpasses human performance in economically valuable work, AGI is arguably already here, and given end-to-end task completion progress, Chetan expects it to be very close or present in 2025.
“It's very easy to argue AGI is here using that lens.”
Incumbents face the innovator's dilemma, and Google defending its business would be extraordinary.
- The innovator's dilemma is playing out more aggressively in enterprise software than in consumer, because incumbent enterprise SaaS vendors cannot easily re-architect their stacks while consumer incumbents are moving to respond.
“innovator's dilemma in enterprise software is playing out much more aggressively in front of our eyes today than it is in consumer.”
- Google's consumer business remains dominant, and its enterprise cloud is now meaningfully scaled, so it may be disrupted but the disruption has not clearly happened, and whether it defends its business is an open question.
“their consumer business is dominant and there has been a perception that they're getting disrupted today.”
- Google's enterprise business is scaled and its consumer business is dominant, but it faces the classic innovator's dilemma, and there are few cases in business history where an incumbent prevented an innovator's attack, so Google defending its business would be an extraordinary achievement.
“there's very few cases in business history of the incumbent preventing the innovator's attack.”
Stable, reliable model APIs let application developers invest confidently in infrastructure.
- Stability at the model layer is a major boon for application developers, because they previously waited on step-function model leaps and did not know what to build, while a reasoning-based paradigm lets them invest confidently in infrastructure and UI.
“the model layer stabilizing is a huge boon for this application layer”
- Developers expect AI APIs to be as reliable as hyperscaler infrastructure, and the unreliability of AI APIs over the last two years has been a major challenge for application developers, who built workarounds.
“the quality of AI APIs has been a huge challenge for application developers.”
- If pre-training scaling is done, AWS is strongly advantaged because it has the biggest cloud, built for resilience, supports open-source model hosting and on-prem and GovCloud deployments, and can offer the most reliable APIs to developers.
“AWS has the biggest cloud. It has really built for resilience in a way that's very, very differentiated.”
Sell-side research gaps and Silicon Valley concentration are under-discussed features of AI innovation.
- Sell-side analysts have not yet produced substantial research on what the test-time compute paradigm means for the economics and structure of the industry, and this coverage gap is a significant under-discussed topic.
“just reading sell-side reports that we haven't seen sell-side reports or analysis on what this new paradigm of test-time compute means”
- Since November 2022 Benchmark has made 25 AI investments with a $500 million fund and five partners, an extraordinary pace compared with the normal five to seven investments per year, with four infrastructure companies and the rest application companies.
“we've made 25 investments in AI companies, and for a $500 million fund with five partners, that's an extraordinary pace.”
- AI innovation is concentrated in Silicon Valley because the labs, application companies, and infrastructure firms are located there or closely connected, and the fragility of this system means it must be protected and continued to be invested in.
“these human network effects are real and that Covid did not destroy them and that work from home did not destroy them”