Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
What this covers
In this conversation on Invest Like the Best, Neil Movva of Sail walks the full stack of AI inference, from model architecture to chips to power, for a company he calls a Token Factory. Sail's stated North Star is designing around serving speeds of one to ten tokens per second rather than fast interactive ones. He suggests that could even mean offloading cache to flash memory, though he flags that as a forecast requiring model architecture changes.
The hardware discussion is the most granular part. Movva explains the memory hierarchy: SRAM is fast but roughly three orders of magnitude less dense than HBM, and Blackwell carries 288 gigabytes of HBM against about 500 megabytes of on-die SRAM. The KV cache often outgrows the model weights, which limits SRAM-only designs. He covers why Cerebras-style wafer-scale chips can reach about 1,000 tokens per second yet handle the cache poorly. He also explains why batching raises utilization while lengthening each user's wait, and why sharding across eight GPUs yields only four to five times the speed.
Several claims are explicitly contested. Movva says performance per watt has barely improved across recent Nvidia generations or process nodes, and that losing TSMC would cost the West only about 2x in performance per watt against alternatives like Intel. He argues inference spend, unlike the Cisco-era buildout, is not speculative. On data, he says the internet was a one-time subsidy, that progress now depends on verifiable-task practice environments, and that distillation cannot be stopped. He also covers power: a megawatt fits in about eight liquid-cooled racks, while gigawatt sites are scarce.
Movva argues AI inference is shifting from low-latency chatbot serving to cheap, throughput-oriented background agents, and that a token factory can win on cost by aggregating stranded chips and power that hyperscalers cannot bid for.
- Background agents that run for hours make latency less important than cost per token, which favors throughput-optimized hardware and software stacks
- Concentrated, legible compute is won by well-capitalized frontier labs, so an inference provider can profit by scavenging fragmented chips and small pools of power
- Software edge in kernels will erode as frontier models improve, so durable advantage lies in supply-side arbitrage and rack-scale efficiency
Selling cheap open-model tokens to users who own their intelligence is the core business and desired future.
- Customers increasingly want to own intelligence and control the models they depend on, which created a robust market for customized or vanilla open-source models because the weights can never be taken away and can be deployed anywhere.
“They want to have control, sovereignty over the thing that they depend on. That created a much more robust market for customized models”
- Sail Research operates a token factory: an API serving open-source large language models at a price it claims is unbeatable, plus long-running agent virtual machines (Sailboxes) built for agents that run for hours, days, or weeks, with the company's North Star being the lowest cost per token in the industry by a wide margin.
“We have an API where anyone can send us requests where they can use large language models, open-source large language models for any task they want. We will serve those tokens to them at a price that is unbeatable in the market.”
- Existing open-model inference companies such as Baseten, Fireworks, and Together focused on low-latency inference because Cursor pulled them that way, which was the right choice a year ago but misaligned with the emerging need for persistence and long-horizon tasks.
“they all focus on low-latency inference and they were pulled in that direction by one very important customer — Cursor.”
- Unlike the Cisco-era networking buildout, which anticipated demand that never came, AI consumption is not speculative because people buy tokens for immediate use, and inference spend will increase monotonically, whereas earlier training-oriented spend was inherently speculative.
“It's no longer speculative. People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately.”
- The desired future has abundant tokens and diverse harnesses, where each company and user builds its own agent and customizes it mostly through in-context learning rather than weight fine-tuning, so people own their intelligence.
“I want abundant tokens and diverse harnesses. I want everyone to build their own harness.”
- At OpenAI pricing, one trillion tokens per day costs at least 5 million dollars, and the company's aim is three to six orders of magnitude cost reduction so that the same volume costs thousands of dollars, which makes large per-person consumption plausible.
“A trillion tokens. At OpenAI pricing, that's at least $5 million for 5.5 or 5.6.”
- There is always demand for intelligence; the real constraint is the on-ramps, the product designs that let people access it, and the company wants free-tier users to receive far more tokens than economics currently allow.
“I never will believe in that. There's always demand for intelligence in the world. The on-ramps to that intelligence are our challenge as a product community.”
Persistent background agents will drive inference demand toward unbounded consumption.
- Test-time compute scaling, where giving an agent more time yields better answers, was theorized about two years earlier but became bettable only around late 2025 with Claude Opus 4.5, the first agent suitable for longer-horizon tasks, and agent task length now reaches about an hour.
“wasn't really something that we could actually bet on until I would say late last year with Opus 4.5.”
- Because the average task length for agents keeps getting longer, only a few data points are needed to extrapolate an exponential curve showing that agents are worth running for longer periods, which justifies building infrastructure for them now.
“it doesn't take many points to have you draw out the exponential and see that agents are worth running for longer periods.”
- Tokens are not the final unit of work or intelligence but are the current unit; the future will shift toward agents completing units of outcome work, with token consumption becoming a dependent variable that agents self-budget rather than a fixed spend set by companies.
“I don't think tokens are the final unit of work or intelligence, but they are what we use today. After tokens, you start to move more towards outcomes”
- For individual users, the most exciting application is proactive background agents, such as a Siri running continuously to model a person's emails, texts, and life, surfacing next actions before the user asks, which requires abundant cheap inference.
“Siri's not very proactive. That's something that we can fix with abundant inference.”
- Background workloads will be about 50/50 with real-time workloads by the end of 2026 and will grow toward a 90/10 split in favor of background, because human attention caps interactive token use while background consumption is unbounded.
“Long-term, I think we're going to end this year at maybe 50/50 background and real-time workloads, but I see this going to 90/10 in favor of background.”
- The best latency is no latency: agents should work in the background on human timescales, checking in daily or weekly, because a human in the loop is the bottleneck on how much useful agent work can be done.
“the more you're in the loop as you prompt agents and wait for a response, in fact, you're the bottleneck in having the agent do more or less work.”
- Deep research over 10,000 or more sources, and building authoritative indexes of the internet that are monitored in real time, are tasks best done in the background, requiring an exabyte-scale intelligence workload that customers such as Parallel Web Systems are pursuing.
“they want to build an index over the whole internet and they want to monitor the internet in real time for changes.”
Background agents tolerate latency, so throughput-optimized, interruptible, shared compute is viable.
- Compute is not marshaled effectively across the world: many GPUs sit in warehouses or private pools allocated to single customers and go unused, and orchestrating the world's compute as a shared resource would greatly improve utilization.
“A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer, just don't get utilized.”
- The company will accept data centers with no backup diesel, minimal redundant networking, and single power sources, even at around 95 percent uptime, because a robust control plane can move workloads elsewhere and background agents tolerate interruptions.
“I'm that first buyer. I will buy 95% uptime.”
- If chips are cheap enough, the company does not mind the capital cost of idle capacity, so low-cost, possibly non-Nvidia racks paired with intermittent power are viable.
“if my chips are cheap enough — they're probably not going to be Nvidia racks — but if they're cheap enough, I don't mind the capital cost of having idle chips.”
- When a GPU is pulled away mid-task, an agent may lose an extra minute to ten minutes of latency on one turn, which is acceptable because customers' long-running agents are not waiting; the company will not control P99 latency and offers unbeatable economics in return.
“our P99, our 99th percentile latency, it's not going to be controlled. It cannot be. And in return, I'll give you unbeatable economics.”
- Sail's pitch to customers is competitive average throughput with an uncontrolled P99 latency, trading tail-latency guarantees for much lower prices, which background-agent customers accept.
“our average throughput is going to be very competitive, but our P99”
- GPUs are fundamentally throughput machines that run best on large batches, but the chatbot shape of AI usage forced the industry to optimize for latency, which leaves the GPU far from its happy path, so a stack built around throughput makes more sense for background agents.
“The GPU is fundamentally a throughput machine. The GPU is happiest when you give it a lot of work to do and let it chew through that work at peak utilization of its compute units.”
- Agentic inference is moving toward long-horizon background tasks running for hours or days, where latency matters less than cost, so an inference stack should be built for throughput and low cost per token rather than fast interactive responses.
“It doesn't matter if it spits out tokens at 100 tokens per second. Maybe ten is just fine if that comes with corresponding advantages in efficiency.”
- NVLink is mandatory for low-latency inference and tensor parallelism is basically Nvidia-only, so Sail does not care much about low latency and instead optimizes flops per dollar using other vendors' chips with alternative parallelism such as expert or pipeline parallelism.
“NVLink is mandatory for low-latency inference. ... what I actually optimize for in most cases, is how many flops does this chip have and how much is it going to cost me per hour to own and operate?”
- Sail's business is arbitrage: it extracts more performance from chips that receive less attention, resells that capacity at a margin, and wins by buying any chip anywhere for any duration.
“One of the ways that I describe what we do is we will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now.”
- Designing a system around serving at one to ten tokens per second, rather than fast interactive speeds, is Sail's North Star, and Neil believes offloading KV cache to flash memory at much larger scale could be achieved with model architecture changes.
“I'm quite passionate about that idea. Everyone on my team knows that I keep banging the drum around what would we have to change about the model architecture to make offloading KV cache to flash work at a much greater level.”
Winning comes from aggregate distributed chip and power supply, not head-on gigawatt frontier competition.
- Building 100,000 GPUs in one data center is far more expensive and difficult than building 10,000 or 1,000, and a gigawatt facility is no longer easy to build in the United States; even 100 megawatts is increasingly hard, so large concentrated power is scarce.
“There's actually a diseconomy of scale to data centers in some way. It's way more expensive and difficult to build 100,000 GPUs in one data center”
- Aggregate power exists in many one-megawatt and ten-megawatt sites, but the market has not valued it because training developers assume workloads must be concentrated in one spot, though a few thinkers now see inference fitting distributed sites.
“there's this incredible floor on the market where you can find lots of aggregate power, but it will not be concentrated.”
- With liquid cooling, one megawatt of compute can be packed into roughly eight refrigerator-sized racks rather than a massive warehouse data hall.
“Now you can pack that into around eight racks worth of compute. Each rack is about the size of a refrigerator.”
- Solar and wind power can be used for inference because workloads can tolerate multi-day outages, and the company can model weather to predict downtime and migrate workloads to other regions, gaining access to power nobody else will touch.
“I think that we're actually not that far from solving that problem.”
- Owning the entire stack from power to tokens is too capital-intensive, so the company will start with software and buy power through purchase agreements with utilities, pursuing vertical integration only after reaching scale that earns the right to take on more.
“The capital we're looking at for owning everything is insane. Software has high leverage, so we have to start with software.”
- The company cannot win concentrated compute against Anthropic or OpenAI and does not want to; instead it accumulates aggregate supply of chips and power that frontier labs do not find legible, building a factory from many small pieces.
“I'm never going to get concentrated supply. I will only get aggregate supply.”
- Nvidia does not sell tokens or neocloud compute directly because it is strategically more valuable to cultivate a diverse community of neoclouds and inference providers that compete to create demand for Nvidia chips, with CoreWeave as an example.
“He wants to create a diverse community of neoclouds and inference providers who are all jockeying to create demand for Nvidia”
- Nvidia engineers chase the theoretical speed of light for a chip, meaning the peak performance implied by its frequency and multiplies per cycle, and the company measures itself by absolute percentage of that limit rather than relative benchmarks against competitors.
“To this day, I tell all my engineers, we're chasing 100% speed of light. I don't care about relative numbers versus the competition.”
- The company hires for curiosity and a love of performance engineering, which cannot be taught, rather than for AI or CUDA experience, because GPU concepts have evolved so much in five years that tenure is a red herring.
“I don't look for CUDA experience at all. That's actually a huge red herring.”
Frontier advantages and durable software edges erode as open source and distillation diffuse capabilities.
- The next phase of data is models practicing hard verifiable tasks inside isolated RL environment gyms where they receive measured feedback on progress, and any task with a self-grading system provides a recipe for self-improvement.
“If you have that, you have the recipe for self-improvement on any task you like.”
- The premium labs charge for being three to six months ahead may not last because enterprises do not adopt bleeding-edge models quickly, with many still on Opus 4.6 or 4.7, and open source will repeatedly fill any vacuum a leader leaves.
“I don't know that the premium for being three to six months ahead is going to last that long.”
- Distillation is partly latent and unstoppable because AI-generated code and artifacts published on GitHub can serve as training signal, so preventing the diffusion of model capabilities is fundamentally impossible; only the speed is open.
“I would not be surprised if you could train a Fable-class model only on the outputs of code you consider good on GitHub that's open source.”
- Unconditioned feedback from everyday users is no longer worth much for training because the median deployed model is far more capable than a typical human rater, so expert preference is now the valuable signal.
“the signal you get from random human preference, unconditioned human preference, is not actually worth anything anymore. You want expert human preference at this point.”
- Software optimization near the speed of light will stop being a durable advantage for companies like Sail because the rising tide of frontier coding models lifts everyone, and kernel engineering is not a worthwhile specialization for model training.
“The rising tide of something like Mythos or GPT-5.6 Sol, that lifts all boats. ... we're all downstream of the frontier in terms of this capability.”
- Kernel work is not finished because new model features like changed positional embeddings (RoPE variants) break existing fused GPU kernels, forcing patches, and the ability to quickly modify kernels is valuable even though new ones are rarely invented from scratch.
“if there's a new model that comes out that has a slightly different way to embed positional information, like the change of the ROPE system — suddenly the kernel that we had is not suitable”
- Frontier labs used to spend most on data but now spend far more on RL environments, which capture recursive self-improvement on verifiable tasks and are the recipe for improvement on any task with a self-grading system.
“If you have that, you have the recipe for self-improvement on any task you like.”
- Open-source frontier models will persist because when one leader pulls ahead a new leader fills the vacuum, and training a frontier-class model gets easier every day.
“Fundamentally, I don't think open source ever goes away. If there's a vacuum because one leader steps out, a new leader will step in.”
- Neil is bullish on Nvidia in the short term and believes one should never bet against them because they reinvent themselves, even though the performance-per-watt stagnation suggests a contrarian view.
“I am bullish on Nvidia in the short term, and Nvidia, you should never bet against them. They're always going to reinvent themselves.”
Performance is capped by balanced structural trade-offs in memory hierarchy, precision, and silicon allocation.
- Few modern MoE models are more than 10 percent dense, and frontier models activate closer to 1 percent of experts, so compute is already used fairly judiciously and is not the main source of waste.
“very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated. And I think the frontier models are closer to 1%”
- Doubling every input would not solve the problem because bottlenecks are balanced across the system, so fixing one, such as a TSMC constraint, merely exposes the next bottleneck immediately.
“Everything grows in balance with each other. If we snap our fingers and double all those things, you might fix a TSMC bottleneck, but you're just going to run into another bottleneck.”
- Sharding a large matrix multiply across up to eight GPUs with NVLink reduces latency but scales sublinearly, so eight times the hardware yields perhaps 4 to 5 times the speed due to communication overhead, making tensor-parallel sharding a poor choice for throughput-optimized serving.
“You'll use eight times more hardware, but you won't get eight times the speed. You might get 4x to 5x the speed.”
- If the West lost access to TSMC, supply would take a shock, but the performance gap to the best Western alternatives such as Intel is only about 2x worse performance per watt at worst, far smaller than chip-war discourse suggests.
“My contrarian take is that it wouldn't be that bad. Supply would take a shock for sure, but the best processes that we have in the West, like Intel — not that far behind”
- Performance per watt for bfloat16 multiply has not improved dramatically from Hopper to Blackwell to Rubin, and moving from TSMC 5nm to 4, 3, and 2nm has not changed performance per watt dramatically, so the bottleneck is not process node gains.
“what is the performance per watt of bfloat16 multiply? It hasn't improved all that much.”
- Batching many users into one GPU pass maximizes parallel utilization, but each batch carries more total work per step, so any single user's request spends longer on the GPU, a trade-off inherent to GPU design.
“Any individual token or any individual user's request in that batch, it's going to spend a longer time on the GPU being carried with other people's traffic.”
- The first tensor cores occupied only about 5 to 10 percent of die area after fierce internal competition with graphics teams, because silicon allocation is guarded and Nvidia had to bet early on deep learning demand.
“We fought tooth and nail and got just a tiny bit of die area, maybe 5% to 10%, something like that for the first generation of these chips”
- The KV cache stores a representation of every token in the conversation and often becomes larger than the model weights, and because its size depends on users and context, it is the main limit on purely SRAM-based low-latency serving.
“it frequently gets to be larger than the weights of the model themselves.”
- Moving training from bfloat16 to FP8 or FP4 is an arbitrary-seeming choice with profound implications for which chips can be used and how hardware should be designed.
“We were training in bfloat16, but now we can train in FP8 or FP4, lower precision data types. That is an arbitrary choice, it feels like, but it has profound implications”
- Cerebras bets on wafer-scale integration with around 50 gigabytes of SRAM per wafer, stacked to about a terabyte and offering roughly 21 petabytes per second of bandwidth, enough to move a trillion-parameter model like Kimi in about a millisecond and reach around 1,000 tokens per second.
“Cerebras quotes petabytes per second, 21 petabytes per second for their Wafer Scale Engine 3. ... You can move all that data in and off the logic cores in about a millisecond or something like that. So there you go. You have a path to 1,000 tokens per second.”
- Chip design revolves around a memory hierarchy trade-off: off-chip memory gives capacity while on-chip memory gives speed, so hybridizing the two is necessary for models that need both large weights and dynamic context.
“You want off-chip memory for capacity and on-chip memory for speed. We want to hybridize these two things.”
Efficiency gains come from rack-scale design and model-written kernels rather than hand-tuned GPU code.
- Engineers should no longer hand-write GPU kernels; instead they design the strategy on a whiteboard, describe the input, output, and work dispatch in natural language, and a model executes the implementation.
“I like to say we write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing”
- GPU kernel efficiency was the company's starting point, but software edge will trend toward zero over time because frontier model improvements lift all boats, so the company is downstream of the frontier in kernel engineering capability.
“The rising tide of something like Mythos or GPT-5.6 Sol, that lifts all boats. It really does.”
- The unit of optimization is now the whole rack or cluster rather than one GPU, since Nvidia ships systems like NVL72 as prescribed racks of 72 units, and whoever programs rack-scale compute most efficiently will win on latency and throughput.
“We don't program one GPU at a time anymore. You should think about the whole rack, and maybe you should think about the whole cluster”
- Training is a superset workload of inference, so any training cluster can serve inference but not vice versa; the key difference is networking bandwidth between chips and how large the cluster must be.
“Training is a superset workload over inference. You can make any training cluster work for inference, but maybe not vice versa.”
- The largest inefficiency is in attention memory use: the KV cache is quite uncompressed, storing many kilobytes per token and likely off by an order of magnitude or two, and its entropy is not earning its keep.
“We're storing many kilobytes of data in the KV cache per token, and that's probably off by an order of magnitude or two.”
Imperfect non-Nvidia chips offer arbitrage, but scarce supply bottlenecks constrain how much can be secured.
- Large-dimension matrix multiplies run at 70 to 80 percent of peak utilization, limited by power rather than software, and Nvidia's quoted peak flops is optimistic because power throttling and thermals prevent hitting it.
“That operation runs at 70 to 80% of peak utilization, and it's limited not by software, but by power.”
- Any chip can be made to work at the right price; the company will run non-Nvidia hardware such as AMD, TPUs, and Trainium, because less-attended chips carry arbitrage value when competitors perceive them as inferior.
“I like to say there's no bad chips, there's only bad pricing. I will make any chip work at the right price.”
- Many people perceive AMD as inferior to Nvidia, which creates an opportunity to buy AMD supply cheaply, although AMD is now popular with large buyers such as Meta and OpenAI and its supply is being allocated.
“there's alpha because other people have this perception that AMD is not as good as Nvidia. That's music to my ears.”
- Nvidia allocates Blackwell compute strategically rather than to the highest bidder, because letting the deepest pockets take everything would hand a single customer long-term power, and relationships and financial backing now determine who can rent large capacity.
“They understand that compute is power today. They're quite strategic about how they allocate compute.”
- New chip startups such as Etched, SambaNova, and D-Matrix are emerging, but their main constraint is scale, specifically whether they can secure enough wafer allocation from TSMC to ship chips in volume.
“I think the main challenge for them is scale. Can they get enough wafer allocation from TSMC to pump out chips”
- TSMC tightens process corners to narrow the gap between best and worst chips, adding process controls that may be unnecessary; the company would accept more rejects and worse dies to lower overall cost per usable power.
“Maybe I'm willing to take a lot more rejects.”
- Chip startups should identify the three to five supply bottlenecks such as TSMC wafers, HBM, advanced packaging, and power, and be spiky by pushing hard in a direction incumbents underprice, such as HBM scarcity, since Nvidia's balanced design is hard to change.
“You want to be spiky. You want to pick something and say, I think they've underpriced the impact of how short we're going to be on HBM.”
- The company writes down characterized bottlenecks in chips such as Blackwell so it can share them with Nvidia and apply learnings to future chip purchases, folding invariant lessons into long-term decisions.
“We want to learn things that are invariant for us or the company long-term and fold that into future decisions that we make.”
- Every stage of chip production is in balance, so doubling one input such as TSMC capacity would just expose the next bottleneck; however, fabs should expose more of their trade-offs, such as process corner tightening, so buyers can make cost-efficient choices about worst-case chips.
“Everything grows in balance with each other. If we snap our fingers and double all those things, you might fix a TSMC bottleneck, but you're just going to run into another bottleneck.”
- Sail teams reason in the limit, planning for where efficiency could be in a month, six months, or a year rather than accepting current machine states, and write down invariant bottlenecks so learnings from one chip generation fold into future purchasing decisions.
“There's a lot of in-the-limit thinking. ... We don't accept the state of the machines we work on as fixed.”
SRAM accelerators must pair with off-chip memory because of capacity and speed trade-offs.
- Because there is no easy way to add memory fabs quickly and memory makers avoid big capex for cyclicals, HBM scarcity will push other products such as iPhones to cut memory and rise in price rather than the world routing around the shortage.
“I think they're going to make everything else more expensive. iPhones will cut their memory.”
- Cerebras- and Groq-style SRAM-heavy accelerators are best used alongside GPUs with off-chip memory, because on-chip SRAM gives speed for weights while off-chip memory gives capacity for KV cache; Nvidia and Groq may be doing exactly this.
“What they are really good at is being used in conjunction with a more traditional GPU-like device that critically has this off-chip memory built in.”
- Custom inference chips could sidestep the HBM shortage by offloading KV cache to flash memory at much greater scale, requiring model architecture changes to make that practical at serving speeds of one to ten tokens per second.
“It's about sidestepping the HBM shortage and focusing on more extreme offload to other forms of memory such as flash.”
- The original design flaw of transformers is placing a fundamentally memory-bound attention layer directly next to a compute-bound MLP layer, and no single chip is good at both, which forces hardware trade-offs.
“I would say the original sin of transformers is that you've taken this extremely fundamentally memory-bound layer and juxtaposed it right next to a compute-bound layer.”
- Wafer-scale SRAM architectures can host a trillion-parameter model's weights and reach about 1,000 tokens per second because weights stream in about a millisecond, but the growing, dynamic KV cache is a persistent limit those chips handle poorly.
“you can move the entire parameter count of a large model like Kimi. You can move all that data in and off the logic cores in about a millisecond or something like that.”
- On-chip SRAM is roughly three orders of magnitude less dense than DRAM-based HBM; Blackwell carries 288 gigabytes of HBM around its logic die against about 500 megabytes of on-die SRAM, though SRAM offers much faster access.
“Blackwell has 288 gigabytes of HBM capacity around the logic die. And the logic die itself maybe only has 500 megabytes of SRAM.”
- By using the whole wafer instead of reticle-limited dies, one can reach about 50 gigabytes of SRAM per wafer and stack wafers to reach around a terabyte, which is the path to very fast weight storage.
“We're going to take the entire wafer and have every die connect to every other die over scribe lines. And we can get to, let's say, 50 gigabytes of SRAM per wafer.”
Transformers scale predictably, but returns diminish as internet training data becomes exhausted.
- Transformers scale because attention lets any token relate to any other, making them general learners that find whatever patterns exist in a sequence, which explains their dominance in language and their ability to scale from millions to trillions of parameters.
“transformers represent any pairwise relationship that you want. Any token in the sequence can attend to any other token in the sequence.”
- The internet was a one-time subsidy on high-quality data, roughly 30 trillion tokens of high-quality text or 300 trillion on a wider definition, which models have already seen many times, so future progress depends on other data sources.
“I like the phrase that the internet was a one-time subsidy on data. We got it for free.”
- Increasing compute by ten times for a transformer yields a logarithmic improvement, but scaling laws have held so far, meaning returns are predictable though diminishing.
“You increase the compute available to a transformer by 10x and you'll get some log improvement somewhere. And so far the scaling laws really work.”
- A dataset-learning strategy is to first over-parameterize and overfit to prove a learning relationship exists, then compress to the smallest parameter count, since compression is what produces generalization.
“Once you can overfit, then you can compress, and the compression is how you get generalization.”
- Machine learning moved from linear models with thousands of parameters like support vector machines to deep learning computer vision with tens of millions and a 150 million parameter model being huge, and now transformers enable trillion-parameter models.
“transformers are the link to go from millions to trillions of parameters.”
- Tensor cores and matrix multiplication are the atomic unit of computation because linear algebra compactly represents arbitrary relationships in data, though Neil admits there is no deeper proof for why matrix multiplies are so central.
“It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships in data.”
Cheap tokens make exhaustive spending on verifiable quantitative problems such as security testing worthwhile.
- Non-verifiable tasks, such as human taste in writing and art, have not been solved and may not be solvable, so the company focuses on quantitative problems and leaves quality judgments to people.
“We have not solved human taste yet, and I don't know that it fundamentally can be.”
- The frontier of intelligence is jagged: no single frontier model finds a superset of software bugs, and small or cheap models sometimes find bugs that large ones miss, which encourages a diverse sampling strategy across many models for security agents.
“You'd find some bugs with a very small model that you don't find with a large model.”
- The unlock for cheap intelligence is being willing to spend tokens without any promise of return, because a verifiable problem has a hidden dollar cost that will fall to thousands, then hundreds or tens of dollars, making definitive answers to research questions affordable.
“It's not millions, it's thousands, and maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question”
- Software security is becoming a proof-of-work problem: how many dollars were spent on frontier APIs trying to break a piece of software is the best indicator of how secure it is, because agents can attack code faster than code can be generated.
“security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on Anthropic's APIs trying to break into your software?”
Full-stack expertise and mentorship are rare, valuable traits worth hiring for.
- Sail looks for people who are both good students and great teachers, runs frequent whiteboard sessions, and treats the state of chips as not fixed, documenting bottlenecks to inform future purchases and vendor conversations.
“We do whiteboard sessions all the time. The collegial environment where everyone has something to teach and something to learn is extremely important for us.”
- A rare, valuable trait is understanding the full stack from gate-level silicon to internet-scale service, and a college advisor showed this by reframing an impatient student's narrow focus.
“it's so rare that someone can actually understand the entire stack from the gate-level silicon all the way to building a great internet-scale service.”
- The kindest acts are mentors who spend time ensuring a student understands a difficult concept, and understanding the entire stack from gate-level silicon to internet-scale service is an exceptionally rare and noble expertise.
“It is such a rare trait. That level of expertise is so noble to chase.”