YouTube11m· Jul 2026· cataloged

This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)


What this covers

This is a Better Stack walkthrough in which Andras tests Colibrì, a pure-C engine with no external dependencies written by Vincenzo Fornaro. It runs GLM 5.2, a 744-billion-parameter model in the same size class as data-center frontier models, on a MacBook M2 Max and on an RTX 5090 workstation. The video starts with why such a model doesn't fit on ordinary machines, since even at int4 quantization it is hundreds of gigabytes (about 357 GB on disk). It then explains how Colibri works and ends with two hardware tests and a verdict.

The explanation leans on mixture of experts sparsity: only about 40 billion parameters are active per token. The roughly 17 billion dense parameters stay in RAM, and individual experts are streamed from SSD on demand and cached. The video lists several supporting tricks. A compressed attention cache shrinks the KV cache about 57 times. A learning cache pins frequently routed experts into spare RAM at startup. Speculative decoding uses a multi-token prediction head. The forward pass is reported as token-exact against a reference transformers implementation. Colibri treats VRAM, RAM and NVMe as one cache hierarchy.

The tests are where the argument turns. On the Mac, the model is far larger than the 32 GB of RAM, and the profiler output shows most of the time spent waiting on disk I/O. An external USB drive, at roughly 1 GB/s against 7 GB/s or more for an internal one, makes this worse. On the workstation, the GPU is barely taxed, and more RAM is what shortens the waiting. Andras then cites Colibri's own benchmarks, in which a GPU-less dual-Xeon server with 1 TB of RAM beats a single 5090. He connects this to RAM prices and hyperscalers buying up supply. His closing position is that running a frontier model and running it well are different things, and that a smooth experience on an average laptop is close but not here yet.

Sharpest takeaway

Running large frontier language models (744B parameters) on consumer hardware is technically possible via Colibri's memory-hierarchical architecture, but practical performance depends critically on RAM availability and storage speed rather than GPU power.

  • Colibri exploits mixture-of-experts sparsity to stream only active experts from disk, reducing active memory needs from 744B to 40B parameters per token
  • RAM and disk speed are the primary bottleneck, not GPU compute—the Mac test showed 80% disk-waiting vs 7 seconds actual work; the workstation test with more RAM cut waiting to 48%
  • Even a dual-Xeon server with 1TB RAM but no GPU outperforms a single RTX 5090, proving memory hierarchy matters more than accelerators for this workload

The claims · ranked31 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.74

GLM 5.2 is a 744 billion parameter language model in the same size class as frontier models running in data centers, but Colibri enables it to run on consumer hardware.

factualhigh valueestablishednovelty 3/4durability 3/4· Andras

“This is a 744 billion parameter language model, the GLM 5.2. For context, that is the same size class as the Frontier models running in data centers full of GPUs.”

0.66

GLM 5.2 uses a mixture of experts architecture where instead of one dense network, the model is split into thousands of specialist sub-networks (experts), and for any given token only a tiny fraction of those experts are active.

factualhigh valueestablishednovelty 1/4durability 4/4· Andras

“GLM 5.2 is a mixture of experts model. We've talked a lot about mixture of experts models on this channel before, but in case you need a refresher, it basically works like this. Instead of one giant dense network, it's split into thousands of smaller specialist sub-networks called experts. And for any given token, the model only uses a tiny fraction of those experts.”

0.66

Out of 744 billion total parameters in GLM 5.2, only 40 billion parameters are actually active per token, which is roughly 5% of the total model size.

factualhigh valueestablishednovelty 1/4durability 4/4· Andras

“Out of 744 billion total parameters, only 40 billion are actually active per token. And that's roughly 5%.”

0.66

An external drive over USB has transfer speeds of roughly 1 GB per second, while internally-mounted drives achieve 7 GB per second or more, a 7x difference that significantly impacts Colibri's performance since it constantly streams model pieces from disk.

causalhigh valueestablishednovelty 1/4durability 4/4· Andras

“An external drive over USB is a lot slower than a drive built into your machine. We're talking maybe 1 GB per second versus seven or more internally. And remember, Kolibri's whole trick is constantly streaming pieces of the model off the disk. So, if the disk is slow, everything is slow.”

0.61

The dense components of GLM 5.2 are only about 17 billion parameters, or under 10 gigabytes at int4 quantization, and remain resident in RAM at all times.

factualhigh valueestablishednovelty 1/4durability 3/4· Andras

“The model's dense components, the parts used on every token, are actually small, about 17 billion parameters, and that's under 10 gigabytes at integer four quantization. And those stay resident in RAM the whole time.”

0.61

Colibri uses speculative decoding with a native multi-token prediction head that guesses several tokens ahead, enabling faster generation than one token per step.

factualhigh valueestablishednovelty 1/4durability 3/4· Andras

“Second, there's speculative decoding. The model has a native multi-token prediction head that guesses several tokens ahead, so it can generate faster than one token at a time.”

0.61

The forward pass in Colibri is validated to be token-exact against a reference transformers implementation, ensuring that the streaming and quantized version produces identical outputs to the full-precision version.

factualhigh valueestablishednovelty 1/4durability 3/4· Andras

“Third, and this is an important one, it preserves quality. The forward pass is validated to be token exact against a reference transformers implementation. In plain terms, Vincenzo checked that this streaming quantized version produces the same outputs as the full precision reference.”

0.61

Colibri uses compressed attention cache (MLA attention) that shrinks the KV cache by about 57 times, which significantly reduces memory footprint.

factualhigh valueestablishednovelty 1/4durability 3/4· Andras

“And it also uses a compressed attention cache, the MLA attention that shrinks the KV cache by about 57 times, which is a big part of why the memory footprint stays so small.”

0.55

On the workstation test, the RTX 5090 GPU was 'barely breaking a sweat'—the real bottleneck was not GPU compute but rather the amount of RAM available to cache the model, making storage hierarchy (how much of the model fits in RAM vs must be fetched from SSD) the critical constraint.

causalhigh valuespeaker onlynovelty 3/4durability 4/4· Andras

“Well, it would be tempting to say, 'Well, obviously, because it has an RTX 5090 on it.' But that's not really the reason. You see, the reason is that this machine has twice the RAM and it was on a faster internal drive. So, a much bigger portion of the model can actually just sit in memory ready to go. Less running back to the disk, less waiting, and more actual work getting done. So, the 5090 was actually off in the corner barely breaking a sweat at this point.”

0.55

A dual-Xeon server with a full terabyte of RAM but no GPU outperforms a single RTX 5090 by a significant margin on Colibri benchmarks, demonstrating that RAM and CPU are more critical than GPU acceleration for this workload.

causalhigh valuespeaker onlynovelty 3/4durability 4/4· Andras

“And notice that the terabyte of RAM server with no fancy GPU at all beats a single RTX 5090 by a mile.”

0.55

The single biggest constraint on running large frontier models on normal hardware is not GPU compute power but RAM availability—specifically, how much of the model can remain in RAM to avoid constant disk I/O.

causalhigh valuespeaker onlynovelty 3/4durability 4/4· Andras

“You see, the reason is that this machine has twice the RAM and it was on a faster internal drive. So, a much bigger portion of the model can actually just sit in memory ready to go. Less running back to the disk, less waiting, and more actual work getting done. So, the 5090 was actually off in the corner barely breaking a sweat at this point. And that's the thing I did not expect going into this experiment. You'd assume that the graphics card is what makes or breaks this, right? Well, it isn't. The single biggest thing holding these models back on normal hardware is actually memory. How much of the model you can keep in RAM at once so you're not constantly waiting on the drive.”

0.55

According to Colibri's official benchmarks, the hardware setups that achieve usable speeds are still serious machines, such as six-GPU rigs or dual-Xeon servers with full terabyte of RAM.

factualhigh valueestablishednovelty 0/4durability 3/4· Andras

“If you look at Vincenzo's own benchmarks on the Calibri site, the setups that actually get usable speeds are still serious machines. The fastest configs are things like a six-GPU rig or a dual Xeon server with a full terabyte of RAM.”

0.55

At int4 quantization, a 744 billion parameter model is still hundreds of gigabytes, which is why large models require data centers full of GPUs and expensive specialized hardware.

causalhigh valueestablishednovelty 0/4durability 3/4· Andras

“Now, what if you could run that model locally on your consumer hardware? It sounds almost impossible, but this developer, Vincenzo Fornaro, just made a tool called Colibri that lets you do just that. Now, how is that even [0:30] possible? How did Vincenzo pull it off? And how does it even work? Well, those are all good questions that we're going to explore in this video, and we're going to test it out on two separate hardware setups to see how powerful it actually is. It's going to be a lot of fun, so let's dive into it. So, normally to run a 700 billion parameter model, you'd need to fit all its weights into memory at once. And at an integer four quantization, 744”

0.55

Vincenzo Fornaro, the developer of Colibri, successfully created a tool that exploits mixture-of-experts sparsity to enable running frontier-scale language models on consumer hardware.

factualhigh valueestablishednovelty 0/4durability 3/4· Andras

“this developer, Vincenzo Fornaro, just made a tool called Colibri that lets you do just that.”

0.55

GLM 5.2 requires approximately 357 GB of disk space to store the full model.

factualhigh valueestablishednovelty 0/4durability 3/4· Andras

“GLM 5.2 is about 357 GB, and to test it out properly, I would have to free up almost half of my MacBook's SSD space, which I can't do.”

0.52

RAM has become very expensive, which explains why hyperscalers are aggressively acquiring all available RAM they can obtain, as it is now a critical limiting resource for large model deployment.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Andras

“So, no wonder all of these hyperscalers are buying up all the RAM they can get their hands on.”

0.52

Yes, a 744 billion parameter frontier model can be run on consumer hardware (laptop and desktop), as demonstrated by successfully running Colibri on both machines and receiving correct answers.

factualhigh valuespeaker onlynovelty 3/4durability 3/4· Andras

“Can you run a 744 billion parameter Frontier model on consumer hardware? Yes, full stop. I did it twice on a laptop and on my desktop, and both of them gave me real correct answers. A couple of years ago, that would have sounded absurd, but Vincenzo genuinely pulled it off.”

0.50

Colibri uses a learning cache that records which experts prompts route to in a file, and automatically pins the most-used experts into spare RAM at startup, improving cache performance for repeated task types.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Andras

“First, it has a learning cache. The engine records which experts your prompts actually route to in a file next to the model. At startup, it automatically pins the hottest, most used experts into spare RAM. So, the more you use it for a particular kind of task, the better it caches for that task.”

0.47

Running a frontier model and running it well are two very different things—while Colibri enables execution on consumer hardware, achieving usable speeds requires more serious machines.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Andras

“But, the honest truth is that running it and running it well are two very different things.”

0.47

Colibri treats the machine storage as one big memory hierarchy, layering GPU VRAM, system RAM, and NVMe SSDs like a cache system, with dense components resident in RAM and experts streamed from SSD on demand.

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Andras

“Colibri treats your machine storage as one big memory hierarchy. Your GPUs VRAM, your system RAM, and your NVMe SSDs layered like a cache. The model's dense components, the parts used on every token, are actually small, about 17 billion parameters, and that's under 10 gigabytes at integer four quantization. And those stay resident in RAM the whole time. The experts, those live on your SSD.”

0.47

The M2 Max CPU is very fast, but the bottleneck on the MacBook is not the CPU—it's that the 744B parameter model doesn't fit in the Mac's 32GB RAM, forcing the computer to repeatedly fetch pieces from the external SSD, which causes most time to be spent waiting rather than working.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Andras

“Well, the M2 Max is a very fast CPU chip. That is not the problem. The problem is that the model is so big that it doesn't fit in the Mac's 32 GB of RAM. So, it has to keep running back to the disk to grab whatever piece it needs next. And in my case, the disk is a slow external drive, which I was forced to use because I ran out of space. So, the computer spends most of the time waiting and barely any time actually working on the request.”

0.46

When the model needs a specific expert for a token, Colibri streams just that expert off disk, uses it, and caches it, rather than loading the entire model into memory.

factualhigh valuespeaker onlynovelty 1/4durability 4/4· Andras

“When the model needs a specific expert for a token, Colibri streams just that expert off disk, uses it, and caches it.”

0.46

Colibri is written entirely in pure C with zero external dependencies, which makes it exceptionally lean and efficient.

factualhigh valuespeaker onlynovelty 1/4durability 4/4· Andras

“And the whole thing is written in pure C with zero external dependencies. And that is the part that makes it so lean.”

0.46

The dream of a frontier model running smoothly on an average laptop is not quite here yet, but we are very close to achieving this.

forecasthigh valuespeaker onlynovelty 2/4durability 2/4· Andras

“So, the dream of a Frontier model humming along on an average laptop isn't quite here yet, but we are very, very close.”

0.43

On the MacBook test, the profiling breakdown showed that of 158 seconds total time for one generation, approximately 145 seconds (over 80%) was spent waiting for disk I/O, while only 7 seconds was actual computation.

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Andras

“So, on the Mac, the whole turn took about 158 seconds. And of those 158 seconds, roughly 145 of them, over 80%, was pure waiting. Waiting on the disk. The actual math or the real thinking part took only about 7 seconds in total.”

0.43

On the workstation test with 64GB RAM, the profiling showed disk waiting dropped to about 48% of total time (half the MacBook's 80%), while actual compute time increased substantially, demonstrating that more RAM reduces disk I/O dependency.

causalhigh valuespeaker onlynovelty 1/4durability 3/4· Andras

“Remember on the Mac, over 80% of the time was just waiting on the disk? Well, on this beefy workstation, the waiting drops to about 48%. Basically, cut in half. And for the first time, you see that the machine is actually spending real chunks of the time doing math instead of just waiting around.”

0.35

On the MacBook M2 Max test with 32GB RAM and an external SSD, Colibri correctly answered 'What is the capital of Belgium?' with 'Brussels', demonstrating that the tool works for basic queries, but with poor speed: over 2 minutes for the first word at about 0.1 tokens per second.

factualhigh valuespeaker onlynovelty 0/4durability 2/4· Andras

“I kept it simple and asked it, 'What is the capital of Belgium?' And here's the good news. It works. It actually answers me. It tells me the capital is Brussels, correctly, running a 744 billion parameter model on a laptop. So, that part is genuinely impressive. The not-so-good news is the speed. I waited over 2 minutes just to get the first word out of it. 2 minutes. And once it got going, it was crawling along at about 10th of a token per second.”

0.35

On the RTX 5090 workstation with 64GB RAM and an internal drive, Colibri achieved 0.8 tokens per second and returned the first word in about 17 seconds—roughly 8 times faster than the MacBook test.

factualhigh valuespeaker onlynovelty 0/4durability 2/4· Andras

“And speed-wise, it hit about 0.8 tokens a second. And the first word showed up in around 17 seconds instead of the whole 2 minutes. So, roughly eight times faster than on my Mac.”

0.34

A token is roughly a word or a chunk of a word, making 0.1 tokens per second roughly one word every 10 seconds on the MacBook test.

definitionestablishednovelty 0/4durability 4/4· Andras

“For context, a token is roughly a word or a chunk of a word. So, we're talking one word every 10 seconds.”

0.24

So, it all comes back to the memory, folks.

factualspeaker onlynovelty 0/4durability 4/4· Andras

“So, it all comes back to the memory, folks.”

0.20

On the MacBook test, this was a cold start where nothing was cached yet, forcing the system to fetch every piece of the model from the external drive from scratch.

factualspeaker onlynovelty 0/4durability 3/4· Andras

“To be fair, this was a cold start. The first question, nothing was cached yet, so it had to go fetch every piece of the model off that drive from scratch.”