What this covers
Jean-Baptiste Kempf and Kieran Kunhya discuss the unseeable machinery that powers video and audio across the internet with Lex Fridman. The conversation centers on FFmpeg, VLC, and related tools—codecs, decoders, and encoders that move terabytes daily but receive almost no public recognition. The two speakers argue that this infrastructure exists because a small, under-resourced volunteer community has chosen engineering excellence and ethical refusal over profit, and that the work requires both extraordinary low-level craft and structural patience to sustain.
The conversation moves through several distinct territories. It covers the history of video compression, tracing how hobbyist anime encoders broke with decades of mathematical optimization to optimize for human perception instead, and how this shift led to x264 and modern codecs. It explores the technical depth: handwritten SIMD assembly that runs orders of magnitude faster than compiler output, the practice of reverse-engineering proprietary codecs from binary blobs, and the sprawl of patent liability that haunts the field—enough that Google, Meta, and Netflix built royalty-free alternatives like AV1 to escape it. The speakers examine how License as Social Contract works in practice, the difference between GPL and LGPL, and why FFmpeg as Rosetta Stone matters to archivists. They discuss Maintainer Burnout frankly, including the XZ attack and personal threats maintainers have received. Throughout, the theme is that a handful of people support more platforms than trillion-dollar companies, that refusal of money and government pressure is structural to the work, and that rewrites usually destroy what they aim to improve.
Kempf and Kunhya argue that the invisible video infrastructure of the internet (FFmpeg, VLC, x264, dav1d) is built and maintained by a small volunteer community pursuing engineering excellence for the greater good, and that this requires uncompromising low-level craft (handwritten assembly), ethical refusal of compromise (no ads, no backdoors), and structural patience (don't rewrite, maintain).
- Handwritten SIMD assembly beats compiler auto-vectorization by multiples, and matters because FFmpeg/dav1d run on billions of devices where every cycle counts
- Open source is sustained by a license-as-social-contract and a small set of maintainers who are under-resourced and under-thanked by trillion-dollar companies
- Refusing money (ads) and refusing power (government backdoors) is core to the project's integrity
Codec standards, testing rigor, and lossless archiving enable long-term preservation and interoperability that proprietary alternatives cannot provide.
- Video codecs remove both spatial and temporal redundancy (e.g., reusing an unchanged cloud across frames) using several orders of magnitude more compute to compress than to decompress, because compression is done once while there may be many viewers, and modern codecs also build in error resilience to recover from lost packets.
“using several orders of magnitude more compute to compress because that's more costly... versus the decompression. So it's asymmetric in that respect.”
- FFmpeg is used by NASA's Mars 2020 rover to compress pictures, as the mission deliberately used as much commercial off-the-shelf technology as possible, making FFmpeg a multi-planetary open source library.
“FFmpeg is used by the Mars rover, the Mars 2020 rover to compress pictures... they really wanted to use as much commercial off-the-shelf technology as possible”
- Reverse engineering a proprietary codec involves finding the decompression module in a multi-megabyte binary, hooking it to dump raw YUV data as a bit-exact comparison reference, then stepping through a disassembler instruction by instruction (sometimes in a VM to handle DRM/encryption) using intuition about codec patterns (DCT, entropy coding) to reconstruct the algorithm; roughly one megabyte of blob equals a month of work.
“one megabyte binary blob to reverse engineer is probably order of magnitude a month of work, and this guy is doing 20, 30 megabyte blobs”
- The video archiving community, lacking budgets but valuing preservation, funded the FFV1 lossless codec and GPU encoding in FFmpeg because they fear compression losing essential details; they need fast, lossless, resilient encoding (recovering only a portion of a frame if bits flip), and treat FFmpeg as a Rosetta Stone written in C so multimedia can be played a thousand years from now.
“They funded the development of the FFV1 codec, so that's a lossless codec... the archiving community is really scared about the act of compression losing things”
- Archivists face a moral hazard because there is a finite, dwindling number of tape heads left in the world to read old tapes from the 1970s-80s, forcing them to decide what to archive and discard the rest, with the added value that lossless archiving may later reveal gems hidden in details that lossy compression would have destroyed.
“there's tape, and there's not enough tape heads in the world left to redo, so they have to decide what they want to archive and throw away the rest of the tapes”
- Most video codecs from the 2000s onward have a bit-exact definition, meaning every implementation must produce exactly the same bits for a given sample; the lack of bit-exactness in 1990s MPEG-2 is acknowledged as one of the biggest mistakes the video industry made.
“every implementation must produce exactly the same bits, bit for bit, in exactly the same data that comes out of a decoder”
- FFmpeg's FATE (FFmpeg Automated Testing Environment) runs on a huge volunteer-hosted matrix of operating systems, compilers, and instruction sets (macOS, iOS, Windows, Linux, PowerPC, RISC, ARM, DEC Alpha) to catch compiler miscompilations where a compiler compiles C code incorrectly, which can cascade into large video glitches due to frame dependencies.
“FFmpeg has a system called FATE, FFmpeg Automated Testing Environment. Because FFmpeg runs on so many different OSs and can be compiled with so many different compilers”
- LLMs understand assembly better than expected but still fail to produce bit-exact assembly, repeatedly making the same non-bit-exact modifications when corrected, because there isn't a large corpus like Stack Overflow and not enough data to train on for this specialized domain.
“I go, 'Is it bit exact?' 'No.' 'Fix it.' And then it just goes and does the same thing... There isn't the corpus of information like Stack Overflow to work on.”
FFmpeg democratized multimedia technology by making expensive, specialized tools universally accessible and enabling creator economy.
- FFmpeg democratized video production by making technology that in the 1990s cost hundreds of thousands of dollars and was the size of a car available to everyone on a near-level playing field, enabling the podcast, YouTube, and streaming revolution and giving voice to individuals worldwide.
“FFmpeg was a big player in that because it democratized this technology that was once in the nineties... you needed equipment that cost hundreds of thousands of dollars... It was the size of a car”
- x264 was a step change because hobbyists (mostly anime encoders) bucked the 20-year industry obsession with mathematical metrics like PSNR/mean-squared-error (which cause blurring) and instead optimized for psychovisual rate distortion and adaptive quantization, biasing bits toward complex areas to look good to humans on ordinary laptops rather than scoring well on metrics.
“industry and academia for 20 years before, was obsessed with mathematical metrics or what's known as peak signal-to-noise ratio... that led to loads and loads of blurring. So hobbyists bucked that trend.”
- VLC and FFmpeg are not competitors but a binary star system that coexists because of each other (analogous to Android being to Linux); x264 (a VideoLAN project) underlies 80%+ of FFmpeg pipelines, VLC gives FFmpeg reach and exposure to weird files and funding, and many developers are shared, forming a virtuous cycle.
“VLC is to FFmpeg as Android is to Linux. So they depend on each other, but they, they coexist because of each other. So they are a binary star system”
- Unlike browsers where multiple engines (Firefox, Chrome) coexist, FFmpeg has been a winner-takes-all in multimedia because each new codec added is worth more than the codec itself, since it makes the whole framework better, creating a compounding network effect.
“FFmpeg has kind of won it all... because of the fact that you can get—every new codec added is actually worth more than the value of that codec itself because it makes the whole thing better”
Corporations inappropriately demand vendor-like support from volunteer open-source projects, ignoring resource and structural imbalances.
- Trillion-dollar corporations treat open source projects like contracted vendors with SLAs and public bug trackers like third-party Jira, demanding free urgent support; as in the XZ fiasco and Microsoft Teams marking an issue high priority then offering only a few thousand dollars one-time, they fail to recognize these are volunteer hobby projects, not suppliers.
“They think an open source project is a traditional vendor that they have an SLA. They think a public bug tracker is actually, you know, a third-party vendor's Jira”
- Google used AI to generate security vulnerability reports on FFmpeg, applied a standard 90-day disclosure deadline, and announced to media how good their AI was before issues could be fixed, placing a disproportionate burden on volunteers given the means imbalance between automated discovery and manual patching, often over obscure 1990s game codecs marked as high priority.
“Google started using AI to create security reports on an open source project, FFmpeg. Volunteers had to deal with that... they even went to the media first announcing how good their AI was before the issues could be fixed.”
FFmpeg achieves extreme performance through low-level optimization techniques and specialized technical knowledge that compilers cannot match.
- FFmpeg/dav1d achieve extreme speed by abusing the machine in unintended ways, including using cryptography instructions for unrelated video processing and bypassing the operating system's function calling convention to create custom calling conventions within their own libraries, avoiding costly register save/restore through L1/L2 cache.
“we abuse the machine. We go and use the machine in ways that the creator didn't expect. Sometimes we use an instruction that's completely unrelated to what we do. We use a cryptography instruction in video processing”
- Handwritten SIMD (single instruction multiple data) assembly delivers 10x-50x (sometimes 62x) speedups over scalar code and is multiple times faster than compiler auto-vectorization or intrinsics, contradicting the common belief that good compilers can match handcrafted assembly; this matters because FFmpeg/dav1d run on order-of-magnitude a billion CPUs where every cycle counts.
“we have a 10x, and not percentage, 10x to 50x speed improvement. That function is 62x”
- Assembly is poorly taught in books and online because it's grammar-focused (going through every instruction) rather than problem-focused; like learning a spoken language by communicating real needs rather than studying sentence structure, assembly should be taught targeting real problems, and it only requires high school math and an understanding of pointers.
“you don't, in general, learn a language from learning the grammar and the structure. You learn a language by asking someone what their name is”