factual

Feistel networks make any function invertible

The Feistel cipher construction turns a non-invertible function f into an invertible two-input operation by keeping x alongside f(x)+y, so x is read directly and y is recovered as z minus f(x); this was ported into neural nets as RevNets (2017), making an entire transformer network invertible so activations need not be stored during training—they are rematerialized in lockstep during the backward pass, trading compute for memory savings.

factualpending

Speaker

Reiner Pope

Evidence Quote

“because it’s invertible, I don’t need to store this at all. I can completely rematerialize it. I run my forwards pass, and then when I’m running my backwards pass, I’m simultaneously in lockstep undoing all of the forwards pass steps”

Source

Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:34:09 AM

Your notes

Loading notes...