factual
Feistel networks make any function invertible
The Feistel cipher construction turns a non-invertible function f into an invertible two-input operation by keeping x alongside f(x)+y, so x is read directly and y is recovered as z minus f(x); this was ported into neural nets as RevNets (2017), making an entire transformer network invertible so activations need not be stored during training—they are rematerialized in lockstep during the backward pass, trading compute for memory savings.
factualpending
Speaker
Reiner PopeEvidence Quote
“because it’s invertible, I don’t need to store this at all. I can completely rematerialize it. I run my forwards pass, and then when I’m running my backwards pass, I’m simultaneously in lockstep undoing all of the forwards pass steps”
Source
Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:34:09 AM
Your notes
Loading notes...