causal

Inference batching favors large organizations

Continual learning may create inference economies of scale because per-company information requires full weight updates rather than low-rank adapters; a sparse model like DeepSeek V3 may need an optimal inference batch size above 2,400 sequences, so individual users serving themselves at batch size 1 could suffer a 100x+ compute efficiency penalty, favoring big organizations.

causalpending

Speaker

Dwarkesh Patel

Evidence Quote

“So the economics of serving personalized weights strongly favor big organizations.”

Source

Dwarkesh Patel Podcast - 8 Predictions for the Era of Continual Learning— Dwarkesh Patel Podcast
Created: 11/17/2026, 6:25:01 AM

Your notes

Loading notes...