batching
A process organization method where work items are grouped together and processed in large groups rather than continuously, creating queuing delays and overhead but often considered more efficient at large scale
As defined in The Man Who Proved Speed Kills Slow Companies | George Stalk Jr.
Other definitions
Grouping similar tasks (email, calls, errands, admin) into a single focused session to avoid the time cost of context switching.
Give Me 12 Minutes and I’ll Give You 30 Years of Productivity Advice
Where it comes up · 5
- Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]3 claims
- Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]2 claims
- Dwarkesh Patel Podcast - 8 Predictions for the Era of Continual Learning1 claim
- The Man Who Proved Speed Kills Slow Companies | George Stalk Jr.1 claim
- Give Me 12 Minutes and I’ll Give You 30 Years of Productivity Advice1 claim
Most important claims · 8 of 8Search all claims for “batching” →
- Continual learning may create inference economies of scale because per-company information requires full weight updates rather than low-rank adapters; a sparse model like DeepSeek V3 may need an optimal inference batch size above 2,400 sequences, so individual users serving themselves at batch size 1 could suffer a 100x+ compute efficiency penalty, favoring big organizations.Dwarkesh Patel Podcast - 8 Predictions for the Era of Continual Learning
- Inference is memory-bound because each parameter must be loaded from memory and used once, so batching many users to reuse each loaded weight amortizes the expensive load, and a transformer ASIC can batch around 2,500 users versus about 64 on GPUs.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Batching similar tasks together (email, calls, errands, admin) recovers time because every context switch costs time, whereas batching gives that time back.Give Me 12 Minutes and I’ll Give You 30 Years of Productivity Advice
- Batching many users into one GPU pass maximizes parallel utilization, but each batch carries more total work per step, so any single user's request spends longer on the GPU, a trade-off inherent to GPU design.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- GPUs are fundamentally throughput machines that run best on large batches, but the chatbot shape of AI usage forced the industry to optimize for latency, which leaves the GPU far from its happy path, so a stack built around throughput makes more sense for background agents.Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
- Batching (processing work in large groups rather than continuously) accounts for roughly one-third of the 95% wasted time in organizations, with another third going to batches queued and waiting to be processed, and the final third going to managing the batch flow.The Man Who Proved Speed Kills Slow Companies | George Stalk Jr.
- The future of models splits into very small on-device transformers for low-latency tasks like autocomplete and giant hyperscale models, while mid-sized models around 100 billion parameters have no niche because they are too big for devices and too small to justify compared with data center models.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]
- Because high efficiency requires huge batches of users grouped together, inference will be centralized in large data centers in much the same way as training.Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]