causal

A/B tests systematically bias toward short-term gains

Because randomized trials are statistically valid only for what they measure (short-term outcomes) and running long experiments is expensive, tech firms are systematically biased toward short-term improvements; summing all experiments might predict 300% revenue growth while actual growth was 10%, because interface changes confuse users initially and firms innovate in the wrong, short-term dimensions while missing radical long-term improvements.

causalpending

Speaker

Susan Athey

Evidence Quote

“they tend to systematically innovate in the wrong dimensions. So they are awesome at incremental short-term innovation--getting the fonts right... But if there is something more radical, they'll sometimes miss it.”

Source

Susan Athey on Machine Learning, Big Data, and Causation— EconTalk
Created: 6/13/2026, 3:35:37 AM

Your notes

Loading notes...