causal

A/B Tests Are Biased Toward Short-Term Gains

Tech firms systematically innovate in the wrong dimensions because running long experiments is expensive, biasing them toward short-term experiments; changes that confuse users initially but help long-term get penalized, so summing the short-term gains across years might predict 300% revenue improvement while actual improvement was only 10%—they excel at incremental short-term innovation but miss more radical changes.

causalpending

Speaker

Susan Athey

Evidence Quote

“they tend to systematically innovate in the wrong dimensions. So they are awesome at incremental short-term innovation... But if there is something more radical, they'll sometimes miss it.”

Source

Susan Athey on Machine Learning, Big Data, and Causation— EconTalk
Created: 6/13/2026, 12:14:45 AM

Your notes

Loading notes...