causal
A/B tests systematically bias toward short-term gains
Because randomized trials are statistically valid only for what they measure (short-term outcomes) and running long experiments is expensive, tech firms are systematically biased toward short-term improvements; summing all experiments might predict 300% revenue growth while actual growth was 10%, because interface changes confuse users initially and firms innovate in the wrong, short-term dimensions while missing radical long-term improvements.
causalpending
Speaker
Susan AtheyEvidence Quote
“they tend to systematically innovate in the wrong dimensions. So they are awesome at incremental short-term innovation--getting the fonts right... But if there is something more radical, they'll sometimes miss it.”
Created: 6/13/2026, 3:35:37 AM
Your notes
Loading notes...