A/B test
An experiment comparing a control group (A) against a treatment group (B) to measure the effect of an intervention.
As defined in John List on Scale, Uber, and the Voltage Effect
Where it comes up · 9
- Susan Athey on Machine Learning, Big Data, and Causation10 claims
- #281 Working with Steve Jobs4 claims
- Gustav Söderström - How Spotify Thinks - [Invest Like the Best, EP.424]3 claims
- Pedro Domingos on Machine Learning and the Master Algorithm2 claims
- Pieter Levels2 claims
- Pulling the Thread1 claim
- John List on Scale, Uber, and the Voltage Effect1 claim
- Noah Shinn - Building Instinct: The Personal Agent - [Invest Like the Best, EP.493]1 claim
and 1 more
Most important claims · 10 of 25Search all claims for “A/B test” →
- Because randomized trials are statistically valid only for what they measure (short-term outcomes) and running long experiments is expensive, tech firms are systematically biased toward short-term improvements; summing all experiments might predict 300% revenue growth while actual growth was 10%, because interface changes confuse users initially and firms innovate in the wrong, short-term dimensions while missing radical long-term improvements.Susan Athey on Machine Learning, Big Data, and Causation
- Because A/B tests provide objective causal validation, tech firms can decentralize innovation: thousands of engineers, even summer interns, can write code, test it, and ship it live to users within days as long as the randomized control trial shows users responded positively and the change follows the rules.Susan Athey on Machine Learning, Big Data, and Causation
- Contrary to the popular image of lone-genius leaps like Steve Jobs designing fonts, most tech innovation is incremental, and the A/B test is the most impactful business process innovation in a long time; firms like Google run over 10,000 randomized control trials per year, and nothing ships without passing one.Susan Athey on Machine Learning, Big Data, and Causation
- Athey's Bing experiment re-ranking search results showed that how a search engine, news site, or commerce platform ranks results has a large causal effect on what people buy and read, implying technology platforms substantially shape the informativeness of the economy and the winners and losers in commerce.Susan Athey on Machine Learning, Big Data, and Causation
- Most of the 10,000 tech experiments are not designed to uncover fundamental generalizable truths about the world; they are specific to a firm's interface, often comparing two black-box algorithms where you never learn why one worked better, so even if published they would not yield generalizable parameters like risk aversion or class-size effects that economists seek.Susan Athey on Machine Learning, Big Data, and Causation
- Connected devices and cloud processing let companies run experiments in a day and A/B tests, making the scientific method super-powered, though founders still need direct customer contact to learn.Pulling the Thread
- A/B tests and efficacy tests use the best inputs a researcher can marshal, but those best inputs cannot be replicated at scale; failing to disclose that a result was an efficacy test causes voltage drops when policymakers later try to scale it.John List on Scale, Uber, and the Voltage Effect
- It is a common misconception that machine learning can only discover correlation; ML can and does discover causation by performing experiments such as A/B tests and randomized trials, and can go further than asking whether a drug works to identify exactly who it works for by examining all the other variables involved.Pedro Domingos on Machine Learning and the Master Algorithm
- Machine learning differs from traditional statistical methods like linear regression in two big ways: it embraces the full gamut of nonlinear models rather than only straight-line relationships, and it learns from data on all individuals rather than small aggregate data that omits important details.Pedro Domingos on Machine Learning and the Master Algorithm
- Google's data-driven design process factored taste out of decisions, so that when a company lacks a leader who understands design principles it eventually runs out of reasons for design decisions and doubt replaces instinct.#281 Working with Steve Jobs