A/B testing
An experimental method (pioneered by Google in 2000) that shows different versions of something to two randomly split groups to measure which produces a better outcome.
As defined in et_920
Where it comes up · 17
- SmileDirectClub: Closing the Gap with Affordability3 claims
- Investor, Essentialist, Patriot2 claims
- How Games Make Life Better2 claims
- A Primer on Performance Marketing2 claims
- 108 | Carl Bergstrom on Information, Disinformation, and Bullshit2 claims
- Finding the Next Figma, Wiz, & Stripe Before It's Obvious | Neil Mehta Interview1 claim
- John Deere: Centuries of Farming Innovation1 claim
- Investing In Brands1 claim
and 9 more
Most important claims · 10 of 23Search all claims for “A/B testing” →
- Continuous compounding improvement in performance marketing comes from the volume of meaningful tests a team can run, and learning from each test about which variables matter most, not from outspending competitors.A Primer on Performance Marketing
- Tailored online advertising is not a purely benign or positive force; it is a segregating force that serves as an opportunity for those with money and education but as a predatory channel for the poor, who get targeted by for-profit colleges and payday lenders—and crucially, when these algorithms fail their targets, the harm is invisible to everyone else, unlike a public financial crisis.Cathy O'Neil on Weapons of Math Destruction
- Physical products require iterating upfront because they ship once and remain fixed, whereas digital products can ship A/B tests constantly and roll back failures, so good design in software comes largely after the initial draft.Dylan Field - Designing The Future - [Invest Like the Best, EP.407]
- Zynga pioneered data-driven A/B testing, giving half of players one version and measuring which keeps them engaged, tracking quit rates and day-two return predictors, rather than relying on unchecked designer assumptions.How Games Make Life Better
- Digital attribution and DTC funnel data now let marketers tie specific messages and timing to sales, and A/B tests on Facebook give quick reads, so advertising no longer has the excuse of being an inefficient science.Investing In Brands
- The application of A/B testing and randomized control trials to poverty eradication revealed that many interventions previously hoped to work did not, while the simple idea of giving people money directly to make them less poor worked surprisingly well.Michael Faye and Paul Niehaus on GiveDirectly
- Through constant real-time A/B testing on their user interfaces, tech companies effectively know more about specific aspects of human psychology—such as which button colors get clicked and which headline wording works—than the collective discoveries of the academic psychology community, and they know it tailored to each individual user.108 | Carl Bergstrom on Information, Disinformation, and Bullshit
- Google and Meta have a unique advantage in using AI for advertising because ad A/B testing at global scale provides a verification loop in which human clicks and purchases validate generated images and text, and predictive models can lift relevance by a few percentage points worth billions of dollars.Ben Thompson on Big Tech, China, and the AI Boom Running Out of Money - [Invest Like the Best, EP.487]
- Recommendation algorithms like YouTube's are optimized to maximize engagement rather than education or accuracy, and through constant A/B testing they discover that people move toward more extreme content over time, which radicalizes users and pushes them into filter bubbles, as when watching an ISS feed leads to flat-earth video recommendations.108 | Carl Bergstrom on Information, Disinformation, and Bullshit
- Agriculture struggles to run A/B tests because no two seasons or soil regions are alike, so input makers find it hard to prove results, while Deere can more easily market equipment outcomes such as higher speed or less spraying.John Deere: Centuries of Farming Innovation