holdout group
A small set of users deliberately withheld from a launch so the team can measure the change's effect against them.
As defined in How to Build a Great Product
Other definitions
A randomly chosen population that is deliberately not shown an offer, used as a control to measure what would have happened without the treatment.
Cardlytics: The Ad Platform with Purchasing Power
Where it comes up · 3
- Cardlytics: The Ad Platform with Purchasing Power2 claims
- How to Build a Great Product1 claim
- Gokul Rajaram - Lessons from Investing in 700 Companies - [Invest Like the Best, EP.456]1 claim
Most important claims · 4 of 4Search all claims for “holdout group” →
- Because Cardlytics sees spending across both the exposed group and a randomly assigned holdout group, it can run a randomized controlled trial and measure incremental spend from an offer to the penny, which is unusual in advertising.Cardlytics: The Ad Platform with Purchasing Power
- Once an ad-free surface starts showing ads, user engagement tends to decline over time as ads siphon engagement, so companies should keep holdout groups that never see ads and set an engagement budget capping the revenue-driven engagement loss, as Facebook did with News Feed.Gokul Rajaram - Lessons from Investing in 700 Companies - [Invest Like the Best, EP.456]
- Major product changes should be launched decisively to nearly everyone at once rather than tiptoed in, because half-launches create social confusion and make users doubt the company's conviction; Instagram Stories launched to about 99.5 percent of users in 2016 with only a brief holdout.How to Build a Great Product
- Cardlytics can run randomized store-level or SKU-level experiments, such as icing selected stores or selling iced lattes, so advertisers see the effect in their own sales data and do not have to trust a report or understand a randomized trial.Cardlytics: The Ad Platform with Purchasing Power