Philip Tetlock on Superforecasting
What this covers
Philip Tetlock and Russ Roberts discuss whether forecasting political and economic events can be treated as a teachable, measurable skill. The conversation centers on the Good Judgment Project, a competition sponsored by IARPA that pit teams of trained amateurs against intelligence professionals and academic experts. Tetlock describes how explicit probabilistic judgments scored by Brier scores create accountability that ordinary punditry avoids through vague verbiage forecasting—imprecise language that shields pundits from being proven wrong. The core claim is that this skill gap is real and substantial: well-organized teams of motivated amateurs systematically outperformed the classified intelligence analysts who had access to secret information.
The conversation examines the techniques that separate superforecasters from regular forecasters. Chief among them is the Inside View vs Outside View distinction: superforecasters begin with base rate reasoning—how often do dictators actually fall, or trade deals actually collapse?—before adjusting downward with case-specific details. Tetlock also explores how the Extremizing Algorithm can boost accuracy by aggregating independent estimates in mathematically sophisticated ways, and how Precision Questioning within teams exposes when members are using fuzzy language to hide uncertainty. The discussion turns to harder puzzles: whether Calibration improvements actually change decision-making in intelligence or policy, the tension between a leader's need to project confidence and the forecaster's duty to acknowledge grim base rate statistics, and why even the best forecasters leave large residual pockets of uncertainty. Throughout, Tetlock and Roberts weigh the Skill vs Luck ratio observed in the tournament—roughly 70:30—as evidence that judgment is neither pure noise nor infinitely refinable.
Tetlock argues that probabilistic forecasting of one-time political and economic events is a measurable, trainable skill, and that well-organized teams of motivated amateurs ('superforecasters') can systematically outperform experts and intelligence professionals when accuracy is rigorously scored over time.
- Vague verbiage forecasting and outcome-irrelevant learning make pundits unaccountable, while explicit probabilities scored by Brier scores create accountability
- Superforecasters win through base-rate (outside view) reasoning, team dynamics, training, and statistical extremizing algorithms
- The IARPA tournament showed roughly a 70:30 skill:luck ratio, proving good judgment is real but not pure noise
Superforecasters outperform intelligence analysts and pundits by cultivating subjective probability as a learnable skill.
- Superforecasters are a varied group—housewives, social caseworkers, pharmacists, Wall Street analysts, Silicon Valley programmers—who form effective 'mini-intelligence agencies' by working out their comparative advantages and allocating labor, generating estimates more accurate than many intelligence analysts.
“In effect what they created were mini-intelligence agencies. That were generating probability estimates more accurate than were coming out of many intelligence analysts.”
- What truly distinguishes superforecasters from seasoned intelligence analysts is the belief that subjective probability estimation is a cultivable skill worth cultivating; sophisticated analysts and pundits dismiss unique geopolitical events as unscoreable 'pseudo-precision,' and this very belief blocks them from learning to do it better.
“what really distinguishes them is that they believe that subjective probability estimation is a skill that can be cultivated and is worth cultivating”
- Superforecasters are measurably different from ordinary people—higher fluid intelligence, more politically knowledgeable, more open-minded—but these traits do not explain their edge over professional intelligence analysts, who match or exceed them on intelligence and knowledge.
“They score higher on measures of fluid intelligence; they are more politically knowledgeable; they are more open minded. But most important... I don't think they have those advantages over professional intelligence analysts.”
- In the IARPA forecasting tournaments, the Good Judgment Project's teams—including motivated amateurs—generated more accurate probability estimates than rival university teams, an internal prediction market (Inkling), and classified U.S. intelligence analysts who had access to secret information.
“we won the tournament by pretty resounding margins in the first two years”
Superforecasters use systematic methods—outside-view anchoring, statistical aggregation, precision questioning—that reliably improve forecast accuracy.
- Getting to truth in groups requires asking questions that may mildly offend, so teams must master disagreeing without being disagreeable and use 'precision questioning'—probing exactly what people mean by key terms—which pushes the limits of precision while maintaining group etiquette and exposes when people cannot actually be more specific.
“mastering the art of disagreeing without being disagreeable, and mastering what the art of what some consultants in California call precision questioning--we found that to be a very useful tool”
- When advisers each offer the same probability (e.g., 70%) but draw on diverse, independent sources of evidence, the mathematically correct aggregate is more extreme than 70%—because each estimate carries incremental information—whereas if they are clones drawing on identical information, the answer stays 70%; the IARPA extremizing algorithm exploited this and outperformed 99% of individual superforecasters.
“if they are drawing on different sets of information, processing it in different ways, each of them still arriving at 70% but not knowing the information the other people had... now, what's the correct probability? ... typically in the IARPA tournament you would extremize. You'd move from 70% to 85 or 90%.”
- Superforecasters are opportunistic and find statistical models in unlikely places—for a football game they will look at Las Vegas odds and aggregators like Nate Silver's FiveThirtyEight, average them as an initial estimate, and give only small weight to soft idiosyncratic information, which turns out to be a strong strategy.
“There are some very efficient information aggregators out there. Like there's Nate Silver at FiveThirtyEight and there's this and there's that. I'm going to take a look at each of those, and I'm going to average those and I'm going to take that as my initial estimate.”
- Superforecasters differ from regular forecasters by starting with the outside view—anchoring on the relevant base rate (e.g., the high survival rate of a dictator already in power more than a year or two) for maximum statistical leverage—before working inward to messy case-specific details that adjust the estimate.
“they try to get as much initial statistical leverage on the problem as they can before they delve into the messy historical details”
- Superforecasters can collectively usefully distinguish roughly 15 to 20 degrees of uncertainty along a probability scale, whereas most people distinguish only about 4 to 5, indicating genuine granularity beyond mere pseudo-precision.
“our best estimate is they can distinguish somewhere between about 15 and 20 degrees of uncertainty, along a probability scale. Most people distinguish about 5, 4.”
Current expert forecasting relies on ambiguous vague language that obscures actual confidence levels and prevents accountability.
- Pundits are smart, articulate, and knowledgeable, but it is nearly impossible to gauge whether their forecasts are correct because they rely almost exclusively on 'vague verbiage forecasting'—phrases like 'distinct possibility' that could mean anywhere from 20% to 80% probability depending on the listener's mood.
“they rely almost exclusively on what we call vague verbiage forecasting--they don't say that there's a 20% likelihood of something happening or an 80% likelihood”
- In a blame-game culture where people pounce on any explicit judgment that lands on the wrong side of 'maybe,' it is rational for forecasters to retreat into vague verbiage to protect themselves.
“If you exist in a blame-game culture in which people are going to pounce on you whenever you make an explicit fallibility[?] judgment that appears to be on the wrong side of 'maybe,' it's pretty rational to retreat into vague verbiage.”
Well-designed tournaments with rigorous daily scoring and adversarial collaboration can measure forecast accuracy and create accountability.
- Rather than hunkering down in philosophical positions (Bayesian vs frequentist) about whether one-time events can be assigned probabilities, the pragmatic approach is to run forecasting tournaments and empirically discover where the real limits of precision lie.
“really the right thing to do here is to run forecasting tournaments and explore what the limits of precision are”
- Forecasts were submitted at 9 a.m. Eastern every day during the tournament and run 'like a bank' with daily transactions, eliminating the wiggle room that plagues other research and allowing the aggregation algorithms to be rigorously tested in real time.
“we submitted forecasts at 9 a.m. Eastern time every day during the forecasting tournament. So there's no wiggle room here... This is being run like a bank, with transaction, every day.”
- The minimum wage debate is a good candidate for adversarial collaboration because many states and municipalities are independently raising wages, creating natural variation that can be turned into rigorously scorable probabilistic questions rather than unfalsifiable arguments.
“I think that minimum wage would be a wonderful example where adversarial collaboration could work, because there are so many states and municipalities taking independent action on that front.”
- Adversarial collaboration tournaments can create accountability in public policy debates: each side nominates relevant questions that pass the 'clairvoyance test' (rigorously scorable after the fact), and victory means answering your opponent's questions better than they can—which is far more awkward to dismiss than answering only your own.
“Each side would have a chance to nominate 5 or 10 questions that it thinks it has a comparative advantage in answering... they have to pass the clairvoyance test--which means that they have to be rigorously scorable[?] for accuracy after the fact.”
Teams of superforecasters consistently outperform individuals by organizing labor and pooling diverse independent evidence sources.
- Effective leadership and superforecasting are in tension: leaders are expected to project confidence (the coach saying 'we'll win') rather than cite the grim base rate, raising the question of when leaders are supposed to be 'liars,' and whether it is better to have a self-deceiving leader or a two-faced one with private and public numbers.
“the need for leadership and the need for confidence are in some degree of tension: the need for circumspection and the need for confidence are in tension with each other”
- An experiment resolved the debate over individuals versus teams: after the first year, teams were significantly better—roughly 10% better—because well-guided teams can be more than the sum of their parts, with many small cumulative factors producing a large overall advantage for superforecaster teams.
“we resolved it by running an experiment. And it turned at the end of the first year that teams were better. Significantly better. How much better? Maybe 10%.”
Forecast accuracy improvement depends on domain-specific decision value and people systematically misremember prior probability judgments.
- Psychologists tend to emphasize overconfidence, but underconfidence is also a genuine danger; President Obama appears to have drawn an underconfident conclusion from the roughly-50% probability judgments offered by advisers in the Osama bin Laden decision.
“Psychologists, you are right, tend to emphasize the dangers of overconfidence. But there is also the danger of underconfidence.”
- The IARPA geopolitical tournament showed roughly a 70:30 skill-to-luck ratio based on observed regression-to-the-mean effects; the existence of experimental manipulations (training, teaming) that reliably improve accuracy proves the phenomenon is not pure noise, since you cannot improve a radically noisy dependent variable.
“Our best estimates are that the geopolitical forecasting tournament sponsored by IARPA had about a 70:30 skill:luck ratio, based on the regression toward the mean effects that we were observing.”
- It is impossible to measure the accuracy of a probability judgment for a single event, except in limiting cases where someone assigns probability zero or 1.0; accuracy can only be assessed across many events and time periods, where good judgment means consistently assigning higher probabilities to things that happen than to things that don't.
“assessing the accuracy of individual events is impossible, except in those limiting cases. But it is possible to assess the accuracy of many events and across many time periods.”
- Whether improved probability estimates lead to better decisions depends on the domain: in options pricing and finance the value is clear, but a senior intelligence official had never even heard the question of whether knowing a Russian incursion was 20% rather than 1% would change action—so the value of accuracy increments must be weighed against the cost of achieving them.
“'if you had known that the probability of Russian incursion into the Ukraine was not 1% but it was 20% during the Sochi Olympics, would you have done something different?' and we got an interesting reaction. It was: 'I've never even heard a question like that before.'”
- A well-calibrated forecasting system that says 75% will be wrong 25% of the time, so throwing out the system every time it lands on the wrong side of 'maybe'—as Leonhardt did with prediction markets after the Obamacare ruling—would leave you with no well-calibrated forecasting systems at all.
“if you are going to throw out a very well-calibrated forecasting system every time it's on the wrong side of 'maybe,' you are not going to have any well-calibrated forecasting systems at your disposal”
- Even the best forecasters in the IARPA tournament on average do not do much better than assigning 75% to things that happen and 25% to things that don't, leaving large residual pockets of uncertainty—but the tournament showed it is possible to do something very smart people had supposed was impossible.
“the very best forecasters on average are not doing much better than assigning a 75% probability for things that happen and then 25% probability for things that don't”
Experts resist entering level-playing-field tournaments due to reputational risk and operate in outcome-irrelevant learning situations resistant to updating.
- People systematically misremember their prior probability estimates, pumping them up retrospectively—experts who assigned low probabilities to the Soviet Union's dissolution later recalled assigning higher than 50%—an effect psychologists call hindsight bias, which persists even when you know the other person knows the true prior.
“people really pumped up those probabilities retrospectively. So, the psychologists call that 'hindsight bias' or the 'I knew it all along' effect. And we saw that in spades in the Soviet Forecasting Tournament.”
- High-status pundits are rationally reluctant to enter level-playing-field forecasting tournaments because the best outcome doesn't justify the reputational risk; the only force capable of inducing them is a public groundswell demanding testable predictions, where refusal begins to damage their credibility.
“The only way we are ever going to induce high-status pundits to agree to participate in level playing-field forecasting tournaments... is if the public demands it.”
- Experts exist in an 'outcome-irrelevant learning situation'—no matter what actually happened with the Soviet Union, people could always interpret the outcome as consistent with their prior worldview, appropriating credit or deflecting blame, so they never updated their views.
“it was as though we were in an outcome-irrelevant learning situation. It didn't really matter what happened--people would be in an excellent position to interpret what happened as consistent with their prior views.”