What this covers

Phil Tetlock explores what forecasters actually deliver and what they should. The conversation ranges across psychology, politics, and history to ask why accuracy alone fails as a measure of forecasting value—people also seek reassurance, entertainment, and protection from regret—and why most forecasters, especially entrenched experts, have little incentive to chase it. The centerpiece is the forecasting tournament, where accountability to accuracy alone, stripped of ideology and status games, reveals genuine skill. Tetlock then turns this tournament logic toward counterfactual reasoning about history, proposing a framework to test historical arguments that currently operate as ideological safe havens.

The discussion moves through several distinct arenas. On forecasting mechanics: simple statistical algorithms outperform human judges far more often than expected, and the skill lies in knowing when to break a trend rather than extend it. On talent and recruitment: why superforecasters tend to be foxes—thinkers of integrative complexity with exposure to multiple cultures—rather than hedgehogs, though scientific progress often requires hedgehog persistence. On barriers to adoption: organizations resist tournaments because high-status insiders cannot win, only tie or lose. On epistemology and bias: academics invoke falsifiability but protect their beliefs like sacred values; even Tetlock's own attempts to override aggregated forecasts degraded his accuracy. On policy: whether epidemiologists inflating pandemic probabilities do forecasting or manipulation, and whether machines can handle the geopolitical questions the intelligence community actually poses. The thread connecting these is value pluralism—how exposure to conflicting frameworks sharpens judgment—and the hard truth that there is no view from nowhere from which to judge anything neutrally.

Sharpest takeaway

Tetlock argues that accuracy is only one of many things we actually want from forecasters, that disciplined forecasting tournaments can isolate accuracy from ideological and status incentives, and that the same rigor should now be extended to counterfactual reasoning about history.

  • People seek reassurance, entertainment, and regret-minimization from forecasters, not just accuracy
  • Forecasting tournaments create monistic accountability that aligns incentives purely toward accuracy
  • Counterfactual reasoning can be validated by linking it to logically connected conditional forecasts

The argument · threads10 threads · 34 claims
0.80

Forecasting tournaments reveal measurable skill by filtering accuracy from ideology and incentivizing honest probability judgment over reassurance.

5 pointscentrality 5/5
  • An aggregation algorithm built from forecasters' judgments outperformed 99.8 percent of the very forecasters from whom it was derived; Tetlock's own attempts to second-guess the composite where he thought it was excessively aggressive significantly degraded his accuracy, dropping him from the top fraction to the middle of the pack.

    “We had an aggregation algorithm that was performing very well at the time, and it was outperforming 99.8 percent of the forecasters from whom the composite was derived.”

  • The spread of forecasting culture runs against psychological and sociological grain because people naturally treat their beliefs as ego-defining, quasi-sacred possessions rather than falsifiable probabilistic propositions, and entrenched subject-matter experts have no incentive to enter tournaments whose best possible outcome for them is a tie.

    “People naturally gravitate toward thinking of their beliefs as ego-defining, quasi-sacred possessions. That's one major obstacle.”

  • In intelligence-community settings, prediction markets typically do not perform quite as well as forecasting tournaments, but because those markets are not deep, liquid, or backed by real money, this is not a decisive rebuke of the market mechanism for eliciting forecasts.

    “when prediction markets fall short — and they do typically not perform quite as well as forecasting tournaments — when they fall short, it's hardly a decisive rebuke of the market mechanism for eliciting forecast.”

  • Forecasting tournaments attract upwardly mobile young people who see them as a fast track to rise, while high-status senior experts inside organizations rationally resist them because being placed on a level playing field against 25-year-olds can only threaten the status they already hold.

    “young people who are upwardly mobile see forecasting tournaments as an opportunity to rise. Old people like me and aging baby-boomer types who occupy relatively high status inside organizations see forecasting tournaments as a way to lose.”

  • Forecasting tournaments create a stark monistic accountability in which only accuracy matters and ideological cheerleading is punished, which is unusual; most real-world accountability occurs in organizational matrices full of distortions where the rational response is strategic attitude-shifting toward important others, evasion, and procrastination.

    “Tournaments create a very stark monistic type of accountability in which one thing and only one thing matters, and that is accuracy.”

0.71

Expert consensus and institutional incentives systematically diverge from accuracy-seeking, distorted by ideology, status, and competing accountability demands.

6 pointscentrality 4/5
  • One of the great challenges of our time is striking the right balance between democracy and technocracy, complicated by reflexive skepticism toward science on both the right and increasingly the left, in a society where expert guidance is increasingly crucial.

    “I think one of the great challenges of our time is striking the right balance between democracy and technocracy.”

  • A public health expert who believes underestimating a virus is far worse than overestimating it is not forecasting at all but engaging in manipulation and social influence to shape public opinion and behavior, which is a legitimate but distinct activity from accuracy-seeking prediction.

    “There, you're not forecasting. You're engaged in manipulation, social influence.”

  • Accuracy is often not the first thing we want from forecasters; we look to them for ideological reassurance, entertainment, and for minimizing regret functions by pumping up the probabilities of events we would regret not anticipating.

    “We want a lot of things from our forecasters, and accuracy is often not the first thing. I think that we look to forecasters for ideological reassurance, we look to forecasters for entertainment, and we look to forecasters for minimizing regret functions of various sorts.”

  • Although most academics pay lip service to Popperian falsifiability and Bayesian probabilism in their epistemological self-concept, how they actually behave when their egos are at stake is quite different, which is why adversarial collaboration — getting opposing camps to jointly test their claims — is extremely rare and hard to pull off.

    “There's an epistemological formal self-concept, which is kind of noble, falsificationist, and probabilistic. And then there's how we actually behave when our egos are at stake in particular controversies.”

  • There is a deep intellectual fissure between social science and conservatives — most social scientists are liberal and conservatives are wary of their advice — and this distrust likely contributed to the slowness of the Trump administration's response to COVID-19.

    “It's about the intellectual fissure that exists between social science and conservatives, that most social scientists are liberal, and conservatives are wary of advice from social scientists.”

  • There is no such thing as pure value neutrality — 'no view from nowhere' — because even with a perfectly objective forecasting system, whoever generates the questions can promote a political agenda and skew the results, so claims of value-freedom by bureaucracies warrant suspicion.

    “The old expression, "There's no view from nowhere." There is no such thing as pure value neutrality.”

0.70

Experts systematically overestimate their predictive advantage over simple statistical baselines and algorithmic aggregation.

3 pointscentrality 5/5
  • Simple extrapolation algorithms are historically hard to beat in forecasting; the real skill lies not in projecting the trend but in knowing when to alter it — accelerate, decelerate, or change direction — and humans have been repeatedly humbled in competitions against simple statistical algorithms.

    “simple extrapolation algorithms, historically, are hard to beat. We're running a COVID-19 mini forecasting tournament right now, and they're proving to be hard to beat. The skill, of course, is when to alter the trend”

  • Empirical evidence from Expert Political Judgment shows subject-matter experts' confident predictions are often not materially more accurate than minimalist statistical baselines like simple extrapolation, which Michael Gove invoked to question expert predictions about Brexit consequences.

    “portions of that book compare subject matter experts to minimalist statistical baselines like extrapolation. Can you predict simple extrapolation algorithms? And the answer was often no.”

  • To build a composite that out-predicts most superforecasters, take the most recent forecasts of the best forecasters and extremize them; when normally-disagreeing diverse observers converge, that convergence signals the weighted-average composite is too conservative and should be pushed further from 0.5.

    “it's also to extremize that forecast to the degree that people who normally disagree, agree with each other. And when you have convergence among diverse observers, that's a signal that the weighted average composite is probably too conservative and you should extremize.”

0.66

Cognitive diversity, perspective-taking, and integrative complexity—cultivated through exposure to conflicting values and viewpoints—strengthen forecasting ability.

5 pointscentrality 4/5
  • People who grow up with a second culture appear to have some forecasting advantage because accountability to conflicting audiences and value pluralism gives them a richer internal dialogue and stronger perspective-taking skills, which are important parts of superforecasting.

    “It ties into accountability. Accountability to conflicting audiences and value pluralism. You have a richer internal dialogue. You learn to balance conflicting perspectives more.”

  • Integrative complexity reflects value pluralism: the more frequently a political thinker confronts cognitive dissonance between conflicting values, the more pressured they are toward complex synthetic thinking, whereas lopsided values permit simpler dissonance reduction like denial or bolstering one value while downplaying another.

    “the more frequently that you, as a political thinker, confront cognitive dissonance between your values, your value orientations, the more pressured you are to engage in integratively complex synthetic thinking.”

  • A combination of fluid intelligence and integrative complexity boosts forecasting accuracy, with fluid intelligence probably being the more powerful predictor, though the effect sizes for integrative complexity are not huge.

    “I think that fluid intelligence is probably a more powerful predictor. A combination of fluid intelligence and integrative complexity, I think, does boost up forecasting accuracy, yes.”

  • To get the most from a team of forecasters or problem solvers, choose a leader who knows how to shut up, withhold opinions at the start of a meeting, and listen, then elicit independent judgments uncontaminated by conformity pressure before allowing free critique and only later revealing who holds which position.

    “I would pick a leader who knows how to shut up, and not reveal opinions at the beginning of the meeting, and knows how to listen.”

  • To become more fox-like, deliberately expose yourself to dissonant viewpoints (a liberal should read the Wall Street Journal, a conservative the New York Times) and cultivate connectable interests outside your field at an optimal distance — far enough to be stimulating but close enough to reconnect to your core work.

    “If you're a liberal, read the Wall Street Journal. If you're a conservative, read the New York Times. Expose yourself to dissonant points of view.”

0.51

Counterfactuals lack ground truth and become ideological refuges, though indirect validation via Bayesian networks and logical consistency is possible.

2 pointscentrality 4/5
  • Because history cannot be rerun and there is no ground truth, counterfactuals are unresolvable and become a refuge where ideologues can invent whatever facts justify their position — e.g., that the Iraq War was worth it because things would have been worse under Saddam — making them central to drawing lessons from history yet impossible to adjudicate directly.

    “counterfactuals a place where ideologues can retreat. They can make up the data, make up whatever facts they want to justify pretty much whatever.”

  • Although counterfactuals about the actual world have no ground truth, you can indirectly validate them by measuring people's counterfactual beliefs and their logically connected conditional forecasts in a Bayesian network — someone who thinks Russia would be hostile regardless of NATO expansion will predictably favor more arms sales and sanctions — and better counterfactual reasoners should show less ideological polarization.

    “you can measure people's beliefs about conditional forecasts that are logically connected to the counterfactuals in a kind of a Bayesean entrance network.”

0.47

Hedgehogs achieve major scientific breakthroughs through relentless pursuit of single ideas despite foxes being better average forecasters.

4 pointscentrality 3/5
  • Although foxes tend to be better forecasters, many of the great achievements in science come from hedgehogs who pursue a single big idea relentlessly.

    “I've always resonated a little more to the foxes than to the hedgehogs, but I think when you look at the great achievements in science, they often come from hedgehogs.”

  • Scientific progress depends on a degree of quasi-irrational persistence and overconfidence, since most projects most scientists embark on do not succeed and a rational Bayesian would simply defer to existing expert opinion rather than attempt the seemingly impossible.

    “Most projects that most scientists embark on, I think, don't succeed. It does take a certain amount of quasi-irrational persistence.”

  • Academic advisors have a tendency to undervalue weird students, even though contrarian weirdness combined with sufficient brightness creates the possibility of doing something distinctively well, as Tetlock's own advisor recognized in him.

    “There's probably a tendency in that direction, yes.”

  • Hyperprofessionalization, compartmentalization, and publication inflation in universities make it hard for people with weird cross-disciplinary interests to gain traction, effectively producing hedgehogs on an industrial scale even though many great scientific advances come from such specialists — but the system leaves too little room for eclectic foxes.

    “There is this tendency for training in universities to become hyperprofessionalized and compartmentalized. So it is harder for people who have weird interests that straddle, say, psychology and organizational science and political science and law and history.”

0.38

Machine intelligence cannot match human judgment on novel geopolitical problems lacking deep base rates and repeated cases.

2 pointscentrality 3/5
  • There is little evidence that machine intelligence is well equipped for the geopolitical problems the intelligence community poses (Syrian Civil War, Russia/Ukraine, Mars settlement) because base rates are elusive, unlike well-defined repeated tasks such as screening credit card applicants where machines dominate humans.

    “The things like the Syrian Civil War, Russia/Ukraine, settlement on Mars — these are events for which base rates are elusive. It's not like you're screening credit card applicants for Visa.”

  • Statistical estimates from rounding experiments show that for the kinds of geopolitical questions the intelligence community poses, the best forecasters can meaningfully distinguish between 10 and 15 degrees of uncertainty — considerably more than the five or seven gradations intelligence bodies assume they can use.

    “Our best statistical estimates are that forecasters, for the types of questions the intelligence community often poses, can distinguish between 10 and 15 degrees of uncertainty, which is considerably more than the seven they think they can now, a lot more than the five they thought they used to be able to distinguish.”

0.29

Professional success, talent identification, and superforecasting ability correlate strongly with fluid intelligence and thrive in rich talent pools.

3 pointscentrality 2/5
  • Given the overlap between superforecasting ability and intelligence, and between intelligence and professional success across many fields, superforecasting techniques almost certainly also help predict how successful people will be.

    “everything we know, from the overlap between superforecasting and intelligence and the overlap between intelligence and success in many different professional lines of work, would suggests that that's almost certainly true.”

  • There is substantial similarity in the cognitive ability and cognitive-style profiles of superforecasters and the staff of elite hedge funds like Goldman Sachs, so a contest between them would be a tight race rather than a clear win for either.

    “I think there's a good deal of similarity in the cognitive ability, cognitive-style profiles of superforecasters and the kinds of people you see on the staffs of Goldman Sachs. It would be a tight race.”

  • Spotting talent is extremely hard when the base rate of talent falls below one in a thousand, but venture capitalists in Silicon Valley succeed partly because the talent pool is relatively rich, raising their odds above the population base rate, and because a few hits pay for many false positives.

    “It's very, very hard to forecast, to identify talent when the base rate falls below 1 in 1,000, 1 in 10,000, 1 in 100,000.”

0.29

Interdisciplinary knowledge integration and structural reforms enabling diverse perspectives face organizational resistance that survives only through institutional champions.

3 pointscentrality 2/5
  • Attempts to integrate disciplines (Jim March at UC Irvine, Penn Integrates Knowledge under Amy Gutmann) run against the grain because departments naturally want to claw back resources, so such programs may not survive beyond the leader who champions them.

    “I'm not even sure it'll survive at Penn beyond Amy. The natural tendency will be for departments to want to claw back the resources.”

  • Inducing epidemiologists to inflate the annual probability of a viral leap into humans would generate a crying-wolf effect, because repeated high-probability warnings that fail to materialize erode credibility.

    “They'd been saying there's a 30, 40, 50 percent chance of a viral leap into human beings each year, and it didn't happen. Didn't happen. Didn't happen. You get a crying-wolf effect, right?”

  • The threat of a SARS-like pandemic from a large reservoir of viruses in horseshoe bats combined with exotic meat consumption was a mainstream epidemiological position and common knowledge in microbiology for the first two decades of the 21st century, so society did not lack the forecast.

    “It was a mainstream position among epidemiologists for the last 20 years or so that you had a recipe for disaster.”

0.05

Pandemic risk was widely known in epidemiology before emergence, showing society possessed the forecast but lacked institutional willingness to act on it.

1 pointcentrality 1/5
  • A survey of prominent historians, reported in Unmaking the West, places the median judgment of when the ascent of the West became more or less inevitable at roughly 1730 to 1740.

    “If you looked at just the unweighted average of judgments from the median, I think was probably around 1730, 1740.”