Joshua Angrist on Econometrics and Causation
What this covers
Joshua Angrist of MIT and host Russ Roberts examine the foundations of empirical economics—specifically, how economists move from data to causal claims. The core disagreement centers on what has actually improved the credibility of economic measurement. Angrist contends that progress stems from better research design, particularly randomized trials and quasi-experimental methods like instrumental variables, regression discontinuity, and differences-in-differences, rather than from the kind of thorough specification sensitivity analysis that Ed Leamer once championed. Roberts pushes back throughout, arguing that empirical findings frequently fail to replicate and that economists often construct after-the-fact explanations to preserve their existing theories rather than revising them in light of contradictory evidence.
The conversation moves across several domains where research design has shaped scholarly understanding. Angrist walks through case studies—the Oregon health insurance lottery, charter school admissions lotteries, David Card and Alan Krueger's minimum-wage work, and regression discontinuity studies of elite schools—to show how credible designs can shift consensus even among skeptics. He argues that flawed earlier work, such as Isaac Ehrlich's capital-punishment regressions, suffered not from inadequate robustness checks but from weak research design altogether. A secondary theme emerges around the role of social science in identifying interventions that do not work—microfinance, computers in early education—thereby redirecting resources elsewhere. Roberts counters with concerns about selection bias in observational settings, the pervasive failure of replication in development and epidemiology, and the difficulty of achieving precommitment in non-experimental work. The pair conclude by discussing how one ought to assess reliability in polarized policy debates, where both honest error and honest disagreement remain ordinary.
Angrist argues that the credibility of empirical economics has improved primarily through better research design (random assignment and quasi-experimental methods like IV, RD, and differences-in-differences) rather than through Leamer-style sensitivity analysis, while Roberts pushes back that findings often fail to convince skeptics or replicate, especially in macro.
- Convincing causal inference depends on credible research design, not on mapping all possible model specifications
- Well-designed studies (RAND, Oregon, Card-Krueger, charter lotteries) have shifted scholarly consensus even when partisans remain unconvinced
- Roberts counters that results often don't replicate and that macro estimates differ even in sign
Natural experiments and causal identification methods yield credible estimates when properly applied to real-world variation.
- The differences-in-differences method depends on the assumption that treatment and control groups would have moved in parallel absent treatment; this is partly testable, and many panel results disappear once you allow group-specific trends, as happened with Angrist's compulsory-attendance-law study.
“We know what that method turns on. It turns on parallel trends. We always say that. It lives or dies with parallel trends.”
- Because oversubscribed charter schools allocate seats by lottery, researchers can compare students offered versus not offered seats and use instrumental variables to adjust for actual attendance, yielding a credible causal estimate of charter school effects.
“oversubscribed charter schools pick their students by lottery... that creates an instrumental variables situation where we compare kids who are and are not offered seats at a charter school and then we adjust for the difference in the likelihood of attending”
- Using regression discontinuity around test-score cutoffs for elite exam schools like Boston Latin and Stuyvesant, students just above the threshold show no achievement advantage over those just below, despite having much higher-achieving peers, casting doubt on peer effects as a driver.
“the RD study of elite high schools shows no--in this case, no achievement advantage for kids who go to these more elite schools. In spite of the fact that their peers are much better.”
- The apparent earnings advantage of attending selective private colleges over public ones largely disappears once you condition on where students applied and were admitted, implying the raw difference reflects pre-existing ambition and ability rather than a causal college effect.
“conditional on where people were admitted, they do about equally well. There's no advantage.”
- The Minneapolis domestic violence experiment, where police were randomly encouraged but not forced to arrest suspected batterers, is an instrumental-variables situation because the treatment cannot be fully dictated, making IV the ideal tool for the messy partial-compliance common in social field trials.
“this is an IV situation because you can't actually tell the police what to do... there's deviation from random assignment. It turns out that instrumental variables is the ideal tool to analyze that sort of scenario”
Better research designs, not robustness checks, are the primary driver of empirical economics credibility.
- Isaac Ehrlich's influential capital-punishment regressions were unconvincing not because of inadequate sensitivity analysis but because they lacked a clear research design; a credible study would use a state panel and differences-in-differences instead.
“Ehrlich's problem is not thoroughness. That's what I'm saying. Ehrlich's problem was the lack of a design.”
- The credibility of empirical economics has improved chiefly because research designs got better, not because of more thorough specification sensitivity analysis; most older empirical work was unconvincing because the designs were lousy.
“The reason that most empirical work was not convincing in the age of, say, Stigler and until more recently was not because there was inadequate specification testing, but because the research designs were lousy.”
- Leamer's complaint about arbitrary specification choices filtered into empirical practice as the now-standard expectation that researchers report plausible robustness checks, and increasingly as pre-commitment to analysis plans in randomized trials.
“Filtered into empirical practice in the form of robustness checks. In the sense that researchers today are expected to report plausible variations on what they've done.”
- Friedman and Schwartz's monetary history, essentially a clever natural-experiment before-and-after analysis rather than sophisticated statistics, is among the most influential documents in social science and remains a worthy benchmark for empirical macro on the causes of depression and inflation.
“One of the most influential documents in the history of social science if Friedman and Schwartz. And it's hard to point to another field where, at least in social science, where anything has been so influential.”
Empirical economics succeeds by identifying both effective interventions like human capital and ineffective ones like microfinance.
- Since Card and Krueger, repeated good studies have failed to find large disemployment effects of the minimum wage, shifting the burden of proof onto those who claim such effects, even though the evidence is not as strong as a randomized trial.
“the burden of proof has shifted towards people who think that the minimum wage has large dis-employment effects. Because it's been hard to find those.”
- A major and underappreciated role of social science, especially amid advocates and commercial interests, is to identify interventions that are not likely to work—such as microfinance, computers in early education, or government training programs—thereby redirecting resources toward more promising uses.
“one of the big roles of a social scientist is to point out what's not likely to work.”
- In education research two consensuses are emerging: that a certain type of charter school is extraordinarily effective in urban districts, and that teachers matter for both student achievement and longer-run earnings.
“a certain type of charter school seems to be extraordinarily effective in urban districts. Another is that teachers matter both for achievement and earnings in the longer run.”
- The solving of high inflation as a central macroeconomic problem, including in developing countries, before the Great Recession, is a genuine achievement of applied macroeconomics and one of the few cases where empirical economics demonstrably improved policy understanding.
“inflation was a central, was the central macroeconomic problem. And that problem in developing countries seems to have been solved... I see that as a feather in the cap of applied macro.”
- On the micro side, the major empirical victory is establishing the general importance of human capital as a causal determinant of earnings, built on Becker's theory and Mincer's empirical work, with schooling shown to boost earnings while many government training programs do not help much.
“on the micro side I would say the general importance of human capital as a causal determinant of earnings.”
Credible empirical work must meet a standard whereby contenders with the evidence cannot ignore it, distinguishing strong from forgettable studies.
- Pre-committing to an analysis plan produces especially convincing findings for randomized trials, but it is difficult for observational research designs where the researcher must see the data before knowing which analytical approach will work.
“the idea of precommitment becomes very difficult in some of the research designs that I use where you really need to see the data before you can decide how to analyze them.”
- The proper standard for empirical work is not whether everyone is convinced but whether a study's design and findings are credible enough that anyone interested in the evidence must contend with them, as opposed to forgettable studies like Ehrlich's or Stigler's empirical work.
“There are studies that are remembered for their findings. You may disagree with the findings... But the findings are worth discussing and thinking about and they have to be confronted. Okay, that's my standard.”
- Regression is not the best method but the natural starting point; any credible empirical paper on cause and effect should at least show controlled regression estimates as a first word, even if not the last word.
“That may not be the last word, but it ought to be the first word.”
- Angrist measures the success of the credibility revolution not by whether critics like Leamer and Sims are convinced but by what Ph.D. students learn and young faculty do, evidenced by Mostly Harmless Econometrics selling about 50,000 copies and becoming a reference point in doctoral programs.
“That's the measure of our success, what Ph.D. students are learning and what young faculty are doing. And I think by that standard, we're winning.”
Selection bias—observed group differences absent causal effects—is the fundamental econometric problem empirical design must address.
- Concerns about whether a study's sample size is large enough are answered by the statistics themselves—the standard errors tell you whether results are statistically precise—so the real hard question is not sample size but whether findings generalize.
“the machinery of statistics tells you whether your sample size is large enough. The answer to that question is in the standard errors.”
- Selection bias is an econometric term for observed differences between groups that are not actually causal effects, such as insured people being healthier mainly because healthier people tend to obtain insurance rather than because insurance makes them healthy.
“'selection bias' is an econometric term for differences observed between groups that are not in fact causal effects.”
- Evidence from two well-done randomized trials (RAND and Oregon) overwhelmingly suggests that poor U.S. health relative to other OECD countries has nothing to do with health insurance, though insurance may be justified for other reasons.
“The evidence overwhelmingly suggests that it has nothing to do with health insurance. And we see that in two randomized trials, extremely well done, very convincing.”
Empirical work is conducted by fallible humans whose mistakes should not discourage pragmatic reliance on good research.
- Empirical economics, like any science, is done by fallible humans who make mistakes (e.g., Reinhart and Rogoff), so an idealistic view of it is bound to be disappointed, but one should not let the perfect be the enemy of the good.
“Science is a human endeavor and I'm not sure that we're worse than other fields”