Adam Ozimek on the Power of Econometrics and Data
What this covers
Adam Ozimek of Moody's Analytics and Russ Roberts debate whether economics genuinely accumulates knowledge through rigorous empirical work or merely clothes ideological positions in the language of statistics. Ozimek argues that econometricians do change their minds when confronted with well-executed micro evidence, citing his own shifts on trade adjustment costs and examples like instrumental-variable studies of the Cash for Clunkers program or research by David Autor, David Dorn, and Gordon Hanson on the China trade shock. Roberts counters that belief in most economic positions—especially in macro—rests fundamentally on theoretical assumption and casual evidence rather than proof, and that economists find ways to dismiss inconvenient results as special cases, a problem rooted in confirmation bias and the difficulty of isolating causation in the real world.
The conversation ranges across policy domains where the two positions clash most visibly. They examine the minimum-wage literature, where Ozimek sees cumulative scientific progress across studies using better data and fresher methodologies, while Roberts worries about measurement on the wrong margins—job counts rather than training or entry opportunities. They parse the stimulus multiplier, where both concede that macroeconomics lacks the degrees of freedom that micro studies enjoy, and debate whether the 2009 stimulus can even be evaluated as one thing. On trade more broadly, Ozimek cites the strength of evidence that it benefits the average person, even as newer work shows durable, concentrated pain in exposed labor markets. Roberts invokes the Seen vs Unseen problem—factories that move abroad are visible, but jobs that expand because cheaper prices free up consumer spending are not—to question whether net job effects can ever be truly measured.
Ozimek argues that economics makes genuine, cumulative progress because well-done econometric evidence does change economists' minds, while Roberts counters that economic worldviews are mostly built from casual evidence and personal narrative that sophisticated statistics merely dress up.
- Ozimek and others changed their minds on trade adjustment costs and other issues based on rigorous micro evidence
- Roberts holds that belief in trade rests on theoretical constructs and casual evidence, not scientific proof
- Both agree macro (e.g., the stimulus multiplier) is harder to settle empirically than micro
Empirical evidence progressively eliminates false claims and accumulates knowledge even when studies reach different conclusions.
- The failure of high inflation to appear despite massive monetary easing after the Great Recession, combined with micro studies finding no structural housing-lock effects, persuaded prominent figures like Narayana Kocherlakota and Arthur Laffer—who had predicted imminent inflation on supply-side grounds—to abandon their models and accept that the recession was demand-driven.
“he thought inflation was right around the corner--because demand-side policies can't matter that much. So this must be supply side... And it didn't show up. It just didn't show up. And that fact... he just sort of changed his mind.”
- Even macro research projects that are largely abandoned—such as the 1980s real-business-cycle attempt to explain recessions without money—leave a useful legacy, because the failed literature lets economists point to evidence ruling out claims like 'recessions are caused by productivity shocks,' so knowledge accumulates through theories that get ruled out.
“the idea that you could explain recessions without money was a big research project in the 1980s... it was revealed to not be a very valid way to describe most recessions. And so we don't still build upon that now.”
- The minimum wage literature is not a fundamentally unsolvable problem where opposing studies merely lob evidence back and forth; instead each new study examines slightly different angles and adjustment mechanisms with better data, so the field is making cumulative progress toward a better understanding and stronger consensus.
“Looking at the literature closely, it doesn't look like a draw where two sides just lob evidence back and forth. It looks like progress to me, where we are getting a better understanding of what happens, and we are using better data and better approaches all the time.”
- The macro evidence that economics progresses is that, while it is easy to focus on disagreements, economists agree on and rule out a very large stock of bad policy ideas that non-economists love, showing the field continually moves closer to truth by eliminating bad ideas.
“we have a massive, massive stock of bad policy ideas that are extremely popular but most economists rule out from that evidence... if you think about the things that economists agree about... it's a pretty big stock of policies and ideas about the way that the world works.”
Economists absorb half-remembered evidence into ideology and must apply greater humility to beliefs resting on narrative and accumulated experience.
- The economic thought and content industry is driven too much by persuading one's own side, so economists rarely revisit fundamental questions or dig up the evidence behind their image of markets; truly changing opponents' minds requires reaching across the aisle by surfacing the evidence that motivates one's beliefs rather than arguing with allies on the margins.
“the economic thought industry, the content industry, is driven far too much about persuading our own side. And so we really don't need to go back to those fundamental questions.”
- The broad literature on compensating variation—showing that wages rise to make up for jobs that are less pleasant or more dangerous—is the kind of evidence that sinks into an economist's mind and helps build a picture of markets operating relatively effectively, even though one forgets that such studies are driving the mental image, illustrating how ideologies form from accumulated, half-remembered evidence.
“the research on compensating variation, where, if a job is less pleasant in some regard or more dangerous in some regard, that wages tend to make up for that... it sinks into your mind; it helps create a picture of markets as operating relatively effectively. But you sort of forget that that evidence is driving that image you have of markets.”
- While econometrics deserves skepticism, the lesson must be extended further to ideology, facts, and personal narratives: because empirical studies put their assumptions on the table to be debated, our biases operate even more strongly and undetectably when our beliefs rest on accumulated lifetime experience and untestable stories, so those sources need even more humility than empiricism.
“at least when it comes down to empiricism we have it all there on the table. I can point to the study; you and I can talk about the assumptions behind the study... But if what you have is a story, that's much harder.”
- Economics and economists need more humility about the limits of their knowledge.
“I agree with you that economics needs humility. I think probably economics needs more humility. And economists need more humility.”
Trade imposes larger and more durable adjustment costs than economists previously believed, concentrated and identifiable labor market shocks.
- Careful micro-level econometric work by Autor, Dorn, and Hanson on the China trade shock showed that local labor markets exposed to Chinese import competition adjusted slowly and painfully, with workers dropping down the labor market and moving onto permanent government assistance like disability, implying the adjustment costs of trade are far larger and more durable than economists expected given flexible U.S. labor markets.
“what they found was that adjustments have taken a long time. And they've been much bigger than you might have expected, especially compared to what economists were thinking in the 1990s”
- Despite revised views on adjustment costs, the economic consensus that free trade benefits the average person remains strong, as shown by the University of Chicago survey of economists; the lesson is to worry more about adjustment costs and policy responses, since doing nothing is more costly than thought, which lowers the cost of doing something.
“If you look at the U. of Chicago--they do a survey of economists, and they still agree, resounding consensus, that free trade can benefit the average person in the United States.”
- The labor market effect of trade exposure, especially to low-skilled countries, appears uniquely large compared to other structural shocks like automation or general manufacturing decline, possibly because it was large, geographically concentrated, and a distinct exogenous shock that is more easily identified econometrically.
“I certainly think that the effect to trade is bigger than the effect to manufacturing and work from all tours[?] also suggested that it's bigger than the effect of automation. So there's something about the effect of trade that's been bigger.”
Macroeconomics faces structural obstacles to causal inference that limit what empirical work can establish compared to microeconomics.
- Macroeconomics is definitively harder to prove things in than microeconomics because there is less data, it is harder to isolate partial equilibrium, there are fewer degrees of freedom, and time-series analysis is a weak tool for establishing causation.
“Macro is definitely harder. There is no doubt about that. There is less data. It's harder to isolate partial equilibrium. You have less degrees of freedom, and time series is tougher to prove things with.”
- The wide range of estimated fiscal multipliers before 2008 arose because the topic was a neglected backwater that few studied closely, and because the multiplier genuinely differs between full-employment and recession conditions while time-series data lacks enough degrees of freedom to identify those differences.
“I think Valerie Ramey called it a backwater, until the 2008, until the Recession. I think one of the reasons that you had such a wide range of multipliers is that, 1. People weren't looking at it very closely for a while; and 2. You have different multipliers: you have full employment multipliers and you have recession multipliers.”
- It is a mistake to ask whether the 2009 Stimulus Act as a whole worked, because it was not a discrete policy turned up like a dial but a bundle of a dozen different things—direct household transfers, green infrastructure spending, tax cuts, Cash for Clunkers—so the right approach is to evaluate the individual components rather than seek one aggregate multiplier.
“The Stimulus Act was like a dozen different things. And so to say that research hasn't told us whether the Stimulus Act was good or not or increased jobs--well, I mean, you could write a 100 page paper on just what was in it.”
Absence of evidence against trade benefits, immigration effects, and wage-adjustment mechanisms supports belief in market effectiveness.
- Lack of evidence shows what hasn't been disproved: because there is no detailed micro evidence showing that producers fail to lower prices, that savings aren't passed to consumers, or that consumer variety isn't increasing, the absence of disconfirming evidence is itself support for the consensus that trade is beneficial.
“Evidence doesn't just show you what is true; but lack of evidence shows you what hasn't been disproved.”
- The best micro evidence on immigration's local labor market effects shows that, unlike the China trade shock, immigration has not been costly, has not taken a long time to adjust to, and has not harmed native workers, even though some argue general-equilibrium and national-level effects must also be considered.
“if you look at similar approaches that focus on what are the local labor market effects of immigration, they haven't been costly; they haven't taken a long time to adjust. And native workers haven't been harmed by it.”
- A large share of U.S. aggregate productivity growth throughout the 1990s came from the retail sector via logistics improvements by companies like Wal-Mart, and this body of evidence is an order of magnitude more important than the unresolved job-effect debate when evaluating big-box stores.
“a lot of the growth in productivity in the United States, aggregate productivity throughout the 1990s, came from the retail sector. And it came from logistics, and it came from companies like Wal-Mart who made retail more productive.”
Methodologically sound but inconclusive evidence on big-box stores and minimum wage effects shows empirical data can remain unresolved.
- The empirical literature on whether big-box stores like Wal-Mart destroy jobs, using distance-from-Bentonville as an instrumental variable, is essentially a draw because the instrument was not robust (it implied implausible effects on manufacturing employment), so the question of job losses was not resolved by the data.
“it wasn't a very robust instrument. And so, I'm happy to call that, I think, fairly, empirically a draw. Or not very persuasive in either direction.”
- Although recent cross-border natural-experiment studies showing no employment effect of the minimum wage are creative and persuasive and have moved economists like Krugman, newer evidence tracking individuals over time, focusing on employment growth rates rather than levels, and examining prices and population shifts suggests a stronger negative effect, so within 5-10 years the field's consensus will move back toward the minimum wage causing job loss.
“I do think that if we fast forward 5 years down the road that the field is going to be moving back towards the minimum wage having a negative effect. And that's because of this new persuasive evidence that continues to flow.”
- Recent, methodologically accessible evidence—looking at many countries over a long period with better pre- and post-trend measures—suggests recessions tend to have long-run impacts, an idea previously unpopular among economists, which persuaded Mark Zandi to change his view.
“Mark Zandi is the chief economist at Moody's Analytics has been persuaded by recent evidence that shows that recessions might tend to have long-run impacts. And that's something that was, you know, not a very popular idea among economists”
- Good micro evidence using an instrumental-variables approach showed that the Cash for Clunkers program did not generate lasting new demand but merely pulled new-car demand forward in time from the next one to four quarters, producing a temporary burst, so it is widely regarded as unsuccessful and unlikely to be repeated.
“what happened was that the demand for new cars was shifted forward in time from a very short period. So you are looking at 1, or 2, or 3, or 4 quarters in the future--all that demand is where it came from. And it was a very temporary burst.”
- Other sciences are not much better off than economics: David Cutler's work shows large geographic variation in health spending unrelated to better outcomes, with a third of excess end-of-life cost driven by physicians not following best-practice guidelines, and an NBER study found that when a prominent life-sciences researcher dies, new entrants and fresh ideas improve the subfield—confirming that science advances one funeral at a time despite having real experiments.
“Cutler's work suggests that if physicians followed professional guidelines, in end of life care, the end of life expenditures would be something like a third less.”