What this covers

Ed Leamer and Russ Roberts discuss the structural problems that prevent econometrics from functioning as genuine science. The conversation centers on Leamer's long-standing critique—originally published in 1983—that economists routinely hide the sensitivity of their findings to modeling choices, allowing different specifications of the same dataset to produce wildly different conclusions. Leamer contends that natural experiments, despite recent enthusiasm in the field, do not cure this problem because local causal estimates from exotic events (such as the Mariel boatlift's effect on Miami immigration) cannot credibly generalize to different social contexts without the same hard theoretical work that observational data demands. The two examine why this situation persists: journals publish only positive results, the profession rewards contrarian and shocking claims, and economists face no real incentive to report honestly about which findings are robust and which are fragile.

The discussion ranges across concrete examples—housing's central but unmentioned role in business cycles, the overextension of the Tiger Woods study into dubious business advice, the logical inconsistency of unemployment insurance policy, and the impossibility of randomized experiments in macroeconomics. Leamer distinguishes between sturdy inference (when reasonable alternative models reach the same conclusion) and fragile inference (when small changes in specification reverse findings), and argues that econometric debate functions more like courtroom advocacy than science, in which readers see only the plaintiff's case. Roberts adds that the profession's transformation over the past 25–40 years—with far more economists gaining fame and fortune through media, books, and the internet than existed in the Friedman and Samuelson era—has made economics more entertaining but not necessarily more truthful. Leamer closes by resisting the idea that any single methodological path will make economics empirically scientific, arguing instead that the field needs tools to distinguish robust from fragile results and a culture willing to report that distinction plainly.

Sharpest takeaway

Leamer argues that econometrics remains fundamentally non-scientific because results are highly sensitive to unreported modeling assumptions, and that modern 'natural experiment' techniques don't solve this because laboratory or accidental findings rarely transfer credibly to broader real-world settings.

  • Different model specifications yield dramatically different conclusions, and economists hide the 'fragile' versus 'sturdy' distinction from readers
  • Natural/randomized experiments produce valid local causal estimates but cannot be generalized across different social contexts
  • There is no incentive for honesty about sensitivity, so econometrics functions more like journalism marshalling facts than like science

The claims · ranked41 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.80

Much published empirical work is taken on faith because it appears in peer-reviewed journals and is assumed to be science, yet many relationships fail to hold up with a different database or ten years later, owing to confirmation bias and the fact that journals generally publish only positive results.

causalhigh valueestablishednovelty 2/4durability 4/4· Russ Roberts

“a lot of them don't hold up when it's a different database a lot of them don't hold up... when it happens ten years later... there's a big confirmation bias problem there's a problem that journals only published generally only publish positive results”

0.79

Housing is central to the business cycle: it is a great leading indicator and contributes a large fraction of every postwar downturn (e.g., construction jobs), yet not a single macroeconomics textbook mentions housing—an omission Leamer attributes to the field's theoretical straitjackets.

factualhigh valuecontestednovelty 4/4durability 3/4· Ed Leamer

“housing is absolutely critical and first of all it's a great leading indicator but it also... contributes a large fraction of the problem... there's not a single textbook that mentions housing in macroeconomics”

0.78

Even a valid Mariel boatlift finding about immigration's impact on Miami tells us very little about the impact of, say, a 2,000-mile fence on the US southern border, because the circumstances are so different; generalizing requires the same hard work of thinking about confounding circumstances as observational data does.

causalhigh valuecontestednovelty 3/4durability 4/4· Ed Leamer

“the problem with the Mariel boatlift is was it tell us about the impact of a fence 2,000 miles fence along our southern border... it doesn't tells us very much because it's so different”

0.75

The economics profession has been radically changed by opportunities for fame and fortune that didn't exist 25-40 years ago: where once only Friedman and Samuelson were widely known (each with a Newsweek column), now a far larger group of economists gains money or fame via the internet, public speaking, magazines, columns, and popular books like Freakonomics; this has made economics more entertaining and widely consumed but not necessarily more truthful.

factualhigh valueestablishednovelty 2/4durability 3/4· Russ Roberts

“30 40 years ago there were two famous economists... Milton Friedman and Paul Samuelson... today an enormously larger group of economists make either money directly or indirectly get some fame from the internet”

0.74

Sturdy inference is achieved only when an adequate range of reasonable alternative models all lead to about the same conclusion; if small, easily-entertained changes in the model lead to dramatically different conclusions, the estimate is fragile and not to be believed.

definitionhigh valuecontestednovelty 3/4durability 4/4· Ed Leamer

“confirmed that all the reasonable miles lead to about the same conclusion in which case you get to sturdy inference or what seem like small changes in the models... lead to dramatically different conclusions at the fragile estimate and not to be believed”

0.73

The 'con' in econometrics is that depending on which models a researcher selects to analyze a dataset, they can get dramatically different estimates and conclusions, and economists fail to alert their audience to this sensitivity, thereby pretending datasets provide clearer information than they possibly can.

causalhigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“the con is that depending upon what models you select happen to use for analyzing a data set you can get dramatically different estimates and dramatically different conclusions an economist had not spent enough effort alerting their customers sensitivity”

0.73

Econometric debate functions like a court of law in which you only see the plaintiff's argument but know the defendant's is out there, so until you see the opposing model fully worked out you withhold judgment; combined with little incentive to get it right scientifically, this produces reconfirmation of prior beliefs rather than consensus.

causalhigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“It's like a court of law in which you see the plaintiffs argument but you're not allowed to see the defendants but you know that it's out there until you see the defendant... you're not going to make a judgment on it”

0.73

Published empirical work hides the 'kitchen' process: researchers who run a single specification usually get nothing significant, then convince themselves they had the wrong specification and iteratively add or drop variables and nonlinear terms until they sculpt a publishable result, so readers cannot judge whether the finding is robust.

causalhigh valuecontestednovelty 3/4durability 3/4· Russ Roberts

“you convince yourself rather easily that you must have had the wrong specification you left out a variable or you included one you should have included until eventually you craft sculpt a piece of work that is a conclusion and you publish that”

0.73

To genuinely establish whether government spending has a multiplier you would need a randomized experiment with a treated economy and a control economy, which is essentially impossible in macro; the closest natural experiment is wartime defense spending, but the World War II defense buildup tells us little about the effect of an Obama-style stimulus package because the situations differ so greatly.

causalhigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“in a case of macro it's very very difficult to think of what is the randomized experiment whether it's purposeful or natural that we can use to make conclusions about the impact of federal... programs”

0.73

Statistical significance (a high T-statistic) only indicates that a relationship is measurable and unlikely due to chance; it says nothing about whether the effect is economically important in magnitude, and conflating 'significant' with 'important' is a key error economists make.

definitionhigh valueestablishednovelty 2/4durability 4/4· Ed Leamer

“significant really means important and it's not the same yeah so I highly recommend that we use the word measurable”

0.72

The Tiger Woods study (finding opponents score about 0.8 strokes worse when he competes) was overextended into the absurd business implication that firms shouldn't hire superstars because a top performer demoralizes others; this ignores that 'top 20%' in a large firm is many people, not just the single superstar, and ignores that incumbent golfers actually worked harder and improved rather than giving up after Tiger Woods arrived.

causalhigh valuecontestednovelty 3/4durability 2/4· Russ Roberts

“his opponents... score higher by eight tenths of a stroke... and yet here it was in the New York Times... the top 20% doesn't just include the best single best person the superstar like Tiger Woods it that's an absurd conclusion”

0.71

There is a huge gap between a conceptual economic theory (which is by nature a simple version of reality, e.g. 'demand curves slope downward') and the explicit complete model a data analyst must build, requiring choices of functional form, lag structure, expectations, and control variables—choices about which economic theory offers no guidance and which are 'relieved' only by distributional assumptions economists have no opinion about.

factualhigh valuecontestednovelty 3/4durability 4/4· Ed Leamer

“there's a huge step between our conceptualization of the problem and building a model that can capture that framework”

0.71

Purposefully randomized experiments yield proper causal conclusions in a laboratory setting, but there is no assurance the laboratory effect translates to the real world—especially in economics, because it is a social system in which people watch each other and form expectations, making transference very difficult.

causalhigh valuecontestednovelty 3/4durability 4/4· Ed Leamer

“you've created that in a laboratory and there's no assurance that that same laboratory effect is going to translate into the real world particularly in economics because we're talking about a social system”

0.71

Instrumental variables / natural experiments are 'accidental experiments' that exploit exotic events (like the Mariel boatlift) presumed uncorrelated with the outcome being studied, but every such instrument opens a conversation about whether it is truly a legitimate randomized treatment or is in fact correlated with the response.

definitionhigh valuecontestednovelty 3/4durability 4/4· Ed Leamer

“the instrumental variables is the reference to accidental experiments in which you scurry around trying to find out something that is is if you had designed experiment”

0.68

Extending unemployment insurance is internally inconsistent with the negative-feedback rationale: paying people more while making it cheaper to be unemployed reduces job-search intensity (since demand curves slope downward), so measured unemployment can stay high longer—even though it is not the only reason unemployment responds slowly to recovery.

causalhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“they kind of forgot about the fact that it encouraged people not to look as hard as they otherwise would so measured unemployment could stay high for a long time”

0.68

There is little incentive for economists to admit humility; if anything the incentive runs the opposite way, because getting into the Wall Street Journal or New York Times requires overconfidence, not humility—and the rise of internet, blogs, and public economics over the last 25-30 years has amplified opportunities for fame and fortune that didn't previously exist.

causalhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“if you want to get in the Wall Street Journal the New York Times you got to be overconfident... humility doesn't help you get there”

0.68

Because more exotic and dramatic findings are more likely to be featured in outlets like the New York Times, and universities reward that publicity, there is a real professional bias toward shocking, contrarian, and bizarre empirical claims.

causalhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“the more exotic and dramatic you're finding the more likely it is you'll be featured in the New York Times and as a result the university likes that so there's a real bias toward shocking claims contrarian claims bizarre claims”

0.68

Economists relinquish their most important decision—whether a variable's effect is economically important—by turning the judgment over to statisticians using context-free significance thresholds that have nothing to do with the specific setting.

causalhigh valuecontestednovelty 2/4durability 3/4· Ed Leamer

“we turn it over to statisticians who decide what significant or not based on these key hires that really don't have anything to do with the setting”

0.68

There is no professional incentive for economists to honestly report the fragility of their results, which is the sociological reason the con persists.

causalhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“being an economist my first thought as to why that is is because there's no incentive unfortunately for it to be otherwise”

0.68

Macroeconomic multiplier arguments implicitly assume that the same structural relationships estimated in past episodes still hold today, even when the cause of the recession and the nature of the spending are totally different—a presumption Roberts finds very strange.

causalhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“implicit in those multiplier arguments is the presumption that it doesn't matter which I find very strange”

0.68

No single methodological path will make economics fully enlightened; there are many ways to create knowledge—doing theory, studying datasets in various ways—each deserving respect, and believing that designing experiments will suddenly turn economics into an empirical scientific discipline is not likely.

normativehigh valuecontestednovelty 2/4durability 3/4· Ed Leamer

“I just don't think any single path is going to get us all enlightened we need to recognize that there are many ways that we can create knowledge insight and wisdom”

0.68

The creators of econometric techniques are typically less enthusiastic and more cautious than their followers; the followers and followers-of-followers tend to be 'drinking the kool-aid' and forget the originators' admonishments.

factualhigh valuecontestednovelty 2/4durability 3/4· Russ Roberts

“the creators of techniques are less enthusiastic than their followers their followers and the followers of their followers tend to be drinking the kool-aid and have forgotten all the admonishments of the creators”

0.68

Leamer's counter-view: although demand curves generally slope downward and large long-run unemployment benefits would increase unemployment, within the context of a business cycle people categorize themselves as working or not working, and that self-categorization is not much affected by where they lost their job or the benefits received—so short-run effects can differ from long-run effects.

causalhigh valuecontestednovelty 2/4durability 3/4· Ed Leamer

“in a context of a cycle people tend to think of themselves as either working or not working and that personal categorization is not impacted much by where they lost a job or the benefits that they might be receiving”

0.66

Empirical work in economics—no matter how careful or clever, whether on Keynesian stimulus, the minimum wage, or school quality—does not change the mind of anyone who isn't already a believer, which means it is not really science.

factualhigh valuecontestednovelty 3/4durability 3/4· Russ Roberts

“the empirical work no matter how careful no matter how clever doesn't seem to change anybody's mind who isn't already a believer and to me that means it's not science”

0.66

Economists need tools that separate sturdy inferences (those that don't depend much on ambiguous assumptions) from fragile inferences (those that change with a very slight change in the model), plus a culture receptive to reporting that distinction.

normativehigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“we need first of all tools that will help us sort the sturdy from the fragile conclusions and then secondly we need a method of communicating that in articles”

0.66

Brown's study answers a different question than the policy-relevant one: it compares how players score in tournaments Tiger enters versus skips within a given year, not whether players improved their underlying ability after Tiger entered the sport; the latter (before-and-after comparison) is the question that would speak to general claims about competition.

factualhigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“her analysis is addressing a different question than the one that you just described so hers are within a given year how do these players play in tournaments that Tiger is in versus the one that it's not”

0.66

Macroeconomics is far too heavy toward theory and has ignored an enormous database that could improve understanding; instead of estimating econometric models, the field imposes a particular structure—a 'straitjacket' (overlapping-generations, rational-expectations, or simple Keynesian model)—that prevents learning how the complex economy actually evolves.

causalhigh valuecontestednovelty 3/4durability 3/4· Ed Leamer

“the profession is way too heavy toward theory and... macro I think has completely ignored an enormous database... they've imposed a particular structure on the data a straitjacket which has prevented them from learning”

0.65

The remedy economics needs is humility: being satisfied with the real but limited progress the field makes, and not imagining it will ever reach the scientific certitude of engineering, physics, or chemistry—and sometimes the right answer is simply to say 'I don't know' rather than 'you've got to do something.'

normativehigh valuecontestednovelty 2/4durability 4/4· Ed Leamer

“humility is what I think we need to have as a profession and be satisfied and happy with the real progress that we make but not to imagine that we're ever going to get through a level scientific certitude”

0.63

Over the decades since 1983 the profession has advanced economic theory and complex econometrics but has made little progress building tools to identify sensitivity of conclusions to assumptions, so the standard operating procedure remains essentially 'cooking the books.'

factualhigh valuecontestednovelty 2/4durability 2/4· Ed Leamer

“we have not made a lot of progress in building tools for identifying sensitivity of our conclusions to the assumptions... we still are pretty much in the same page... same operating procedure that we were thirty years ago which is to is to cook the books”

0.63

Macroeconomists are largely not engaged in soul-searching after the financial crisis; they continue in their cocoons treating the crisis and policy in its midst as someone else's problem that doesn't affect their conceptual frameworks.

factualhigh valuecontestednovelty 2/4durability 2/4· Ed Leamer

“we have a way to continue to live in our own little cocoons and think of the financial crisis and issue of how to devise policy in the midst of all that is somebody else's problem”

0.62

Leamer's 'Housing IS the Business Cycle' paper, written before the 2008 crisis, was prophetic in identifying housing's centrality, yet the macroeconomics profession largely greeted it with benign neglect or dismissal ('we knew that already'), even though virtually every macroeconomist concedes housing precipitated the downturn—illustrating that even prescient data-driven work fails to penetrate theory-committed economists.

factualhigh valuecontestednovelty 3/4durability 2/4· Russ Roberts

“this guy was a real prophet he was... at the very beginning of this problem... you'd think you'd get a little more attention for that paper”

0.59

Economic theory is fiction—sometimes good, insightful fiction—a fictional representation of the world by its very nature, and econometric analysis is really journalism: marshalling facts persuasively rather than doing science.

definitionhigh valuespeaker onlynovelty 4/4durability 4/4· Ed Leamer

“economic theory is fiction sometimes good fiction... it's a fictional representation of the world... econometric analysis is really journalism and the journalists job is to marshal the facts as effectively as possible”

0.58

About a quarter of those unemployed since the start of the recession in December 2007 were in construction, and following a housing-market collapse these workers face unusual uncertainty—whether to stay in construction or switch fields—that differs from more general past downturns, illustrating how context-specific the current unemployment is.

factualhigh valueestablishednovelty 2/4durability 1/4· Ed Leamer

“about a quarter of the people who were unemployed since the peak of the since the start of the recession in december of seven... are in construction and they are going to have a hard time trying to figure out whether they should stay in construction”

0.57

Leamer's model of the economy: in a healthy economy someone losing their job does not precipitate further job loss, but in an unhealthy economy a negative feedback loop emerges where job loss generates further job loss; government demand management is justified only during those few episodes when that negative feedback loop is evident, and in a self-healing phase the government's job should be to leave decision-making to the private sector and reduce the uncertainty it creates.

causalhigh valuespeaker onlynovelty 3/4durability 3/4· Ed Leamer

“in a healthy economy is someone who loses their job it doesn't precipitate more job loss but when the economy becomes unhealthy then you get a negative feedback loop... the government needs to help prevent that negative feedback loop... but only during those few episodes”

0.57

When asked to name a sophisticated econometric study that caused the profession to reach consensus on a previously controversial issue, most people struggle to name one, and some economists name only their own work—suggesting such consensus-forming studies essentially do not exist.

factualhigh valuespeaker onlynovelty 3/4durability 3/4· Ed Leamer

“can you name an econometrics study that caused the profession to come to a consensus about something that was controversial and most people struggle... some economists named their own work”

0.57

Economics, especially macro, yields not scientific truth but opinions—and the goal is to produce persuasive, well-thought-out opinions rather than dismiss everything; combining stories (economic theories) with pattern-seeking in data, working together, helps form better opinions even though it isn't science.

normativehigh valuespeaker onlynovelty 3/4durability 3/4· Ed Leamer

“this isn't scientific truth but all we have in the area especially a macro is opinions and they're either persuasive well-thought-out opinions or not well thought out”

0.52

The presence of a dominant competitor like Tiger Woods can plausibly lower a player's score through multiple distinct mechanisms—lowering effort/giving up, being lulled into attempting impossible shots that hurt the score, taking more chances (higher variance), or becoming overly cautious to secure lower-place prize money—so a single average effect conceals many heterogeneous behavioral responses.

causalhigh valuespeaker onlynovelty 2/4durability 3/4· Ed Leamer

“I could easily be lulled into thinking I can make those same shots and giving it a try and end up harming my score... there's a hundred things that the way people... go through a all up and down a set of emotions and strategies”

0.52

Even when the final econometric model is silly and the final table of T-statistics is irrelevant, the process of analyzing a complex dataset (like doing theory) helps economists think more clearly and form judgments, and the social conversations help the profession collectively make progress—often not to the right conclusions, but with some scope for progress.

normativehigh valuespeaker onlynovelty 3/4durability 3/4· Ed Leamer

“the process helps us think better as an economist analyzing datasets complex and vigorous settings... the process of analyzing the data set helps us think clearly”

0.49

This is the Golden Age of economic education because, unlike the past when only a handful of books could be recommended, there are now dozens of accessible books and many thoughtful, non-sensational blogs—available at zero monetary cost—where authors interact with data carefully, stating what we know and don't know without extravagant claims.

normativehigh valuespeaker onlynovelty 2/4durability 2/4· Russ Roberts

“this is the Golden Age of economic education... now there's 30 books you can recommend... there's 10 blogs that are full of interesting information... many of which are actually people trying to figure out how the world works”

0.34

A generation of economists was trained essentially as theorists with very little contact with the real world, so the recent increase in real-world engagement (even if imperfect) is on balance not such a bad thing.

factualcontestednovelty 1/4durability 2/4· Ed Leamer

“we went through a generation of economists or basically theorists who were allowed to have very little contact with the real world so a little more isn't such a bad thing”

0.12

A T-statistic is a measure of how likely or unlikely it is that a relationship found in data is due to chance, with a high T-statistic meaning the relationship is very likely real rather than a fluke.

definition· Russ Roberts

“the T statistic on a variable in a statistical study is a measure of how likely or unlikely is it that this relationship that we found in the data is due to chance”