How to Avoid Lying With Statistics (with Jeremy Weber)
What this covers
Jeremy Weber of the University of Pittsburgh joins Russ Roberts to examine the gap between how statistics is taught in universities and how it must be used in policy and business. Weber argues that academic training emphasizes technical mechanics—running regressions, calculating standard errors—and aims to satisfy journal reviewers, but leaves practitioners unprepared for the actual work of deploying numbers honestly in front of city councils, nonprofits, and elected officials. The conversation moves through concrete failures: analysts who run data through endless subsets until a story emerges, organizations drowning in measurement without understanding what their calculations mean, and statistics wielded as a kind of undeserved authority—a shaman's tool that intimidates rather than clarifies. The remedy Weber proposes is not more technical training but deliberate slowness: think before touching data, learn something from it, then step back to reflect rather than immediately reporting.
The episode ranges across context-dependence in social science—why a minimum wage policy or fracking development produces different effects across different places and moments—and the chronic overemphasis on statistical significance at the expense of magnitude of causation. Weber illustrates with lived examples: a basement flood blamed on climate change when a drain clog was the primary culprit; uranium import policy swayed by simple graphs showing historical price-production gaps rather than sophisticated modeling; oil production gains attributed equally to regulation and price, when prices drove roughly 95 percent of the change. A recurring theme is honest counsel: analysts should function like lawyers with clients, surfacing weaknesses and limitations rather than cherry-picking pleasing numbers. The conversation also touches data dredging, confirmation bias, and the risk that poor statistics education leaves students "inoculated" against genuine understanding, mistaking the textbook for the truth.
Weber argues that statistical education teaches the mechanics of analysis but not the thoughtful judgment needed to use statistics honestly and usefully in real-world policy settings, and that closing this 'textbook vs. cookbook' gap requires emphasizing context, magnitude, audience, and intellectual honesty over technical sophistication.
- Statistics is taught by academics who train students to speak to a journal-reviewer 'tribe' rather than the policy/business audiences students will actually serve
- Context-dependence means statistical findings often cannot be generalized across settings, unlike physical laws
- Statistical significance is overemphasized while magnitude and prioritization of causes among many factors are neglected
Identifying and clearly communicating the true magnitude and primary causes of effects is more important than detecting whether any relationship exists.
- In the uranium import case, two simple descriptive graphs showing that historically high uranium prices had not induced the domestic sector to produce anywhere near the proposed six million pounds proved more persuasive than sophisticated analysis, convincing officials that the Commerce Department's $55/pound assumption was unrealistic and that restrictions would sharply raise electricity prices.
“I brought in two figures--two graphs--that simply showed, look, in recent history, the price of uranium has been way above your price for several years and the domestic sector didn't come anywhere close to producing what you're saying they would produce now at a much lower price.”
- The rising U.S. oil production in 2018-2019 was primarily (about 95%) driven by higher energy prices to which producers responded, not by deregulatory efforts; presenting both factors together in political speech as equivalent contributors misleads audiences about their relative magnitudes, even if deregulation was causally related in a minor way.
“The rising oil production in 2018 and 2019 was primarily driven by higher energy prices. Did deregulatory efforts help? It certainly didn't hurt... 95% of it was just producers who are responding to price.”
- Weber's basement flooded not because storms had become more severe (the climate-change explanation his neighbor offered) but because a clog in his drain was the primary causal factor; even completely solving climate change would have left the drain overwhelmed, illustrating how an intuitively plausible but tertiary cause can distract from the real one.
“we could have solved climate change completely, and my drain would still have been overwhelmed at the next storm. It was a secondary/tertiary issue. The primary issue--the primary causal factor--was the clog in the drain.”
- Failing to understand what a calculated number actually means--rather than miscalculating it--creates two problems: it confuses an audience the statistics were meant to clarify, and when discrepancies can't be reconciled (e.g., 25 reported deer-vehicle incidents vs. 600 collected carcasses), the analyst suffers a credibility collapse where nothing they say is believed.
“the error, again, isn't--it's not that you calculated the total wrong. It's not that you manipulated the data. You just weren't thinking much about the data and the number you actually calculated.”
- Statistics culture overemphasizes statistical significance--asking only whether any causal relationship exists (whether the coefficient is zero)--and uses it as a crutch for judging importance, when for policy purposes the priority should be distinguishing large effects from small ones, since many factors are causally related to an outcome but only the large-magnitude ones merit policy attention.
“you really should be thinking of correlation and the magnitude of causation... almost always we care about the magnitude of the impact, not just that they're correlated.”
Statistical education teaches calculation mechanics but never teaches how to think about what numbers mean for real-world purposes and audiences.
- The abundance of data and instant computation creates a 'Shallows'-like effect in statistics: employers report graduates can manipulate and calculate data well but cannot tell what their calculations mean for the organization's purpose--so the remedy is to slow down, think before touching data, and reflect afterward to build genuine understanding rather than just reporting measurements.
“it's like: Data, data everywhere, but understanding is nowhere. And, that's very easy to do when at a click, you've got oodles of data, oodles of regressions you can run.”
- Statistics education teaches the mechanics of tools (how to run regressions, calculate standard errors) like a cooking or vocational class, but never teaches students how to use these tools thoughtfully--how to integrate context, purpose, and audience--so practitioners learn that only by luck in the real world.
“our statistics education, I would say teaches--it shows you: Here is the saw. And: Here are the parts of the saw... Actually going out and cutting down trees, like, do that--we don't do that.”
- Thoughtful statistical use isn't taught because statistics instructors are academics whose own skills are honed for satisfying journal editors and reviewers; they teach students to 'speak to that tribe' (refining standard errors and identification), while most students will instead serve businesses, nonprofits, or city councils--a different tribe--causing systematic miscommunication and neglect of basic questions like whether the data are good and whether an effect's magnitude matters.
“our statistics education, I would say teaches--it shows you: Here is the saw. And: Here are the parts of the saw... Actually going out and cutting down trees, like, do that--we don't do that.”
- The book aims to address what neither introductory nor advanced statistics courses do well: communicating statistics to non-academic audiences and integrating context, purpose, and significance of the moment into what is presented.
“the communication of statistics to a non-academic audience, the integration of context and purpose of the moment or of the organization or of the audience into what you're presenting--its significance for the situation at hand--we tend to not do that well”
Analysts must think deliberately about purpose and context before analyzing data, then reflect after to build genuine understanding rather than just report calculations.
- Effective communication of statistics requires tailoring which numbers to present to what the audience understands: Casey Mulligan advised using a GDP/output figure rather than a welfare figure because the audience grasps output and production but not opportunity cost, and presenting both would confuse rather than clarify.
“he's like, 'Let's just stick with the GDP number. The people we're talking to, they get output, they get production. They don't understand opportunity costs. Let's not distract them with this welfare number that they're not going to know how to reconcile the two numbers.'”
- One should never run a regression without a clear purpose, because meandering through data invites being led to manufactured stories: a researcher gets a hunch, subsets the data until the story emerges, and ends up selling a coherent but fabricated narrative--a danger amplified by modern abundant data and instant computation.
“never run a regression without a clear purpose for doing so. It is so easy to be led in strange places just by kind of meandering through the data.”
- Before touching data, one should stop and think clearly about the problem and purpose; then learn something from the data; then step away for a few days to reflect on what was calculated--this deliberate process produces deeper meaning and understanding rather than just reporting a pile of measurements.
“'Before you touch the data, stop. Think.'... It's just: Slow down and think about what you're trying to accomplish. What is the problem? And then, with that clarity of understanding, you go and you learn something from the data. Then step back. Get away from the data for a minute and reflect for a few days.”
Poor statistical understanding inoculates people against truth-seeking, while rigorous fact-checking and straight-shooting about evidence weaknesses preserves credibility and constrains specious claims.
- Teaching statistics poorly--as may happen in mandatory high school courses taught by people who don't grasp the complexity of randomness, data, and causation--can be worse than not teaching it, because students go out thinking they know it and are 'inoculated' against the real thing, like confusing a prosperity gospel with the true gospel.
“anything taught poorly can be problematic because it's kind of worse than not having been taught it, because you go out thinking you know it, and so you are in a sense inoculated to the real thing.”
- A policy analyst should function like a lawyer in private with a client--shooting straight about the weaknesses and limitations of evidence rather than cherry-picking--because the client (e.g., the CEA Chairman) will face thoughtful, aggressive opponents who find holes, so a 'straight shooter' who privately surfaces weaknesses serves the client far better than a 'data lackey' who just supplies pleasing numbers.
“The Chairman of the Council of Economic Advisers would not be well-served by me being a data lackey... Because Kevin is going to turn around and go to a meeting with thoughtful, sometimes aggressive people who are going to find holes in it and potentially make Kevin look very bad. So, Kevin actually wants somebody who is a straight shooter and not a data lackey.”
- We cannot help making statistical claims about what is common, general, or causal--politicians and nonprofit leaders will make them regardless--so it is better to tether such claims to actual observation; a culture of good statistical use acts as a constraining power on specious claims, which is the answer to the 'lies, damned lies, and statistics' skeptics.
“Statistical claims are with us always. We cannot help but make statements--claims--about what is common, what is general, what is causing X versus Y. We will make those statements. It's better if we tether them to actual observation.”
- The CEA maintained a rigorous fact-checking process--every memo's factual claims and calculations verified step-by-step by an uninvolved checker before reaching the Chairman--because the agency's currency in the White House was its credibility; without that 'straight shooter' reputation, no one would read its work, and the high cost of public errors (versus low cost in academia) justified going slow.
“I think it made a lot of sense, because CEA's currency in the building was its credibility... if we were sloppy with that and word got out and we couldn't really trust them, nobody would read all the stuff that we produced anyway.”
- Even when statistics are used primarily as political weaponry rather than for truth-seeking, if accurate and well-grounded measurements are available to be 'picked up' by competing parties, the resulting back-and-forth tends to avoid major errors and produce better policy outcomes.
“if there are, in a way, better weapons out there that can be picked up, I think you have a greater tendency to some major problems being avoided or opportunities pursued.”
Social science causal relationships are conditioned by historical moment, place, and policy environment, making context-dependent effects fundamentally different across settings.
- Being 'right' in policy analysis does not mean perfectly predicting the future or getting numbers exactly correct; it means the work is defensible to a reasonable, statistically savvy person given the information available at the time, even if future data later reveals the estimates were somewhat off.
“Is, as I say in the book, right. Meaning: it's defensible to reasonable people. It doesn't mean it's perfectly predicting the future... given the information at the time, what was done is defensible to a reasonable person--a statistically savvy journalist.”
- Fracking raised property values in Texas because natural gas wells are taxed as property, so drilling expanded the tax base, allowed lower property tax rates and more school revenue, raising area home values--whereas in Pennsylvania, which does not tax wells, fracking lowered home values especially for groundwater-dependent homes.
“in Texas you tax natural gas wells as property. So, when you drill a well, the full value of that well enters the tax base... they lowered the property tax rate and so people's tax bills were going down, the school was getting more revenue, and property values generally went up in the area.”
- Unlike physics, where gravity is essentially context-independent, social science relationships (like minimum wage effects) are conditioned by historical moment, place, and policy environment, so the same phenomenon can produce fundamentally different effects in different settings--making field-wide consensus on causal magnitudes hard to achieve.
“as long as we're in the earth realm, gravity is pretty contextless. It's not context-dependent. Social settings are so varied, and so the situations in which we estimate these relationships are oftentimes conditioned by the moment in history, the place.”