What this covers

Susan Athey speaks with Russ Roberts about where machine learning and econometrics meet, and where they diverge. The conversation treats machine learning primarily as a tool for prediction and data-driven model selection rather than as a solution to causal inference. Athey argues that the field's future lies in delegating flexible control-variable modeling to machines while keeping human judgment focused on identifying causal effects—and being transparent about which variables were chosen after looking at the data. The discussion covers applications ranging from minimum-wage policy to drug trials to the experimental practices of tech firms like Google and Amazon.

The core technical tension concerns the multiple comparisons problem: when you search through many candidate models to find one that fits your data well, standard statistical measures like p-values and confidence intervals become unreliable. Athey explains how sample splitting—training a model on one half of the data and validating it on the other—can restore honest inference even after aggressive data-driven search. A parallel concern appears in how randomized controlled trials at tech firms, though causally rigorous, systematically bias toward short-term improvements, since running long experiments is expensive; firms may measure 300% predicted revenue growth across thousands of experiments while realizing only 10% actual growth.

The conversation identifies a deeper boundary: machine learning does not solve the fundamental problem of causal inference. Distinguishing treatment variables from covariates and setting up valid counterfactual comparisons requires assumptions about the world that no amount of computing power can verify. Athey illustrates this through policy examples—why comparing high- and low-minimum-wage cities is misleading, how to construct valid control groups, and why extrapolating from college-educated workers' wages to the effects of expanding college attendance requires care about equilibrium effects.

Sharpest takeaway

Athey argues that machine learning's power lies in data-driven model selection for prediction, but it does not solve the fundamental problem of causal inference; the future of applied econometrics is to delegate flexible control-variable modeling to machines while being honest about model selection and focusing human effort on identifying causal effects.

  • Causal inference requires distinguishing the treatment variable from control covariates and treating them differently statistically
  • Sample splitting and averaging protect against the overfitting / p-value hacking that data mining introduces
  • A/B tests at tech firms achieve near-perfect causal identification but systematically bias toward short-term effects

The argument · threads8 threads · 58 claims
0.85

Machine learning optimizes prediction but sacrifices robustness and generalizability; econometrics prioritizes interpretable models across settings.

12 pointscentrality 5/5
  • Econometrics has always prioritized interpretable, reliable models that hold up across different settings and reflect how the world works, which implicitly sacrificed predictive power; machine learning techniques are much better at fitting any particular dataset but worse at producing models robust to changing environments or free of spurious relationships.

    “that push in econometrics has caused us to sacrifice predictive power. And we never really fully articulated the sacrifice we were making”

  • Machine learning does not solve the fundamental problems of correlation versus causation, because the intuitions and theorems about identification already assume infinite data; if an effect is 'not identified'—you cannot determine the minimum wage's effect without assumptions like similar time trends—then using a different estimation technique does not change those fundamentals.

    “Machine learning doesn't solve any of those problems. Because most of our intuitions and theorems about correlation versus causation already kind of assumed that you have an infinite amount of data”

  • The future of applied work is big-data causal inference: combining machine learning to handle wider datasets and many variables with economists' causal-inference strengths, while being honest about delegating model selection to a transparent, correctable algorithm rather than hiding arbitrary specification choices 'in the kitchen,' freeing researchers to focus energy on the genuinely hard causal-inference and robustness work.

    “let's go out there and get good at big data causal inference, with systematic model selection, being honest about how we are using the machine to choose our model and getting out valid statistical inference at the end, rather than... hiding it all in the kitchen”

  • Machine learning (high-dimensional statistics) lets researchers specify a very large list of possible covariates—potentially more variables than observations—and use the data itself to determine which ones matter, replacing the subjective 'art form' of choosing one or two theoretical proxies.

    “machine learning has this brilliant kind of characteristic: that it's going to be data-driven model selection. So, you are basically going to be able to specify a very large list of possible covariates; and then use the data to determine which ones are important.”

  • For the sake of scientific progress, smoother organizations, and rapid innovation, every group—academic or business—tends to narrow its focus and get really good at a few things while ignoring others: the machine learning community systematically ignored causal inference and got excellent at prediction, while economists got good at small-data causal inference but weaker prediction.

    “every group, whether it be academic or business, tends to narrow their focus and get really good at a few things. And ignore other problems.”

  • For the sake of progress and smoother functioning, every group—academic or business—systematically narrows its focus, getting very good at a few things while ignoring others; the machine learning community ignored causal inference to become excellent at prediction, while economists got good at small-data causal inference, leaving the gap of big-data causal inference where Athey works.

    “every group, whether it be academic or business, tends to narrow their focus and get really good at a few things. And ignore other problems.”

  • Machine learning does not solve any of the fundamental problems of distinguishing correlation from causation, because econometric identification theorems already assume infinite data; without assumptions like parallel time trends, you cannot identify the minimum wage effect, and changing the estimation technique does not change these fundamentals of causal identification.

    “Machine learning doesn't solve any of those problems. Because most of our intuitions and theorems about correlation versus causation already kind of assumed that you have an infinite amount of data”

  • High-dimensional statistics, colloquially machine learning, is effective precisely when you have more covariates than observations, enabling data-driven model selection that replaces the subjective, artistic process of hand-picking a few proxy variables.

    “machine learning has this brilliant kind of characteristic: that it's going to be data-driven model selection. So, you are basically going to be able to specify a very large list of possible covariates; and then use the data to determine which ones are important.”

  • Econometrics has always prioritized interpretable, reliable models that hold up across settings and reflect how the world works, implicitly sacrificing in-sample predictive power; machine learning is far better at fitting any particular dataset, while econometrics is better when you need robustness to changing environments and protection against spurious relationships.

    “that push in econometrics has caused us to sacrifice predictive power. And we never really fully articulated the sacrifice we were making”

  • For causal inference, you must distinguish the treatment variable (the minimum wage, the drug) from the control covariates and model them differently: apply machine learning to flexibly select controls, but give a causal interpretation only to the treatment variable, since data-mined controls are uninterpretable proxies that capture correlated information parsimoniously.

    “if you're going to be doing data mining on these variables, you can't give them a causal interpretation. But the whole point is that you really only should be giving causal interpretations to the policy variables you were interested in to start with”

  • Athey's advanced techniques incorporate sample splitting behind the scenes without throwing away half the data: by repeatedly splitting the data (selecting a model on one half, estimating on the other) and averaging the results across many splits, each model element is honest, so the average is honest and yields valid confidence intervals while using all the data.

    “You repeatedly take half the data to select a model, in the second half to estimate. You do that over and over again, with different splits, and then average up the results... if each individual one is honest, the average of all of them is honest. But you've used all of your data.”

  • Athey envisions a future of applied work where researchers are more honest by delegating the model-selection component they were embarrassed about to a machine and disclosing how it works—since it is easier to correct for a transparent algorithm than for arbitrary human selection among spreadsheet columns—freeing human energy for the genuinely hard causal-inference and robustness work.

    “let's go out there and get good at big data causal inference, with systematic model selection, being honest about how we are using the machine to choose our model and getting out valid statistical inference at the end, rather than... hiding it all in the kitchen”

0.80

Causal inference demands systematic model selection, sample splitting, and transparent specification to prevent p-hacking and spurious findings.

10 pointscentrality 5/5
  • Instead of having a research assistant secretly run a hundred regressions and reporting only the chosen result as if the 'gods of economics' had specified it, machine learning lets you systematically optimize a defined objective function (goodness of fit) over thousands of models on one half of the data, making model selection explicit and honest.

    “Instead of asking your research assistant to run 100 regressions, let the machine run 10,000. Then pick the best one. But you only use what you learned on a separate set of data.”

  • Advanced techniques can incorporate sample splitting behind the scenes without wasting half the data: repeatedly take half the data to select a model and the other half to estimate, do this over many splits, and average the results—since each individual split is honest, the average is honest, giving valid confidence intervals while using all the data and barely hurting performance.

    “You repeatedly take half the data to select a model, in the second half to estimate. You do that over and over again, with different splits, and then average up the results... if each individual one is honest, the average of all of them is honest. But you've used all of your data.”

  • Searching across many model specifications and reporting only the one that confirms your hypothesis produces an incorrect p-value (p-value hacking), because the reported probability ignores the many specifications tried; this has caused a replication crisis in which large fractions of studies in fields like psychology cannot be replicated.

    “if you try a hundred ways, and there's a good chance one of them will come up with your negative result.”

  • To protect against finding spurious patterns when searching many variables, use sample splitting: use one dataset to figure out the right model and a separate, clean dataset to estimate effects, so that you gain the better functional form from data mining while still obtaining correct standard errors.

    “to sort of protect yourself against that kind of over-fitting, by a very simple method... is to do sample splitting. So you use one data set to figure out what the right model is; and another data set to actually estimate your effects.”

  • If you have a thousand characteristics for a thousand patients, you can almost certainly find some characteristics that the few patients with great outcomes share in common, so you can construct a spuriously favorable treatment effect for some subgroup even when the drug has no real effect—this is impossible to do reliably with only one or two characteristics.

    “if you have 1000 characteristics then you can almost certainly find something that they have in common, and so you can pretty much always find a positive effect for some subgroup.”

  • If you measure a thousand characteristics on a thousand patients and look for what the 15 best-responding patients share, you will almost certainly find common traits by chance alone, letting you 'prove' a spurious subgroup treatment effect even when the drug has no real effect; this is why the FDA requires pre-analysis plans.

    “if you have 1000 characteristics then you can almost certainly find something that they have in common, and so you can pretty much always find a positive effect for some subgroup.”

  • Instead of having a research assistant secretly run 100 regressions and reporting only the chosen one as if the 'gods of economics' specified it from the start, researchers should let the machine optimize an explicit objective function over 10,000 models, but validate on separate data, making the selection process systematic and honest rather than hidden in the kitchen.

    “Instead of asking your research assistant to run 100 regressions, let the machine run 10,000. Then pick the best one. But you only use what you learned on a separate set of data.”

  • Pre-analysis plans prevent subgroup data mining but are inefficient because most drugs genuinely have heterogeneous effects; forcing researchers to specify all possible subgroups in advance throws away the valuable information that data could reveal afterward about which groups respond.

    “most drugs do have heterogeneous effects. And they do work better for some people than others. And it's kind of crazy to have to specify in advance all the different ways that that could happen. And so you are actually throwing away good information by not letting the data tell you afterwards.”

  • If you try many model specifications and report only the one that confirms your hypothesis with a standard p-value, the reported significance is wrong because the p-value ignores the many specifications searched; this 'p-value hacking' has produced a replication crisis, with large fractions of studies in some psychology subfields failing to replicate.

    “if you actually tried a hundred to get this, that's an incorrect p-value. And we call that p-value hacking. And it's actually led to sort of a crisis in science.”

  • In a predictive model, the marginal effect of any component should never be given a causal interpretation; political sentiment correlating with employment does not mean it causes employment, and drawing causal conclusions requires a designed model justifying the necessary assumptions.

    “predictive models--the marginal effect of any component of that model should not be given a causal interpretation. So, I don't want to say that just because political sentiment was correlated with employment, it doesn't mean that political sentiment caused employment.”

0.64

Credible counterfactuals require designed comparisons using difference-in-differences, synthetic controls, and placebo tests to rule out confounding.

9 pointscentrality 4/5
  • Rather than relying on a single comparison city, a synthetic control group (developed by Alberto Abadie) builds a weighted average of multiple similar cities, chosen so that they match the treated city across many characteristics and on its pre-treatment time trend, producing a more credible counterfactual.

    “a more complex thing developed by Alberto Abedie is a method called Synthetic Control Groups”

  • Because labor markets have time trends even without policy changes, measuring a minimum wage's causal effect requires constructing a counterfactual—what would have happened absent the change—via difference-in-differences, using a comparable city's time trend as a control group for the treated city.

    “you need to know what would have happened if you hadn't changed the minimum wage. And we call that a counterfactual.”

  • A placebo test validates difference-in-differences assumptions by running a fake analysis on a period before any policy change; if the fake analysis spuriously shows an impact (e.g., because the treated city was on a steeper underlying slope), the underlying parallel-trends assumption is invalid.

    “if you did a fake analysis a year ago when there wasn't really a change, that fake analysis might also make it look like there was an impact unemployment. We call that a placebo test.”

  • Naively comparing employment across places with high versus low minimum wages is misleading because the places that choose to raise the minimum wage are systematically different—often booming tech cities with price-insensitive customers and scarce labor supply—so the effect observed there will not generalize to places like rural Iowa.

    “the places that have chosen to raise the minimum wage are not the same as the places that have not.”

  • Naively comparing employment across high- and low-minimum-wage places is misleading because places that raise the minimum wage are systematically different (e.g., booming tech cities with high rents and price-insensitive customers), so a policy that works in San Francisco or Seattle may fail in rural Iowa.

    “the places that have chosen to raise the minimum wage are not the same as the places that have not.”

  • Rather than relying on a single comparison city, a synthetic control group constructs a weighted average of multiple cities chosen so they match the treated city across many dimensions and reproduce its pre-treatment time trend, yielding a more credible counterfactual.

    “a more complex thing developed by Alberto Abedie is a method called Synthetic Control Groups”

  • Because cities' labor markets have time trends even absent policy change, measuring a minimum wage effect requires constructing a control group (e.g., a similar city) to estimate the counterfactual time trend, which is the basis of the difference-in-differences technique.

    “one of the techniques that economists try to use in dealing with that is something that is called 'difference in differences.' So, essentially you can think about trying to create a control group for the time trend of the cities or states that made a change.”

  • To get valid statistical inference after data-driven model selection, use sample splitting: use one half of the data to choose the model and the other half to estimate the effect; though throwing away data seems horrifying in a small-data world, the gain in functional-form fit justifies it and yields a correct standard error.

    “one of the ways that I've advocated modifying machine learning methods for economics is to sort of protect yourself against that kind of over-fitting, by a very simple method... is to do sample splitting. So you use one data set to figure out what the right model is; and another data set to actually estimate your effects.”

  • You can validate a difference-in-differences assumption by running a 'fake' analysis on a period before any actual policy change occurred; if that placebo test detects a spurious effect, it reveals that the underlying parallel-trends assumption is invalid.

    “if you did a fake analysis a year ago when there wasn't really a change, that fake analysis might also make it look like there was an impact unemployment. We call that a placebo test.”

0.53

Tech firms' randomized experiments are numerous and causally powerful but too short-term and firm-specific to yield generalizable social science.

10 pointscentrality 3/5
  • Random assignment lets tech firms separate correlation from causality almost perfectly, so they can act on a causal effect (e.g., dark blue brings users back more) even without understanding why; with often two black-box algorithms, you learn one is better and use it without knowing the mechanism.

    “they are able to separate correlation from causality almost perfectly.”

  • Because changes are validated by randomized control trials rather than executive judgment, A/B testing decentralizes innovation: thousands of engineers or even a summer intern can write code, test it, and ship it live to users within days, provided it follows guidelines and the trial shows users loved it.

    “what that allows is decentralization. That allows thousands of engineers, even interns, to make changes.”

  • The A/B test is probably the most impactful business process innovation in a long time: most innovation at big tech companies is incremental rather than the mythical genius leap, and firms like Google run over 10,000 randomized controlled trials a year, with nothing shipping—fonts, colors, or algorithms—unless a randomized control trial shows users loved it.

    “the A/B test is probably the most impactful business process innovation that has occurred in a very long time. So, just to think about how does Google get better, they run 10,000 or more randomized control trials every single year.”

  • Tech firms are highly scientific and statistically sophisticated because their experiments drive bonuses, promotions, and billions in revenue; they enforce strict statistical significance and must correct for issues like correlated outcomes when the same user appears in a treatment group multiple times, making p-values central to how innovation happens.

    “these boring formulas about p-values and statistical--that's like right at the heart of how innovation happens at these firms.”

  • Unlike economists' experiments aimed at generalizable parameters (risk aversion, class size effects), tech firms' 10,000 experiments are specific to their interface and often compare black-box algorithms, so even if published they would not reveal fundamental generalizable truths—the firms are not hiding social science, that's simply not what the experiments are about.

    “economists' experiments are often designed to understand risk aversion, or understand the effect of class size, or some kind of generalizable parameter... An experiment that's done at tech companies is really specific to their interface”

  • Tech firms systematically innovate in the wrong dimensions because running long experiments is expensive, biasing them toward short-term experiments; changes that confuse users initially but help long-term get penalized, so summing the short-term gains across years might predict 300% revenue improvement while actual improvement was only 10%—they excel at incremental short-term innovation but miss more radical changes.

    “there's sort of a bias towards running short-term experiments... they tend to systemtatically be biased toward things that are good in the short term. And bad in the long term.”

  • Most of the 10,000 tech experiments are not designed to uncover fundamental generalizable truths about the world; they are specific to a firm's interface, often comparing two black-box algorithms where you never learn why one worked better, so even if published they would not yield generalizable parameters like risk aversion or class-size effects that economists seek.

    “often it's just Algorithm A versus Algorithm B. And both algorithms are black boxes and you won't really know why one of them worked better than another. It's just--it was better, so we use it.”

  • Contrary to the popular image of lone-genius leaps like Steve Jobs designing fonts, most tech innovation is incremental, and the A/B test is the most impactful business process innovation in a long time; firms like Google run over 10,000 randomized control trials per year, and nothing ships without passing one.

    “the A/B test is probably the most impactful business process innovation that has occurred in a very long time. So, just to think about how does Google get better, they run 10,000 or more randomized control trials every single year.”

  • Because randomized trials are statistically valid only for what they measure (short-term outcomes) and running long experiments is expensive, tech firms are systematically biased toward short-term improvements; summing all experiments might predict 300% revenue growth while actual growth was 10%, because interface changes confuse users initially and firms innovate in the wrong, short-term dimensions while missing radical long-term improvements.

    “there's sort of a bias towards running short-term experiments... they tend to systemtatically be biased toward things that are good in the short term. And bad in the long term.”

  • Because A/B tests provide objective causal validation, tech firms can decentralize innovation: thousands of engineers, even summer interns, can write code, test it, and ship it live to users within days as long as the randomized control trial shows users responded positively and the change follows the rules.

    “what that allows is decentralization. That allows thousands of engineers, even interns, to make changes... you can test it; and it can go live on users within days. As long as it follows certain rules and guidelines, and as long as the randomized control trial showed that users loved it.”

0.49

Causal inference requires treating policy variables distinctly from covariates, using machine learning only for covariate selection.

3 pointscentrality 4/5
  • The starting point for causal inference is to treat the causal variable of interest (e.g., minimum wage, the drug) differently from all the control covariates: use machine learning to data-mine the covariates but model the treatment variable carefully and separately, rather than putting everything into one regression with the same statistical treatment.

    “the starting point for doing causal inference, which is mostly what we're interested in, in economics, is that you treat these things differently.”

  • Variables selected by machine learning regularization cannot be given a causal interpretation because the method finds a parsimonious proxy among many highly correlated covariates; causal interpretations should only be given to the policy variable of interest, which is what should have been the focus all along.

    “if you're going to be doing data mining on these variables, you can't give them a causal interpretation. But the whole point is that you really only should be giving causal interpretations to the policy variables you were interested in to start with”

  • A predictive model fits relationships in the data to predict an outcome (e.g., employment without a minimum wage change), but the marginal effect of any component must not be given a causal interpretation—correlation of a variable like political sentiment with employment does not mean it caused employment.

    “predictive models--the marginal effect of any component of that model should not be given a causal interpretation.”

0.49

Micro-level data with natural variation provides far greater statistical power to detect causal effects than macro-level aggregates.

3 pointscentrality 4/5
  • Modeling phenomena at the micro level rather than the macro level is powerful because instead of having only two recessions (two data points) to study an industry, you can use city- or county-level data where each locality experiences its own mini-recession, giving far more variation and many more natural experiments to build accurate predictions—but only when the phenomena are genuinely micro-level rather than diffuse macro effects like NAFTA.

    “each locality would have experienced their own mini-recession... So you could really have much more variation in the data: many more mini-experiments.”

  • Modeling phenomena at the micro level rather than macro level is powerful because instead of a few national data points (e.g., two recessions), each locality experiences its own mini-recession, providing far more variation and many mini-experiments to build better models, but only when the phenomenon is genuinely micro-level rather than a macro effect like NAFTA's cross-city movements.

    “each locality would have experienced their own mini-recession... you could really have much more variation in the data: many more mini-experiments. Which allows you to build a better model and make much more accurate predictions.”

  • The higher wages of college-educated workers cannot simply be extrapolated to what would happen if more people went to college, both because the new attendees differ from current ones and because expanding supply produces equilibrium effects that lower college graduates' wages.

    “if you send more people to college, you'll decrease the scarcity of college graduates, which can also lower the equilibrium wages.”

0.40

Technology platforms exert large causal effects on economic outcomes through algorithmic ranking and user choice architecture.

5 pointscentrality 3/5
  • Policy effects like NAFTA's are hard to evaluate because one side—a factory shutting down—is easy to count, while the benefits, such as many consumers paying lower prices and gaining disposable income to spend elsewhere, are diffuse and therefore much harder to measure, making net job effects difficult to quantify reliably.

    “the benefits from, say, lots of consumers having cheaper goods and thus having more disposable income to spend on other things... are much harder to measure because they are more diffuse.”

  • A large Facebook experiment that randomized voting nudges found that showing people their friends had voted encouraged them to get out and vote, demonstrating that social-science collaborations with tech firms can uncover important generalizable facts.

    “there's a large experiment on Facebook where they randomized nudges for voting. And they found that, you know, when your friends, when you show people that their friends voted, that encouraged people to get out and vote.”

  • Athey's experiment re-ranking Bing search results found the causal effect of moving a link from top to third position, yielding the generalizable insight that how a news site, search engine, or commerce engine ranks results has a large causal effect on what people buy and read—meaning technology platforms shape the informativeness of the economy and the winners and losers in commerce.

    “how a search engine or how a commerce engine ranks results has a very large causal effect on what people buy. What they read. It tells us that technology platforms have a big impact on the informativeness of the economy as well as the winners and losers in commerce.”

  • The observed wage premium for college graduates cannot be assumed to apply if more people are sent to college, both because new entrants differ from current graduates and because increasing the supply of graduates reduces their scarcity and lowers equilibrium wages.

    “if you send more people to college, you'll decrease the scarcity of college graduates, which can also lower the equilibrium wages.”

  • The FDA requires a pre-analysis plan to prevent spurious subgroup mining, but this is inefficient because most drugs genuinely have heterogeneous effects that work better for some people, and forcing researchers to specify all subgroups in advance discards good information that the data could reveal afterward.

    “the problem with that, the inefficiency there, is that most drugs do have heterogeneous effects. And they do work better for some people than others. And it's kind of crazy to have to specify in advance all the different ways that that could happen.”

0.21

Technology platform effects are difficult to measure because benefits disperse widely while costs concentrate or confound with other changes.

3 pointscentrality 2/5
  • Athey's Bing experiment re-ranking search results showed that how a search engine, news site, or commerce platform ranks results has a large causal effect on what people buy and read, implying technology platforms substantially shape the informativeness of the economy and the winners and losers in commerce.

    “I did release the result of an experiment I ran at Bing where I re-ranked search results. And so I found the causal effect of taking a link from the top position to the third position.”

  • Tech firms often allow social scientists to publish generalizable findings more readily than proprietary business secrets, producing important facts such as the Facebook voting experiment showing that displaying friends' voting behavior nudges people to vote, and Yelp health-score information affecting food poisoning rates.

    “there's a large experiment on Facebook where they randomized nudges for voting. And they found that, you know, when your friends, when you show people that their friends voted, that encouraged people to get out and vote.”

  • Policies like NAFTA are hard to evaluate because one side of the ledger (a factory shutting down, jobs lost) is concentrated and easy to count, while the benefits (cheaper goods raising disposable income for many consumers) are diffuse and hard to measure, biasing perception toward the visible costs.

    “one side can be easy to measure, like a factory shutting down. But the benefits from, say, lots of consumers having cheaper goods... those benefits are much harder to measure because they are more diffuse.”