What this covers

Jim Manzi and Russ Roberts examine the Oregon Medicaid study—a rare randomized controlled trial created when Oregon ran a lottery to allocate limited Medicaid expansion slots—and unpack what its null results on physical health actually tell us. The conversation moves through the study's design, its headline findings (no significant improvements in cholesterol, blood pressure, or blood sugar over two years, but increased health service use and lower depression), and the interpretive challenges that arise when translating experimental results into policy conclusions. Manzi's core argument is that the study's limitations demand epistemic restraint: the findings do not settle broad questions about Medicaid's worth, but instead answer only the narrow question of what happened in Oregon in these circumstances.

The discussion homes in on several overlooked details. Roughly 40% of lottery winners never even submitted applications, a behavioral fact that complicates causal attribution—if the people who enrolled were more prudent or conscientious than non-enrollees, improvements might reflect that selection rather than Medicaid itself. Manzi argues this marker of non-enrollment predicts non-compliance with the chronic disease management regimens needed to move the metrics being measured. The conversation also highlights the persistent conflation of statistical significance with policy importance: observed effects were both statistically insignificant and substantively small (a 0.81-point blood pressure reduction when prior work expected around five points), and when you apply intention-to-treat logic—comparing all lottery winners against all losers, not just those who enrolled—the treatment effect shrinks by roughly 75%. Manzi further notes an unexpected uptick in smoking among Medicaid recipients, small in magnitude but closer to statistical significance than most results, which he explores through the Peltzman effect lens. He concludes that reliable knowledge about policy interventions requires not one definitive trial but many fast, cheap experiments embedded in program delivery, allowing patterns to emerge across replicated studies rather than resting on a single trial's interpretation.

Sharpest takeaway

Manzi argues the Oregon Medicaid study's null physical-health results, properly interpreted through intention-to-treat logic and the distinction between statistical and policy significance, support epistemic humility rather than sweeping conclusions about Medicaid, and that progress comes from many fast cheap experiments rather than one definitive trial.

  • 40% of lottery winners never submitted applications, a behavioral marker that complicates ascribing effects to Medicaid versus the prudence of those who enrolled
  • Statistical significance is routinely conflated with policy importance, obscuring that observed health effects were both insignificant and small
  • Experiments answer only very narrow causal questions, so reliable knowledge accumulates pointillistically across many trials

The argument · threads6 threads · 21 claims
0.73

The study was underpowered to detect realistic clinical effects given prior evidence on expected treatment magnitudes.

3 pointscentrality 5/5
  • Statistical power is analogous to the magnification of a microscope: a low-powered instrument that fails to detect a tiny effect cannot establish the effect's absence, because failure to observe reflects the instrument's precision rather than the true state of the world; the Oregon study may be 'underpowered' to detect realistically expected clinical effects.

    “If I use that exact same microscope to try and find... not a bug but a tiny microbe... I don't see the microbe, but I can't reliably say that therefore there is no microbe there... my failure to see the microbe is a statement about the precision of my instrument, not about whether there's really a microbe on the leaf.”

  • The only prior randomized study of varying levels of health insurance coverage, the RAND health insurance experiment of about 30 years ago, showed no effect on health outcomes despite increased health care usage, which is why the Oregon null result did not surprise Manzi.

    “the only reason it didn't surprise me is, out of interest in the methodology, I'd looked at the only prior randomized study, which looked at the effect of varying levels of health insurance coverage, which was the RAND study of 30 years ago, which showed no effect.”

  • For diastolic blood pressure, the study could reliably detect a reduction of about 3 points, while prior studies (RAND at 3 points, another at 6-9 points, averaging about 5.25) suggest the expected effect of coverage is around 5 points; therefore the experiment was adequately powered for this metric, yet it found only a 0.81 point reduction.

    “the precision of that microscope can go down to a measurement of a size of 3; the effect they would expect is 5.”

0.73

Analyzing only lottery winners who enrolled reintroduces selection bias and confounds treatment effects with prudence differences.

3 pointscentrality 5/5
  • The same behavioral characteristics that lead a lottery winner to not submit the Medicaid application within 45 days (low conscientiousness, unstable living situations, low prioritization) are intuitively correlated with failure to comply with the chronic-disease management regimens needed to improve blood pressure, blood sugar, and cholesterol, so non-enrollment is a marker for likely non-compliance.

    “the fact that you didn't submit the form within 45 days is a marker for your likely behavior over the 2-year coverage period that's being studied here”

  • Once you stop analyzing the full randomized groups (intention-to-treat) and instead analyze only the subset of lottery winners who actually obtained Medicaid, you reintroduce the very selection-bias problem randomization was meant to eliminate, because you no longer know which losers are comparable to the self-selected enrollees; Manzi calls this introducing 'the bacillus of econometrics.'

    “As soon as I start to take subgroups of the group that won, I immediately find myself back in the same problem: How do I know the subgroup, which is the 60% who submitted the application... how do I know which of the people who lost the lottery are most like them?”

  • If the lottery winners who enrolled in Medicaid are more prudent, conscientious, and self-disciplined than those who didn't enroll, then comparing the enrolled (all prudent) against a control group that is a mix of prudent and imprudent people will mis-ascribe improvements actually caused by prudence to the effect of Medicaid coverage.

    “As soon as I start to take subgroups of the group that won, I immediately find myself back in the same problem: How do I know the subgroup, which is the 60% who submitted the application... how do I know which of the people who lost the lottery are most like them?”

0.57

Depression improvement within one month suggests placebo effect; single experiments answer only narrow causal questions, not policy.

5 pointscentrality 3/5
  • The statistically significant reduction in depression apparently occurred within the first month of the study, which makes it less plausible that it reflects actual clinical treatment and more like a placebo effect; combined with construct-validity concerns about measuring depression via a questionnaire, this complicates interpretation.

    “a key issue to remember when you see a significant reduction in depression is that apparently it all occurred within the first month of the study. You can see it immediately. So it makes it much less plausible that this is an effect of actual treatment”

  • A randomized controlled trial is the gold standard because, unlike regression or matching that attempt to hold observable factors equal, random assignment ensures that even unobserved or unmodeled differences between treatment and control groups are approximately equally distributed, subject only to sampling error.

    “If we randomly select individuals into the test group versus the control group... I know that even if there are factors I haven't thought of, I should have them approximately equally distributed in the test group and the control group.”

  • An experiment answers an extremely narrow question, not 'Does Medicaid help or hurt?' but only 'When I randomized these people into this lottery in Oregon on these dates, what was the causal effect of being randomized in or out?'; stepping even an inch beyond that platform risks fooling yourself into thinking your knowledge extends more broadly than it does, which is why broad knowledge must be built pointillistically from many narrow experiments.

    “It's not: Does Medicaid help or hurt? You know, what you answered is: When I randomized people to this lottery in Oregon on these dates, what was the causal effect of being randomized in or out of that lottery?”

  • The p-value is the probability that an observation as extreme as the data would occur by chance given that the true effect is zero (the null hypothesis is correct); it is not the probability that the result is due to the treatment.

    “the p value, or the 1 minus 95%, or .05, is the probability that an observation as extreme as that data would occur by chance, given that the true effect was 0.”

  • No single experiment can settle policy debates once and for all; instead, we should embed the capability to run many fast, cheap experiments within program execution (e.g., 85 experiments across 50 states per year) so that reliable engineering conclusions emerge from replicated patterns rather than from one underpowered 'moon shot' trial.

    “I think the idea that we're going to do some experiment and we are going to slam our fist down on the table and say, now we've settled the issue, now we know the answer, is extremely unrealistic.”

0.53

Only 60% of lottery winners even applied for Medicaid, making intention-to-treat effects roughly one quarter of reported effects.

2 pointscentrality 4/5
  • Of the roughly 35,000 people who won the Oregon Medicaid lottery, only about 60% submitted an application within the 45-day deadline, meaning roughly 40% declined or failed to claim very inexpensive or free health coverage.

    “of the 35,000 people who won the lottery, about 60% filled in and submitted the application within the deadline. Therefore, obviously 40% did not.”

  • Because only about 25-30% of lottery winners actually received Medicaid, the intention-to-treat effect (the causal effect of winning the lottery) is roughly one quarter of the reported treatment effect; you can divide the headline per-person effects by four, and Manzi's critique of the study only strengthens under this stricter analysis.

    “you can essentially take any of the estimated effects in the headline results in the paper and divide by 4. It's about 24, 25% of the effect, which you would expect, because only a small subset, something like 30% or so of those who won the lottery, got the treatment.”

0.40

Reducing cost of risky behavior's negative outcomes can via moral hazard increase that behavior, possibly including smoking.

2 pointscentrality 3/5
  • Reducing the cost (in the broad, non-monetary sense) of the negative outcomes of a risky behavior like smoking can, via the Peltzman/moral-hazard effect, increase the incidence of that behavior; in this context insurance coverage might most plausibly cause people to delay quitting smoking rather than to start, or alternatively reduced financial stress from coverage frees money to spend on cigarettes.

    “if I reduce the cost of a risky behavior, I will, on net induce the incidence of that risky behavior. Because the thing that I've changed is the cost of the bad outcome is not quite as bad as it used to be.”

  • In Table 5, the estimated causal effect of Medicaid coverage was an increase in smoking incidence, from about 43% in the control group to about 48% in the test group, a five-percentage-point increase that was not statistically significant (p-value about 0.18) but had a closer p-value than nearly any other result in the study.

    “In the control group it was about 43% of people who smoked. And the causal effect as indicated by the study was a 5% increase, 48% smoking in the test group--which was not a statistically significant result.”

0.32

Small average effects may reflect large reductions in high-risk subgroups and prior studies show physical health hard to move.

3 pointscentrality 2/5
  • The Oregon study population smoked at roughly twice the rate (43-48%) of the general US population of similar age (under 25% at the time), reflecting that smoking is an incredible class marker in the United States, far more class-defined than in countries like France.

    “you are really talking about a group of people who are smoking, roughly speaking, around twice the rate that you would expect for people that age in that time period. The obvious observation is, smoking is an incredible class marker in the United States.”

  • A small average effect such as a one-point blood pressure reduction is not necessarily clinically insignificant, because it could arise from a large reduction concentrated in a small high-risk subgroup, which could still translate into a material reduction in total mortality.

    “it could be that what you have is 99% of people are at, you know, 75, and you have 1% of people that at 140, and the way you got the 1 point reduction was take that 1% of people and bring them down substantially.”

  • The study confirms rather than contradicts the prior RAND result that physical health indicators are very difficult to move over a measurement period like this; this does not mean Medicaid is a bad idea, but anyone whose support is predicated on the belief that it will make many sick people physically better should be more hesitant about that belief.

    “if your support for that change is predicated on the idea that it's going to make lots of sick people better, physically, this ought to make you a bit more hesitant about that belief.”