What this covers
Anupam Bapu Jena and Russ Roberts discuss how chance events—road closures, doctor absences, inspection weeks—reveal the actual causal levers in medicine. Jena draws on natural experiments to isolate what changes outcomes when patients are, in effect, randomly assigned to different conditions. The conversation moves from specific empirical findings to larger questions about medical training, judgment, and how to measure a good doctor.
Three findings anchor the discussion. Marathon road closures raise cardiac mortality about 20% in affected areas, a delay-driven effect proven by the fact that victims are elderly patients with dementia and dialysis needs—not runners. During major cardiology conferences, when some cardiologists leave town, patients hospitalized for acute cardiac conditions fare better and certain procedures drop roughly 30%, suggesting marginal interventions carry risks that outweigh benefits. During Joint Commission inspections, mortality falls without evidence that compliance actions themselves improve care, pointing to a Hawthorne effect where observation sharpens focus.
From these anchors the conversation spirals outward. Jena argues that attribution asymmetry—doctors credit good outcomes to their actions and bad ones to patient risk—leads to over-treatment, and that the real problem in medicine is not knowledge but judgment: knowing when to deviate from clinical guidelines, how much effort to invest in speed, and which qualities distinguish excellent diagnosticians. He questions whether the selection and training of doctors actually identifies and cultivates the traits that matter now—arguing medical school emphasizes encyclopedic knowledge and rare diagnoses while missing diagnostic reasoning and common-condition mastery. Roberts raises how hospitals might elicit the costly focus that appears during inspections across all 52 weeks, and both speakers explore whether the best physician may be one who applies guidelines roughly 70% of the time but preserves space for intuition grounded in experience rather than gut feeling.
Jena argues that natural experiments reveal hidden, non-obvious forces shaping medical outcomes—road closures, doctor absences, regulatory observation—and that medicine's real challenge is not knowledge but judgment under uncertainty: knowing when to act, when to deviate from guidelines, and how to sustain focus.
- Marathon road closures raise cardiac mortality ~20%, proving time-to-care causally matters
- Patients do better when cardiologists leave for conferences, suggesting overuse of marginal procedures
- Mortality falls during Joint Commission inspections, evidencing a Hawthorne/focus effect
Physicians cannot distinguish beneficial from harmful interventions because feedback and incentives systematically distort their beliefs.
- A given cardiologist could confidently identify their highest-value procedures, but in practice they are not constrained to those and proceed down the risk/benefit distribution to patients for whom risk outweighs benefit without knowing it; the conference studies show physicians can in fact prioritize and intentionally skip such procedures when forced to, because the patients who go untreated during meetings are selected deliberately, not at random.
“in the real world, they're not constrained to doing it in 100 people. They can go all the way down the distribution of risk/benefit; and they may end up in a place where the benefit is outweighed by the risk. And, they don't know that.”
- Insurance companies and regulatory agencies measure physician quality by guideline adherence (e.g., treating 100% of hypertension patients), penalizing deviation, but these metrics don't allow for the possibility that some deviations are warranted, so we should empirically determine whether physician deviations from guidelines are actually good or bad for patients rather than presuming non-adherence is failure.
“an insurance company might, being a doctor for deviating from that, a regulatory agency that is measuring the quality of a doctor's care might say... 'Why aren't the other 15% being treated?'”
- Jena poses a thought experiment of three doctors—one who ignores evidence and practices from old experience, one who follows the latest guidelines like a cookbook without deviation, and one who follows guidelines ~70% of the time but deviates ~30% for unstated reasons—and argues the empirical question of which has the best outcomes is genuinely open, with intuition favoring the third only if their deviations are systematically good.
“there's three types of doctors... the first doctor is a doctor who does not know any of the evidence... the second doctor is the polar opposite... they follow that information like a cookbook... a third type of doctor who--let's say, 70% of the time... follow the cookbook... But then, 30% of the time, they're doing something different.”
- The higher procedure rates when senior cardiologists are present are not primarily driven by fee-for-service financial incentives—because these are academic teaching hospitals where physician pay is less tied to volume and the responsiveness of care to payment, while real, is small—but rather by a physiologically reasonable desire to 'do something' to fix a visible problem absent convincing contrary evidence.
“in most of these settings, we're looking at academic teaching hospitals where their doctors are not often paid the same way that they might be paid outside of that setting... it's not nearly as big a deal as in other places.”
- Doctors face a feedback problem: when a treated patient does well they credit their intervention, and when one does poorly they attribute it to the patient's underlying risk, allowing them to rationalize beliefs; combined with growing (often rational) confidence from experience, this leads them to ascribe random good outcomes to their treatments and over-apply interventions to similar future patients.
“If you do that to the same sick person in front of you and you do a procedure, they do poorly, you don't think to yourself: 'Wow, the procedure caused that.'... So, there's ways that people can rationalize what they want to be true.”
Natural experiments using as-good-as-random assignment reveal causal effects of medical practice on mortality that randomized trials cannot.
- On marathon days, mortality from cardiac arrest and heart attack rises about 20% in areas along the marathon route compared to surrounding days, driven by ambulance transport delays (up ~30% in mornings/early afternoons) caused by road closures rather than by participants collapsing—evidenced by the fact that the affected patients are late 70s and older, including dementia and dialysis patients who don't run.
“mortality goes up quite a bit on the day of a marathon compared to the surrounding days in the areas that are right on the marathon route. And the second thing is that ambulance transport times go up by about 30%”
- To credibly claim a natural experiment, one must first demonstrate that the exposed and unexposed groups are similar on all observable characteristics—demographics, chronic conditions, procedures performed, staffing levels—because any imbalance suggests the exposure was not truly as-good-as-random, and even then a valid experiment establishes the effect but not the mechanism.
“the first thing that we show is that on all observable characteristics that we have, that the groups are similar... We show on demographics that they're similar. We show that in terms of the chronic conditions that they have, that they are similar.”
- During major cardiology conferences, patients hospitalized for acute cardiac conditions have lower mortality if some cardiologists are out of town, and rates of certain procedures fall by about 30%, suggesting that on the margin some interventions performed in normal times carry risks that outweigh their benefits.
“patients who have cardiac problems, acute cardiac problems, they actually do better on the dates of those meetings, if they by chance are hospitalized when the cardiologists or some cardiologists are out of town.”
- Using natural experiments where patients are as-good-as-randomly assigned to hospital doctors, younger physicians (fewer years out of residency) achieve lower patient mortality for general medical conditions like pneumonia, heart failure, and infection—likely due to recency of training—but this experience disadvantage disappears for high-volume older doctors, suggesting volume substitutes for recency.
“younger doctors tend to have better outcomes as measured by mortality. So, a doctor who is, let's say, five years out of residency, the mortality rate for their hospitalized patients will be lower than the doctor who's 10 years out, 15 years out”
- Although economists have used natural experiments for decades and randomized trials are common in medicine, natural experiments—where people are exposed by chance to something like a hurricane, a policy threshold, or their age rather than by an investigator's hand—are not used as often in medicine despite their power to reveal causal effects.
“In economics, a lot of economists have, for decades now, been using natural experiments. We don't see them being used as often in medicine.”
Current physician quality measurement by guideline adherence and outcome data alone misses what actually matters for patient welfare.
- Mortality of patients who by chance are admitted during a Joint Commission inspection week falls compared to surrounding weeks, and since the patients are demonstrably similar and there is no evidence the specific compliance actions improve quality, the best explanation is the Hawthorne effect—being observed makes clinicians more focused and less distracted during high-effort tasks.
“mortality of people who, by chance, are admitted to the hospital during that week, their mortality falls compared to people who are admitted to the hospital in the surrounding weeks.”
- Statistical significance (the result is unlikely due to chance given sample size) must be distinguished from economic or clinical significance (the magnitude is large enough to matter in practice); a result can be statistically significant yet trivial, or so large that it signals a data error or an unobserved confounder rather than a true effect.
“we always want to distinguish between a pure statistical significance versus economic significance; and, very closely related to economic significance, when you're talking about medical issues, I'd call that clinical significance”
- A better way to measure physician quality than outcomes data alone may be revealed preference: observe which doctors medical professionals repeatedly choose for their own and their family's care, and test whether those doctors align with standard quality measures—if they don't, it signals we are mismeasuring quality or omitting important factors.
“there's more of a revealed preference way to measure quality, which is to say, 'Let's look at doctors who have medical problems... and let's look at who they go to for care.'”
- The way doctors are selected and trained is arcane: medical school teaches much that is irrelevant to practice and fails to teach critical skills like interpreting the quality of clinical evidence and diagnostic reasoning, even though misdiagnosis is now recognized as a key quality problem; meanwhile the outcome gradient between prestigious and non-prestigious schools has likely narrowed because encyclopedic knowledge matters less in an era of computers.
“the way that we pick doctors and the way that we train doctors is arcane.”
Physicians should deliberatively weigh intervention timing and intensity against imposed costs and risks to other patients.
- Emergency medical service planning for large public gatherings—marathons, concerts like Taylor Swift shows, July 4th celebrations—focuses on the safety and health of event participants rather than bystanders affected by road closures and delays, a blind spot that could be addressed with measures like helicopter pods near routes evaluated through cost-effectiveness analysis.
“we focus on the safety and health of the participants of those events, not so much the bystanders.”
- Doctors should process personal mistakes through a structured sequence—asking what happened, how bad it was, whether the bad outcome resulted from something they did and could do differently (versus a systemic cause they couldn't change), and what changing their practice would cost in terms of new risks imposed on other patients—rather than over-learning in an extreme way that simply substitutes one kind of error for another.
“ask yourself, 'All right, what happened?'... 'how much do I think that this bad outcome was a result of something that I did and could do differently?'”
- While faster medical care intuitively seems better, sooner is almost always more costly, and in many situations later care is no worse and substantially cheaper, so the real decision is how much effort to invest in timeliness for a given acute problem—a tradeoff the marathon study helps quantify because nature provides the delay data a randomized trial never could.
“Sooner is almost always more costly.”
- More people would be projected to die from marathon-related road closures in any given city in any given year than died in the Boston Marathon bombings, reframing which marathon risk is actually larger.
“more people would be projected to die from marathon-related road closures in any given city in any given year than died in the Boston Marathon bombings”
- Common medical ailments are frequently misdiagnosed precisely because they are common and present in non-obvious ways, suggesting a training reform where most doctors are made diagnostically excellent in common conditions while specialists handle rare cases ('zebras')—yet medical school instead over-emphasizes the zebras, causing trainees to miss the common things they should never miss.
“There's a lot of medical ailments that are very common and that are very commonly misdiagnosed--because they're common, but also because they present in ways that it's not obvious to the diagnostician.”
- Bad weather has offsetting effects on mortality: it can increase deaths through care delays, but it can also reduce traffic fatalities because fewer people drive and those who do drive more carefully, so the net effect depends on the relevant denominator and behavioral response.
“if people drive on the roads less often because of storms... you might see actually a reduction in traffic fatalities. Or people drive more carefully.”