220 | Lara Buchak on Risk and Rationality
What this covers
Sean Carroll interviews philosopher Lara Buchak on how rational choice under uncertainty should work. The conversation centers on Buchak's argument that expected utility theory is incomplete: it cannot explain why ordinary people systematically make certain choices—like avoiding a coin flip that favors them slightly while accepting one that favors them massively—without attributing implausibly extreme preferences to them. Buchak proposes adding a separate risk-weighted expected utility parameter that captures how much weight a person places on worst-case versus best-case outcomes, independent of their utility function. Under this framework, preferring certainty to a favorable gamble, or seeking out a risky bet, are both fully rational ways of taking the means to one's ends, provided a person is consistent.
The conversation moves across several domains where this distinction matters. Buchak walks through the Allais paradox and Matthew Rabin's coin-flip examples, showing how they expose the limits of standard theory. She then translates the framework to social choice, demonstrating how Risk-Weighted Expected Utility sits between John Rawls's maximin rule and John Harsanyi's utilitarian averaging, offering a middle ground that weights the worst-off heavily but not exclusively. The discussion also touches on when more information is actually harmful—a striking implication of risk-averse rationality. Throughout, Buchak clarifies what decision theory does and does not claim, and contests the standard arguments typically marshaled in favor of expected utility maximization, arguing they establish weaker principles than proponents often assume.
Buchak argues that rational choice under uncertainty requires a separate risk-attitude parameter beyond utilities and probabilities, so that weighing worst-case outcomes more heavily (or best-case more heavily) is fully rational rather than a defect of one's utility function.
- Standard expected utility theory cannot capture systematic patterns like the Allais and Rabin preferences without implausible utility functions
- Risk attitude is a distinct psychological feature (risk-avoidant vs risk-inclined) that can be modeled as a weighting over the probability distribution
- This framework illuminates both individual choices and societal distribution debates between Rawls and Harsanyi
Expected utility theory wrongly declares risk-averse or risk-seeking people irrational when they consistently weight outcomes differently.
- If a person puts more weight on worst-case scenarios than best-case scenarios in a single decision, that can be modeled by expected utility; but if they do this across every decision, there is no utility function and probability function combination that can capture their decision making, so expected utility theory must declare them irrational.
“if in every decision you care more about the worst case than the best case, there isn't going to be a utility function that can capture you.”
- In the Allais paradox, most people prefer a 10% chance of $5M over an 11% chance of $1M, yet also prefer a sure $1M over a gamble of 89% chance of $1M, 10% chance of $5M, and 1% chance of nothing; this pattern arises because the sure option has a guaranteed $1M worst case, and anyone weighing worst cases even slightly more than best cases cannot be captured by any utility function.
“in the second choice I gave you, one of the options gave me $1 million no matter what. The worst I can do in that gamble is get $1 million. And so if you're the kind of person that weighs worst cases heavily... your expected utility maximization isn't going to be able to capture you because there's no utility function that allows for that pattern of preference.”
- Both the risk-avoidant person and the risk-inclined person are behaving fully rationally; the view that people not following their utility functions are simply irrational is wrong, because being rational means being consistent in your preferences, and differing risk attitudes are consistent ways of taking the means to your ends.
“I think both people are behaving fully rationally.”
- In Matthew Rabin's example, most people reject a coin flip between losing $100 and gaining $110 but accept a coin flip between losing $1,000 and gaining $6M; under expected utility theory with a smooth (everywhere differentiable) concave utility function, rejecting the small gamble at all wealth levels implies such extreme concavity that losing $1,000 would be more valuable than gaining any amount of money whatsoever, which is absurd.
“if you don't like that initial coin flip between losing $100 and gaining $110, then if you're an expected utility maximizer, your utility function has to be pretty concave... once you get to losing $1,000, that's actually going to be more valuable than gaining any amount of money whatsoever.”
- Two people can make the identical choice (preferring one cup of coffee over a coin flip between zero and two cups) for entirely different psychological reasons: one because they don't want more than one cup (a utility fact), the other because they value each cup equally but weigh worst-case scenarios heavily (a risk-attitude fact); expected utility theory must assign them the same utility function and cannot capture this difference, motivating a theory that separates risk attitude from utility.
“because we have the same preferences, expected utility theory is going to have to say we have the same utility functions. But there are two really different psychological explanations for why we chose why we did... you might think we'd like a theory that captures the differences between our two preferences.”
- Under expected utility theory, preferring a sure $50 over a fair coin flip between $100 and $0 is fully accommodated by saying utility is not the same as money: the person simply values the second $50 less than the first, giving a concave utility function, so apparent risk aversion is explained away as a feature of utility.
“if you prefer the $50, the sure thing $50, as most people do, then the explanation, according to the expected utility theorist, has to be exactly what you said that I like the first $50 more than I like the second $50. Which means on average, that coin flip will have less utility.”
- From a decision theorist's perspective, slot machines look like trash cans for money because their expected payoff is negative.
“as a decision theorist, you look at those slot machines, they just look like trash cans for money.”
Risk attitude is a stable, independent feature of rational choice distinct from utility, requiring a weighted average model.
- Risk attitude can be reframed by treating each possible state of the world as a future possible self: the expected utility theorist insists you must give all 100 future possible selves equal weight, but Buchak holds it is up to you how much weight to put on the worst-off possible self versus the best-off, since only one will become actual, making it a reasonable way to take the means to your ends.
“the expected utility theorists will say there's a unique answer to that question about how Sean should value his future possible Seans. You should give them all equal weight in decision making. I say, no, actually it's up to you.”
- Instead of maximizing average utility, Buchak's theory maximizes a weighted average in which a person who gives more weight to the bottom portion of their probability distribution is risk-avoidant (convex risk function) and one who gives more weight to the top portion is risk-inclined (concave risk function); this adds a risk-attitude curve alongside utilities and probabilities, and both attitudes are fully rational.
“instead of maximizing the average utility, you maximize a weighted average where if you give more weight to the bottom portion of your probability distribution than the top portion, I call you risk avoidant. And if you give more weight to what goes on at the top portion, I call you risk inclined.”
- Risk attitude could be treated as an additional stable personality dimension, proposed half-jokingly as a sixth trait alongside the big five personality traits.
“Good question let's make it the big six and put risk attitude as the number six.”
- The added risk parameter is not mere curve-fitting because the goal is to capture what it is to reason instrumentally; taking the means to your ends does not require taking an average, and it is genuinely open to you how much weight to put on worst cases versus better cases, so this is a legitimate degree of freedom in rational choice rather than an arbitrary extra parameter.
“what I'm interested in is whether these parameters sort of capture what it is to reason instrumentally, which means reason about taking the means to your ends. And I think that choice involves not just taking an average... it's actually open to you what to do. It's open to you how much weight to put on worst cases or better cases.”
Rationality means consistency in preferences, which is a substantively rich notion separate from whether beliefs match evidence.
- There are two distinct senses of rationality in philosophy: purely formal consistency (analogous to logical consistency, e.g. believing P and P implies Q but not Q is irrational regardless of content) and substantive rationality (whether a belief matches the evidence). Decision theory studies the consistency sense.
“Two different senses of the word rationality, even in philosophy, and one is just purely consistency. So you can think of it by analogy to logical consistency.”
- Decision theory concerns what is rational to choose or prefer, not necessarily the best method for actually making decisions; expected utility maximization could be the correct theory of rationality while being a terrible decision procedure in practice, as when calculating expected utilities to dodge an oncoming car would take too long and get you hit.
“even if we think expected utility maximization is the right theory of rationality, it might be the case that you nonetheless shouldn't use it in a lot of decisions. For example, a car is coming at you, you need to swerve to the left or the right. If you bothered to calculate the expected utilities, it would take too long and you just get hit by the car”
- Decision theory describes how people should behave rather than how they do, but actual behavior is relevant as a datum: if people systematically diverge from a theory without good explanation, that pressures us to investigate whether the theory of rationality is wrong or whether the behavior is genuinely irrational.
“I'm trying to describe how people should behave, but how people actually behave is relevant because if we've sort of, if people behave very differently from the theory and we don't have a good explanation for that, then we're gonna want to investigate whether the theory is actually correct”
- Although decision theory only requires consistency, consistency turns out to be a difficult notion with a lot of content, so it strongly dictates what a person should do; everyone agrees we should be consistent, and the real debate is over what consistency means.
“consistency turns out to be a pretty difficult notion to say what it means and it actually turns out to have a lot of content. So consistency really dictates what you should do pretty strongly.”
- Decision theory is about rationality in what you choose or prefer under risk or uncertainty, and it requires at least two ingredients: probabilities, which measure how likely you think things are to happen, and utilities, which measure how much you like various outcomes.
“decision theory involves, at the very least probabilities, which measure how likely you think various things are to happen and utilities, which roughly measure how much you like various things.”
Weighted risk attitudes apply to distribution across people, justifying intermediate positions between maximin and utilitarian aggregation.
- To fix how much weight a just society should give the worst-off, Buchak builds on the individual-society analogy: when making decisions for other people we should be as risk-averse as it is reasonable to be (don't take a risk for someone else unless they approve), so whatever weight is reasonable to put on the bottom 10% of a distribution when deciding for someone is the weight society should put on its worst-off 10%.
“when making decisions for other people, we are as risk averse as it's reasonable to be... if we can think about how risk averse it's reasonable to be... we're going to end up putting that much weight on the worst off 10% in society.”
- We should care more about the worst-off not merely because money can buy them more things (diminishing marginal benefit), but even after accounting for how much well-being each dollar provides; the priority to the worse-off is an independent feature of risk-avoidant distribution, not reducible to a concave utility of money.
“it's not just that they count less because money can buy them less things. It's that even taking into account how much well-being each bit of money can give you, we should care more about the worst off.”
- Rawls and Harsanyi both treated choosing a society as choosing which society to live in behind a veil of not knowing who you will be, but diverged sharply: Harsanyi assigned equal probability to being each person and maximized expected utility (indifferent between equal and unequal distributions of the same total), while Rawls refused to assign probabilities and applied a maximally risk-averse maximin rule, picking the society with the highest minimum.
“Harsanyi thought, well, you assign equal probability to being each person and you maximize expected utility... Rawls, on the other hand, thought, I can't assign probabilities to my being any particular person... I use a maximally risk averse or pessimistic or ambiguity averse rule, namely I pick the society or I pick the choice with the highest minimum. I maxim in.”
- Rawls's maximin is too risk-averse because it cares only about the single worst-off person and ignores everyone else, while Harsanyi's averaging ignores distribution entirely (rating an unequal society as good as an equal one with the same total); Buchak's risk-avoidant decision theory yields an intermediate position that gives most weight to the worst-off, significant weight to the middle, and a little to the best-off, matching how many people actually think about distributing resources and being sensitive to both total wealth and its distribution.
“My view allows for an intermediate position between these two guys because I think in the original position, if I don't know which member of society I'm going to be, I should use a risk avoidant decision theory. I should give more weight to worst case scenarios, but not exclusive weight to worst case scenarios.”
The standard arguments for expected utility maximization fail; repeated-decisions, averaging, and consistency arguments only establish weaker principles.
- On the standard theory that Buchak argues against, once you have a person's utilities and probabilities, you have all the information you need: the rational choice is to maximize expected (average) utility, multiplying each outcome's utility by its probability and summing.
“according to the standard theory, the one that I argue against in my book, that's actually all the information we need. Once we have your utilities and your probabilities, we can figure out exactly what to do. So on the standard theory, you should maximize expected utility or average utility.”
- The justification for maximizing expected utility by appeal to repeated trials does not apply to one-shot decisions: in poker many hands let losses be made up by gains on average, but for a single gamble or major life choice like which job to take or who to marry, you cannot reason that repeating it many times would yield a better average, so the averaging argument fails.
“if I'm just going to make a choice once, like these gambles... we can't employ the reasoning that says, look, if this were to be repeated many times, I'd do better on average by taking this thing because it's not going to be repeated many times.”
- The standard arguments in favor of expected utility maximization do not succeed in establishing it; each one only establishes some weaker principle, as with the repeated-decisions argument that fails because we do not always repeat decisions, so the case against Buchak's view largely rests on arguments of that flawed kind.
“there are these popular arguments in favor of expected utility maximization. And I try to show none of them work. They just end up with some weaker principle.”
Utility simply means whatever you care about; imperfectly rational people have usable utility functions derived from their consistent preferences.
- Even imperfectly rational people have utility functions because many of their central preferences are consistent and accessible; we determine the utility function from that reliable subset, discard inconsistent ones, and then use it to advise on more complex decisions like playing poker or investing where people lack direct intuitions.
“often, many of your preferences, or many of your central preferences are pretty consistent... we can figure out what your more complicated preferences should be from them... I can then use your utility function when I advise you how to play poker, or how to invest in the stock market”
- The word utility in decision theory means just whatever you care about, so you maximize your own personal utility even if you are perfectly altruistic; an altruist simply has a utility function that assigns higher utility to things good for others, which is distinct from utilitarianism's aggregation across people.
“the word utility in decision theory has come to mean just whatever it is you care about. So you can still be maximizing your utility, your own personal utility, even if you're perfectly altruistic.”
- There are two distinct moves in utility theory: the relatively innocuous one that a number reverse-engineered from preferences captures something intuitive (how much you like money depends on how much you already have), and a less innocuous one that utility represents nothing in the head or world but is merely a representation of preferences, which forces the claim that people are bad at knowing their own utility functions.
“The second number, the second move that people often make is something like, and utility doesn't describe anything in the head or in the world. It's simply a representation of your preferences. So it's meaningless in itself.”
- Even though people lack the precise utility and probability numbers the theory uses, decision theory is useful for broad-brush lessons; for example, the theorem that an expected utility maximizer always prefers to make decisions on the basis of more information is a simple insight that requires complicated math to discover.
“If you are an expected utility maximizer, then you're always going to want to make decisions on the basis of more information rather than less. Like that's a pretty simple principle that we have to do this complicated math to discover.”
- Utilities are typically determined by reverse-engineering them from a person's preferences over many options; for instance, observing many preferences over coin flips between dollar amounts can reveal whether someone's utility function is linear in money.
“The way that we typically do that is to reverse engineer them from your preferences. So I know lots of your preferences over a lot of different things. From those preferences, maybe I discover that your utility function is linear in money.”
Risk-averse decision makers can rationally decline information that shifts certainty toward a riskier state they would not then accept.
- More information is not always directly about what is true in your decision; in some cases acquiring information will talk you out of taking a risk that would have been a good risk to take, which is why a risk-avoidant person can rationally do worse with more information, whereas information that directly settles the truth (like the coin's outcome) is always good.
“there's going to be some cases in which information will talk you out of taking a risk that would have been a good risk to take. And those are the examples in which the risk avoidant person is not going to do better with more information.”
- Unlike expected utility maximizers, who always prefer to decide on more information rather than less, risk-weighted expected utility maximizers will sometimes prefer to decide with the information they have; specifically, when you already have a high probability that your course of action will work, acquiring information that might shift you toward a 50-50 state can talk you out of a risk that would have been good to take, so the risk-avoidant person rationally declines that information.
“there's going to be some cases in which information will talk you out of taking a risk that would have been a good risk to take. And those are the examples in which the risk avoidant person is not going to do better with more information.”
Different risk attitudes explain COVID disagreements and life choices without assuming people have fundamentally different values or irrationality.
- During COVID, disagreements over restrictions like dining out led people to assume opponents had different values (e.g. not caring about grandparents, or not caring about friendship and livelihoods), but if you recognize that people can have different risk attitudes that do not show up in the utility function, you see they often shared values and were both being rational; one group simply weighed worse scenarios much more heavily than the other.
“I think both people are behaving fully rationally.”
- A practical upshot of the theory is that more is up to you than expected: you must decide whether to be the kind of person who does pretty well no matter what or the kind who goes for outcomes that might turn out amazing or horrible; there is no right or wrong way to set this risk attitude, so both the all-in romantic and the cautious person are doing something fine when choosing a job, partner, or place to live.
“that's going to be a feature of you that you have to set. And there's like no right and wrong way to set it. The romantic that's like, as long as there's a chance, I want to go for it, is doing something fine. And then the sort of cautious person... is also doing something fine.”
- Nothing in the formalism prevents being risk-seeking: a person may rationally prefer a 50-50 chance of $100 over a sure $50, and you can prefer anything you want as long as you're consistent. Empirically, people tend to be risk-averse when buying insurance (paying more than the actuarial cost) and risk-seeking when buying lottery tickets (paying more than expected value).
“people tend to be risk-averse when it comes to buying insurance... you'll pay more for that insurance than is one in 100 the cost of your house. And people tend to be risk-seeking when buying lottery tickets.”
Quantum many-worlds interpretation raises the possibility that risk attitudes carry moral implications for actual outcome distribution.
- If the many worlds interpretation of quantum mechanics is true so that all possible outcomes are actual, then a decision made via a quantum random number generator becomes a problem of distributive justice across actual people, and one's attitude toward weighting worst-off outcomes (risk avoidance) could change ethical conclusions, suggesting many worlds may carry moral implications after all.
“if you think all possible Seans are actual Seans, that's the view, right?... Then actually you're facing the problem of distributed justice... If you think the latter thing and you have the physics interpretation you just explained, then maybe you should be risk avoidant.”
- There is a joke that people study the thing they are bad at; Buchak says she is terrible at making quick decisions, which is partly why she pursued decision theory.
“there's this joke that people study the thing they're bad at. So, you know, other people knew how to make decisions, so they didn't have to study it.”