
Russ Roberts (and Robin Hanson) on Truth and Economics 01/26/2009
What this covers
This EconTalk episode has Russ Roberts reflecting on what years of interviewing economists have done to his trust in empirical work, with Robin Hanson of George Mason University as counterpoint. Roberts suspects that sophisticated studies almost never change anyone's mind. Micro-econometric work rarely moves economists, macroeconomics remains a field of competing theories, and explanations of one-time events like the Great Depression or the financial crisis are narratives fitted after the fact by both sides. He credits Friedman and Schwartz as decisive because they looked at plain facts such as money growth and inflation rather than running elaborate regressions.
The minimum wage serves as the main case study. Roberts argues that a single study like David Card's doesn't outweigh dozens pointing the other way, and that regression can't isolate a policy touching about 3 percent of workers amid many other changes. He also disputes Brad DeLong's claim that Fannie, Freddie and the Community Reinvestment Act predate the crisis too far to matter, pointing to the sharp expansion of their mission around 1995.
Hanson supplies the explanations. In his favored account, economists know a lot, but incentives such as jobs, funding and public approval bias which opinions reach the public, and selection effects silence the unsure majority. He argues that people leak insincerity, so biased experts must sincerely believe. He proposes prediction markets as a fix and is pessimistic that arguments can change group-driven beliefs. The conversation also touches pragmatism and the grandmother test, the reputations of Hoover and George W. Bush, and why physicists lose respect on policy questions.
Roberts and Hanson argue that economists' empirical disputes persist because incentives, selection effects, and motivated reasoning decide which evidence gets heard, so sophisticated statistics rarely settle questions and economics is more useful as a framework for thinking than as a predictive science.
- Few empirical studies in economics have changed expert consensus, and micro-econometric regressions rarely overturn established views
- Regression analysis cannot credibly isolate small policy effects amid many confounding changes, as with the minimum wage
- Public debate selects the most confident advocates, and incentives reward believing and arguing for one's side
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Pragmatism's critique of the Cartesian strategy of throwing out all beliefs and keeping only the right ones is that people cannot analyze their beliefs objectively, so they keep convenient or profitable ones; beliefs must be judged by the competitive process that has tested them, like floorboards of a boat.
“the pragmatist argue that that was a fallacy that was a misleading strategy why because it's very hard to to analyze your beliefs objectively”
Individual voters have little incentive to evaluate policy carefully because their vote rarely affects the outcome; their main payoff from political opinions is looking informed and showing allegiance to their group.
“I don't affect the outcome very much anyway all main thing I can do is look like I'm informed look like I'm with it”
Most of one's reasoning is unconscious: when a person reasons from A to B, the mind supplies only an outline of the process and omits most of the detail, so confidence in one's reasons is unwarranted.
“most of our reasoning is unconscious to us so that's one thing right there to realize”
A third theory explains expert disagreement: economists know a lot, but the process by which opinions reach public attention is biased by incentives, so the expert testimony that surfaces in high-stakes disputes such as antitrust trials is not the consensus view.
“a third theory is we know a lot about a lot of things but the process by which these opinions come to a public attention is biased by the incentives of that process”
Humans leak insincerity, so the only way people can say what is not true and get away with it is by believing what is not true, which makes sincere conviction the main mechanism of biased expert opinion.
“we humans leak our in sincerity we would we leak our in sincerity it's hard to be a good actor”
The grandmother test shows that a person who cannot give a reason for a social norm may still be right, because social norms can be the product of a competitive process that rejected other ideas, so cultural norms are not mere superstition.
“your grandmother is right you know you may think it's okay to do X Y or Z and your grandmother will say no no that's just wrong”
Public debates overstate disagreement because selection effects put forward the most confident, stark advocates, while the roughly 75 percent of reasonable people on each side who are unsure stay silent and are not invited to talk shows or debates.
“maybe 75% of each side is a reasonable person who once they look at it saying well I guess I'd I don't really know”
Economics teaching should repeatedly show students that confidently true but counterintuitive lessons exist, and also show that people who overreact to the consensus being wrong often hold a sea of crazy views, so averaging opinions can be better than any single contrarian view.
“you have to show them both both sides of that both that often the consensus is just terribly wrong”
Arguments for local production fail with audiences because people hold views for emotional and social reasons, not because they think them true, so changing hearts and minds toward discerning views is quixotic.
“they hold the views they hold partly for emotional and social reasons not because they think they're true”
Economists should be cautious about the minimum wage evidence, because a single study like Card and Krueger's showing employment gains does not outweigh dozens of studies showing harm, but the bigger problem is that any one study is an attempt to isolate a single policy from many simultaneous changes.
“maybe actually that all of them are kind of flawed and that you know we're trying to tease out the effect of one social policy in the midst of dozens and dozens hundreds and thousands of other changes”
Explanations of a single one-time event like the financial crisis or the Great Depression are ex post narratives created by both sides to fit the facts, so they are not a good place to do science or generalize about what will happen under similar policies.
“both sides the pro government and the pro market cider both of us are trying to create an ex post narrative to explain a single event”
Physicists lose public respect when they talk about policy-relevant topics such as global warming or nuclear winter, even though they are respected when discussing technical matters like string theory dimensions, because public debate becomes political and money is at stake.
“people really respect physicists say except when physicists start talking about something public cares about like global warming”
Staking out a position raises the chance of getting a job in an administration, and funding from interested parties pushes economists toward particular conclusions, so the more interest at stake, the less honest analysis of theory and data is what emerges.
“by staking out a particular position you raise the chance that you'll get a job in an inserted ministration”
Sloppy statistical studies win in public debate because the ultimate customer, the public, is not discerning and does not scrutinize persuasive statistics, so faulty incentives in academia and foundations ultimately trace back to what influences public opinion and policy.
“the reason why I say sloppy statistical studies win is because the customer isn't very discerning”
Regression analysis cannot credibly isolate the employment effect of a minimum wage that applies to about 3 percent of the population when the outcome measured is the overall unemployment rate affected by many other factors, because controlling for enough variables is an unverifiable assumption.
“you have a Social Policy that applies to about 3% of the population you're going to be using the overall unemployment rate”
Empirical data are unreliable because it is too easy to make up data that fits what one wants, and this is true of all sides, while theory is often more powerful than data but fails to persuade wider audiences.
“data is not very reliable it's too easy to make up data that fits whatever you want that's true of all sides theory is more powerful often than data”
Prediction markets reward people for forecasting carefully and staying silent when they do not know, because bettors who are wrong lose money, making them a more reliable source of advice than talking heads and prestigious publications.
“those people have an incentive to think carefully about it and shut up if they don't know because if they bet wrong they'll lose money”
Economists often claim their side has better empirical evidence, but the speaker suspects most sophisticated empirical studies do not actually change anyone's mind, since people accept only the studies that agree with their existing views.
“why do we accept some empirical work and not others”
Hoover's reputation as a do-nothing free-market ideologue is wrong, since he did much government intervention, and similarly George W. Bush is labeled a free-market ideologue despite a huge increase in government spending, showing that blame for failures attaches to whoever is stuck with the outcome.
“President Hoover has this reputation of being a do-nothing free-market ideologue it's not the case but it's interesting”
The root problem is that individuals lack sufficient incentives to be honest, so asking people to believe the truth against contrary incentives is demanding, and prediction markets are a mechanism that rewards people with money for accurate forecasts and penalizes them for bad bets.
“a betting market would be a more reliable source”
Brad DeLong's critique that government-driven explanations of the crisis fail because Fannie, Freddie, and the Community Reinvestment Act all existed long before 2008 is weak, because Fannie and Freddie's affordable-housing mission expanded sharply around 1995.
“Fannie and Freddie's mission did change a lot around 1995 which is there was a starting to be a big push for them to get into affordable housing”
Friedman and Schwartz's Monetary History of the United States was decisive because it relied on looking at facts such as money supply growth and measured inflation rather than sophisticated regression analysis, and it forced opponents to confront monetary explanations of inflation.
“it's Friedman and Schwartz monetary history the United States that study to me was the decisive piece of scholarship”
Why people accept some empirical work and reject other work is the key question: people judge studies by whether they confirm their side, and a logical argument is often judged inferior to an impressive-looking study with a big appendix of data analysis.
“the logic argument is often I think inferior in the eye of the listener ear of the listener to the oh I've got a study”
Micro-econometric studies almost never change economists' minds, whereas macroeconomics remains a field of competing theories, so the agreement that exists in microeconomics is not produced by empirical studies.
“I think it's very difficult to find examples where micro economic econometrics studies change people's minds”
Public opinion polls mostly measure people making up answers: people answer as if they have opinions and are not strongly committed, and the correlation between answers to the same question three months apart is very low.
“you can ask people the same question three months later and the correlation is really low it looks like honestly people just make up”
Larry White's explanation of the financial crisis (government guarantees of Fannie and Freddie, bad tax policy, Greenspan's low rates, and the Community Reinvestment Act) is a plausible ex post story, but it is shaped by the Austrian free-market perspective of its author.
“the government's guarantee of Fannie and Freddie pushing up the demand for housing bad tax policy”
A very large stimulus package (Krugman's proposed trillion dollars) is not supported by much empirical evidence, and Roberts's logical objection is that government spending must come from lenders or bond buyers, so it is deficit-financed and unlikely to raise total demand as Keynesians claim.
“the government spending that money is going to have to take it away from probably lenders people who are going to buy government bonds”
Prediction markets face a basic customer problem: if listeners want opinions that signal social membership, they will not care whether the opinion is true, so betting markets are unlikely to be used by the people who most need them.
“if what I wanted to do was just pick an opinion that had good social support so that I could be with the dominant side”
Economic logic strongly suggests Walmart does not lower wages because its entry increases labor demand, yet an author's study found Walmart lowered wages in small towns and not in large metropolitan areas, a result that is harder to explain given competition effects, and the author dismissed the neoclassical logic.
“I think there's a strong logical argument that says that Walmart doesn't lower wages”
Economists who disagree with peers who share their conclusions should recognize that the same forces that might lead those peers to motivated reasoning may apply to themselves, and the question is whether they have evidence they are more objective.
“you have to ask yourself why do I think I'm different is there some sign that I have”
Roberts now considers his earlier debate-era confidence in economic logic suspect, and he suggests the best one can do is recognize one's own likely biases and downgrade certainty about compelling results when reasoning in a one-sided mode.
“there's the mode I get into what I'm just thinking something through as an as a one-sided you know person”
Sophisticated empirical work (two-stage least squares, instrumental variables, complex econometrics) is fundamentally not scientific but a form of scientism that gives the gloss of science, which Roberts calls faith-based econometrics.
“sophisticated empirical work sophisticated statistical and econometric analysis is fundamentally I'm starting to think at least is fundamentally not scientific but scientist a form of somewhat Hayek called scientism”
Weak signs of honest, principled truth-seeking include being willing to reject one's side's conclusions, refusing to endorse something that looks wrong despite it favoring one's side, and declining to change a conclusion for a journal editor; these signs raise but do not establish confidence in one's objectivity.
“if somebody asked you to endorse something that you said you know that's on my side but it doesn't look right that's a red flag”
Brad DeLong's diagnosis that Larry White's explanation is shaped by an Austrian tradition baseline assumption that all evils have their origin in government is a fair critique in principle, but DeLong applies the same reasoning to his own preferred explanations, so ideological bias runs both ways.
“one could make the same critique Brad DeLong and I revenues Brad I log use Joseph Stiglitz”
The output of economic debate is mainly determined by what went in: people's initial predispositions produce their conclusions, so the process is not a reliable guide to advice.
“what's coming out is mainly determined by what went in people have initial predispositions and that's what's coming out”
In his debate with Bill McKibben on buying local, Roberts's logical arguments were less persuasive to the audience than McKibben's citation of empirical studies, even though Roberts believed his logic was decisive.
“I found that my argument against his argument of invoking those studies was not very effective I think it's right but it's not effective”
Honest people should first examine their own honesty, asking how much they are willing to sacrifice to say 'I don't know' and to accept evidence against their preconceptions, and then look for others who care about honesty to work with.
“first of all with ourselves with our own honesty we have to ask how important is honesty to us even if nobody listens”
Roberts found that the public perception of expert disagreement is misleading because selection effects mean the most confident voices get the airtime, so the exaggerated disagreement in media is larger than among thoughtful ideologues, but persistent one-sided disagreement among ideologues still signals that something is wrong.
“the level of confident disagreement that you see presented in the media could be an exaggeration”
Economists may be underrated, because the parts of economics they agree on are substantial and useful, and disagreement is concentrated where the field has real open questions, so the visible argument makes the field look less complete than it is.
“it looks half ready but it's really eight seven eighths full right you know be thankful that we know some things”
Roberts once came to the financial crisis as an ideologue assuming government caused it, but investigating Fannie and Freddie's role convinced him it was substantial, while he now believes the Community Reinvestment Act was not the source of the problem.
“I came at it initially as an ideologue I know anything about I knew virtually nothing about governments involved in housing”
It is easy to concede human frailty and uncertainty in the abstract, but when a specific compelling data argument appears, such as local production, LoJack, or the financial collapse, the general skepticism disappears from view.
“when we get onto something concrete and specific like local production or the financial collapse or LoJack”
Counting studies is a weak argument against the minimum wage: Roberts's dozens of studies showing job losses do not outweigh a single contrary study such as Card and Krueger, because the number of studies is not a good measure of evidence, and each study can be criticized on its flaws.
“that's not really a very good argument the fact that I have more of them now”
Believing that one sees the entire picture of a topic is an illusion; other people can be aware of considerations one has not considered, so one should give others' reasoning credit and remain uncertain.
“this illusion that somehow you're seeing the entire picture is crazy it's you're almost always wrong”
The main advantage of economics is understanding what not to do and providing a language and framework for thinking rigorously about complex matters, not providing a reliable science of predicting outcomes.
“the main advantage of economics is to understand what not to do but in a general sense I'm coming to believe that what economics is most useful for”
A person can manage the tension between advocacy and skepticism by alternating between a one-sided advocate mode that builds a case and a corrected mode that steps back and downgrades certainty, asking how the same mistake would look in someone else's work.
“there's the mode I get into what I'm just thinking something through as a one-sided person with a point of view”
Graduate programs reward theoretical brilliance and clever empirical work, and graduate students must go native to assimilate the academic culture, so they learn to adopt jargon and sophisticated econometrics regardless of whether those tools are reliable.
“the profession rewards theoretical dazzling theoretical brilliance and clever and creative empirical work”
Benjamin Franklin's line that when fortresses and virgins get to talking the end is in sight shows that people under pressure find an after-the-fact reason to yield, so the reasons they give for conceding are not reliable.
“when fortresses and virgins get to talking the end is in sight”
Economists largely agree on some things, such as there being no risk-free debt guaranteed prudently, and the view that the economy is a house of cards built on debt is an overreaction that confuses imprudent debt with debt itself.
“the problem is is imprudent debt either lent imprudently or borrowed and prudently guaranteed yeah worse yeah prudently guaranteed so it's risk-free risk-free debt”
Economists broadly agree on microeconomic policy such as opposing tariffs and the minimum wage, yet they disagree most on macro and on pro- versus anti-intervention policies, and claims that one's data are better than an opponent's are a common way to justify policy views.
“the agreements bigger than you think in the area of microeconomics there's lots of agreement”
States with high minimum wages having low unemployment does not refute the minimum wage view, because of simultaneity: states with low unemployment may be the ones able to afford higher minimum wages, and many other state differences also exist.
“the states with low unemployment are the ones that pass the higher minimum wages because they can afford it”
Economists do know some things non-economists do not: students' answers on exams show markedly more coherent understanding than amateurs, and economists agree on things such as that there is no truly risk-free way to make money from debt.
“there's no such thing as risk-free okay so if we understand some things”
Academic and expert claims that their side's studies are better are a common justification for policy disagreements, and when pressed, such claims often reduce to asserting that one's own evidence is superior.
“that's a common view that people use to justify their own policy views they say well you know I don't disagree with you but I have the evidence”
A person who cannot articulate reasons for a social convention is still disturbing evidence for reexamining it, but academics should still seek reasons for social conventions and be disturbed when they cannot find them.
“nevertheless academics should be focused on finding reasons for things and be disturbed by not finding reasons”
Roberts's Chicago training held that vigorous potential competition matters even for pricing, whereas other schools emphasize imperfect competition, monopolistic competition, and game-theoretic exploitation of consumers, and Roberts asks what evidence supports the imperfect-competition view beyond mathematical appeal.
“we tend to argue when I was taught there that competitions very vigorous potential competition is important even in affecting pricing”
Public debates among experts are unusually intense and personal compared with private settings, where experts are more nuanced and willing to concede; the public arena rewards strong positions.
“the intensity and animosity these sometimes personal animosity you see in these conversations and these debates publicly but in a different setting there's much more nuance”
Lott's study that concealed-handgun laws reduce crime is compelling, while Ayres's LoJack study showing the car-tracking device deters theft is also empirical; the two critics do not respect each other's work, and it is unclear which is right.
“John lot doesn't respect aires' study and Ayers does not respect John Lott study”
The pro-local-production studies cited in the University of Vermont debate were typically not peer-reviewed and were often done by consultants or localities seeking to justify buying local, so they are likely biased, yet Roberts's counter-argument against them was ineffective.
“those studies are typically not peer-review they're not published in academic journals but they are often sophisticated empirical studies are often done by consultants”
Most people, including thoughtful ones, find it hard to say 'I don't know' consistently, and people with opinions tied to their identity feel a need to hold ascendant positions; this produces overconfident public opinion.
“it's hard to consistently say I don't know it's much more fun to say I do know”
A mature academic should recognize that academia is not an ideal truth-generating system: a large volume of papers does not make a topic interesting, a theoretical model does not make a claim true, and a dataset does not make a result meaningful.
“just because there's a bunch of papers on something doesn't mean this is an interesting topic just because there's a way to do put make a theoretical model of it doesn't make it true”
Two hypotheses explain expert disagreement: economists know things that are not useful for real problems, or economists know little about the real problems and argue only where they lack knowledge.
“there's two basic hypotheses for what's going on in all these disputes you have one is the things we understand aren't very useful”
People are skeptical of empirical studies, knowing that statistics can be manipulated, but are not similarly skeptical of theory, even though all theories are simplifications by definition and should be scrutinized.
“they know that it's easy to lie with statistics but they're not more skeptical of theory unfortunately”
The Community Reinvestment Act was given real teeth in 1995 under the Clinton administration, creating a large incentive for banks to make loans to people who previously would not have qualified, though its magnitude as a cause of the crisis remains uncertain.
“in 1995 it was given teeth for the first time in a real sense by the Clinton administration giving banks a huge incentive”
When asked to name an empirical study that changed an economics consensus, Ian Ayres cited the work on abortion legalization reducing crime, which Roberts considers ironic because that study was widely rejected and did not produce consensus.
“the example Ian Ayres gave was the work on abortion legalization reducing crime an ironic choice in my opinion because I think that studies been roundly rejected”
Researchers who say they have looked at the data and therefore need no theory make a claim that is hard to sustain, because one cannot credibly hold constant all the factors needed to evaluate a case like Walmart's wage effects.
“do you really think you've been able to hold constant the other things you need to hold constant”
The current economy needs trust and optimism to recover, and economics has little to say about how to create trust or optimism.
“what is needed to get the economy going again is trust yeah a sense of optimism I don't think economics as much to say about how to create trust”
DeLong's critique of Larry White is an instance of the Austrian bias that assumes all evils originate in government regulation, which DeLong contrasts with the more fruitful road of studying how human psychological limits lead to bad private-sector contract design.
“the baseline assumption that all evils must have their origin in some form of government miss regulation”
Tying congressional or presidential compensation to future unemployment would give direct financial stakes, but accountability is muddied because it depends on contested interpretations of what caused what.
“congressional compensation should be based on the unemployment rate twelve months from now”
When academics engage topics that matter to outsiders, the public's deference to them falls sharply unless the academics say what the audience wants to hear.
“as soon as we academics start talking about something that matters to people outside academia then are their level of deference to us goes way down unless we're saying what they want to hear”
Roberts is forced to ask whether his belief against the minimum wage rests on evidence or only on the logic of the case, which suggests that logical arguments are often used to hide weak empirical grounds.
“again it's my belief against the minimum wage really based on the evidence or is it just the logic of the case”
Hosting guests with opposing views has made Roberts less interested in hammering them; the show's aim is an intellectual exchange in which listeners and host learn, and he has become more receptive to alternative views.
“I've gotten less interested in hammering people and I really I view this show as a chance for you the listener and for me to learn”
Robert Frank's economic explanations of everyday life are persuasive in some cases and questionable in others, and Roberts's disagreement with Frank on competition shows that competition-based and imperfect-competition-based explanations lead to different conclusions about which explanations are compelling.
“as I read the book and I realized there are many of the explanations in the book that I found compelling and there are many that I thought were silly”
Roberts's experience hosting the show over time has changed his views on economics and on the reliability of expertise, as much of his change came from interviewing many interesting people.
“most of that change has come i from being the host of this show and talking to a lot of interesting people”