156 | Catherine D'Ignazio on Data, Objectivity, and Bias
What this covers
Sean Carroll hosts a conversation with Catherine D'Ignazio, an MIT researcher in data science and urban planning, about how bias enters data and algorithms at every stage of their creation and use. Rather than treating data as a neutral mirror of reality, D'Ignazio argues that data is fundamentally a reduction of the world shaped by human choices—which variables to collect, how to classify them, which details matter—and that these choices inevitably encode the biases and blindnesses of those making them. The discussion moves through concrete examples: why facial recognition systems fail on dark-skinned faces, how car safety features harm women because crash test dummies modeled only average male bodies, and how zip codes function as a proxy variable that lets algorithms sort people by race without explicitly naming it. These are not cases of malicious intent but of Garbage In Garbage Out—systems trained on biased training data drawn from an unequal world.
Rather than abandon objectivity, D'Ignazio proposes what feminist theorists call strong objectivity and situated knowledge: achieving more reliable knowledge by including more people from more places, acknowledging historical context and power asymmetries, and breaking down who has been excluded from research and data collection. The conversation covers how root cause analysis must account for centuries of discrimination in lending and housing rather than treating algorithmic systems as race-blind starting from time-zero, how counter data—such as databases of femicide compiled to hold governments accountable—can be a tool for justice, and how mixed methods research pairing quantitative breadth with qualitative depth captures strengths both approaches alone miss. D'Ignazio also addresses the data is the new oil metaphor, noting that like oil extraction, the data economy generates large negative externalities borne by those least able to mobilize data for their own benefit.
D'Ignazio argues that data and algorithms are never neutral because every stage of their creation involves human choices that encode existing social biases, and that the remedy is not abandoning objectivity but adopting a feminist, situated, and historically-aware practice that broadens who is included and uses data itself to hold power accountable.
- Data is a reduction of the world shaped by human classification choices at every stage of the pipeline, so confusing data for reality breeds error.
- Algorithmic bias arises from biased training data drawn from an unequal world, not usually from malicious intent.
- A 'situated knowledge' / strong-objectivity approach that includes more people and accounts for history is both more honest and more useful than imagined neutrality.
Selective attention and framing make objective description of reality inherently impossible.
- Even attempting to describe your immediate environment in a completely objective way is impossible, because among the infinite number of objectively true things you could say, it is you—with your biases, history, and interests—who chooses which features matter and how to divide the world into interesting and uninteresting facts, which is itself how non-objectivity creeps into our characterizations.
“There're an infinite number of things you could say that are objectively true, but it's you who are always gonna be a little bit fallible and have your biases, have your history and your interests, and so forth, that choose for you what features of the environment matter”
Human bias embedded in data creation and algorithmic processes prevents objectivity in characterization.
- Neither the collection, analysis, nor use of data and algorithms is fully objective, because all of these processes are created by human beings who bring their foibles and biases, so non-objectivity inevitably creeps into how we characterize the world.
“it's often the case that neither the collection of data, nor the analysis of data, nor the use of algorithms are completely objective. They have biases built into them, because all of these facts about the world or ideas about how data should be analysed are created by human beings.”