weighting
Statistical adjustment of poll responses to match a target population—e.g., counting an underrepresented group's respondents more heavily.
As defined in 389 - The Politics Of Risk
Where it comes up · 2
- Doug Rivers on polling 7/21/200811 claims
- 389 - The Politics Of Risk2 claims
Most important claims · 10 of 13Search all claims for “weighting” →
- Of about 1,300 polls Rivers examined from the 2008 presidential primaries, fewer than 50 reported anything other than the margin of error for a simple random sample with no weighting—meaning the reported margins of error are systematically misleading, which Rivers calls scandalous.Doug Rivers on polling 7/21/2008
- Pollsters typically select 10 to 20 times as many phone numbers as the number of completed interviews they want, meaning the people who actually respond are a small, non-random slice that skews toward more women, higher education, and higher income, requiring weighting and adjustment.Doug Rivers on polling 7/21/2008
- Weighting creates two distinct problems: it roughly doubles the real variability compared to a simple random sample, and it leaves residual skews (unobserved or uncorrected biases) that do NOT shrink as sample size grows, so conventional sampling-error estimates tell you how the procedure varies sample-to-sample but not how systematically wrong the procedure is.Doug Rivers on polling 7/21/2008
- Weighting underrepresented groups by large factors (5-10x for some groups) drastically inflates variability: with a weight of 10 in a sample of 1,000, a single miscoded respondent shifts the whole sample by 1%, so a single error can destroy a nominal 3% margin of error.Doug Rivers on polling 7/21/2008
- Massive consumer and voter databases that did not exist 25 years ago now provide detailed information (income, home value, etc.) on most people, allowing pollsters to draw a randomly selected target sample from a voter list and then match closely-similar respondents from a large opt-in internet panel, creating a sample that mimics a random sample across many dimensions and removes skews that simple demographic weighting cannot.Doug Rivers on polling 7/21/2008
- Even the best telephone surveys get only 3-5% response rates, so poll respondents are systematically weird—skewed toward the politically engaged—which pollsters correct with aggressive weighting (e.g., counting a Black respondent twice and an old white woman half) introducing more subjectivity, and non-response bias driven by low civic trust is a hard problem to overcome.389 - The Politics Of Risk
- Modern polling is not as simple as calling people and recording answers because most people don't answer polls, requiring extensive weighting, modeling, and data massaging that inserts human judgment—where the pollster's political views, sense of narrative, or vibes can influence the result.389 - The Politics Of Risk
- Weighting pollsters by their historical accuracy (as fivethirtyeight.com does) is questionable because past success may be partly random luck and because pollsters who erred likely changed their methodology afterward, so historical outcomes don't reliably predict future accuracy—analogous to picking mutual funds by past manager returns.Doug Rivers on polling 7/21/2008
- Apparent poll movements late in a campaign are often illusory: a methodology shift (e.g. Gallup switching from registered-voter to likely-voter weighting) creates a bump unrelated to real opinion change, and pollsters who err have a ready ex-post story of a late undetected surge—what Rivers calls the last refuge of scoundrels in polling—when in fact the same poll repeated would have produced the same wrong answer.Doug Rivers on polling 7/21/2008
- Likely the New Hampshire 2008 polling miss came from over-representation of college- and graduate-degree voters (who favored Obama, over-represented by 2-3x and under-corrected) and from unreliable self-reports of voting intention, rather than from racism (the Bradley effect).Doug Rivers on polling 7/21/2008