Calibration
A macroeconomic practice of setting model parameters using values from various separate studies rather than estimating them jointly, criticized by Heckman for combining mutually inconsistent data.
As defined in et_509
Other definitions
The correct balancing and adjustment of one's thinking to match reality; avoiding overstatement or understatement of actual threats and facts
Peter Thiel On The Rise Of The Machines And Xi, The CCP, and SBF
Where it comes up · 13
- Wanna Bet?4 claims
- The Four Sources of Alpha2 claims
- John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI2 claims
- Guillermo Calvo on the Crisis, Money, and Macro2 claims
- Elon Musk And Neuralink Team2 claims
- Why Everyone Is Wrong About AI (Including You) | Benedict Evans1 claim
- Jim Collins — Good to Great — The Knowledge Project #671 claim
- Glenn Loury1 claim
and 5 more
Most important claims · 10 of 20Search all claims for “Calibration” →
- A well-calibrated forecasting system that says 75% will be wrong 25% of the time, so throwing out the system every time it lands on the wrong side of 'maybe'—as Leonhardt did with prediction markets after the Obamacare ruling—would leave you with no well-calibrated forecasting systems at all.Philip Tetlock on Superforecasting
- Powerful inertia in academia—graduate students must follow advisers and publish in good journals, and thinking is costly—keeps producing theoretical NBER papers whose key assumption is a perfectly functioning credit market, then applying them to policy debates, when that very assumption assumes away the central problem of the crisis.Guillermo Calvo on the Crisis, Money, and Macro
- When Brier scores are tracked and fed back to decision-makers such as meteorologists or medical professionals, they become better calibrated, showing that structured feedback improves probabilistic judgment.The Four Sources of Alpha
- Open loop calibration (where the user has no cursor control and follows a moving target by intention) is preferable to closed loop calibration partly because it is far easier to debug—there is no feedback cycle with the user in the loop—and a key heuristic for choosing which problem to tackle is selecting the one easiest to debug since a tighter iteration loop predicts success.Elon Musk And Neuralink Team
- Because a researcher can pick from many alternative assumption sets and calibrate them with enough degrees of freedom, successfully reproducing an observed phenomenon B gives no license to conclude the chosen assumptions are correct or to transport them to explain other phenomena.Paul Pfleiderer on the Misuse of Economic Models
- Calibration—taking a purely theoretical model, feeding in shocks and adjusting them until the model replicates observed data—creates a false sense of validity, because enough tinkering will eventually hit the mark without bringing the model closer to reality, and it absorbs effort into programming rather than thinking about the real world.Guillermo Calvo on the Crisis, Money, and Macro
- Models have some calibration about what they know—which is why they don't hallucinate that badly—and that same ability to know what they know could automatically give rise to active learning, the capacity to identify and seek out knowledge gaps.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Pre-training trains a model to imitate all internet content by maximizing log probability of the next token, which forces it to assign probabilities to everything and ends up making the model very calibrated, capable of both generating and assigning probabilities to all kinds of web content and personas.John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Discretionary managers should attach probabilities to thesis signposts and track them with Brier scores, which provides intermediate feedback that improves calibration over multi-year horizons when stock outcomes are noisy.The Four Sources of Alpha
- Musk's predictive apparatus was strikingly miscalibrated when he put the chance he was wrong about US COVID cases at one in a thousand, at a moment when Harris was so confident the US would exceed 30,000 cases within a month that he would have bet his children's lives on it—evidence that genius-level reputation does not guarantee sound calibration.427 - Ai Friends Enemies