reliability
In measurement, how consistently a signal (e.g., scowling) co-occurs with a state (e.g., anger); Barrett cites ~30% as low reliability for scowls and anger.
As defined in 123 | Lisa Feldman Barrett on Emotions, Actions, and the Brain
Other definitions
In psychometrics, the consistency or repeatability of a measurement; a reliable test produces similar scores when administered multiple times; necessary but not sufficient condition for validity
2017 Personality 18: Biology & Traits: Openness/Intelligence/Creativity I
Where it comes up · 5
- No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever3 claims
- AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference2 claims
- 123 | Lisa Feldman Barrett on Emotions, Actions, and the Brain2 claims
- 2017 Personality 18: Biology & Traits: Openness/Intelligence/Creativity I1 claim
- Morten Jerven on Measuring African Poverty and Progress1 claim
Most important claims · 9 of 9Search all claims for “reliability” →
- GDP data should be assessed on two separate dimensions: 'validity' (whether the level estimate is correct, which in rich countries is off only a few percent but in sub-Saharan Africa can be plus-or-minus 50-100%) and 'reliability' (whether the methodology stays consistent over time); the bathroom-scale analogy shows that a consistently miscalibrated scale is fine for tracking change, but a scale someone secretly recalibrates in the night ruins time-series comparison, while comparing yourself to a neighbor using a different scale ruins cross-country comparison.Morten Jerven on Measuring African Poverty and Progress
- Data show people in Western (and urban Eastern) cultures scowl in anger only about 30% of the time—better than chance but meaning their face does something else 70% of the time—and people also scowl when confused, concentrating, hearing a bad joke, or having gas; so the scowl has low reliability and low specificity as an indicator of anger, which is just a Western stereotype of anger.123 | Lisa Feldman Barrett on Emotions, Actions, and the Brain
- Companies building emotion-recognition technology confuse the observation of a facial movement (e.g., detecting a scowl) with its emotional meaning; software can detect scowling but cannot tell you what it means, because reading a face requires high-dimensional spatial and temporal context including the observer's own internal body state—so deploying such tech for consequential decisions like school admission, criminal charges, or hiring is a serious error.123 | Lisa Feldman Barrett on Emotions, Actions, and the Brain
- The main bottleneck limiting current large language models' utility is not reasoning capability or knowledge breadth, but reliability—the degree to which a user can trust the model will consistently succeed on tasks similar in difficulty to ones it has already succeeded on.No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever
- Self-driving cars represent a positive model for AI development: they were first demonstrated two decades ago, progress has been steady but slow as developers incrementally scale deployment (from 1,000 to 10,000 to 100,000 miles) to improve reliability through real-world feedback, and this sector-by-sector learning from domain experts is the model that generative AI should follow.AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference
- Generative AI has genuine transformative potential and is useful to every knowledge worker, but current companies have failed to translate capabilities into reliable products because they tried a general-purpose prompting approach that requires reliability rates of 99.99% which current systems cannot achieve.AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference
- Reliability scales with model size: larger models are more reliable across broader task distributions than smaller models, making size and reliability directly correlated.No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever
- IQ tests are created by generating a large pool of questions that require knowledge, having diverse raters identify which questions measure intelligence, then taking random subsets and correlating performance across them; high reliability (r = 0.9+) across different item sets indicates a unified construct (g).2017 Personality 18: Biology & Traits: Openness/Intelligence/Creativity I
- The models' ability to provide legal advice at a high level of reliability may require significantly larger models than are currently used for general purposes, but the high value of reliable legal advice justifies the inference cost of larger models.No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever