reproducibility
The ability to obtain the same results from a study or computation; in open source it is a function of being able to execute the published source code.
As defined in Balaji Srinivasan
Where it comes up · 6
- Balaji Srinivasan1 claim
- Conversations with Tyler - Jared Diamond1 claim
- Brian Nosek on Truth, Science, and Academic Incentives1 claim
- James Heckman on Facts, Evidence, and the State of Econometrics1 claim
- Pedro Domingos on Machine Learning and the Master Algorithm1 claim
- 301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability1 claim
Most important claims · 6 of 6Search all claims for “reproducibility” →
- Business school professors study leadership largely through sports coaching because sports are reproducible: teams play weekly under the same rules, with many teams in one league, whereas business outcomes are far harder to replicate and compare.Conversations with Tyler - Jared Diamond
- Despite many reasons to worry that the research literature is biased, there is little empirical evidence in economics, psychology, or any discipline estimating how reproducible the science actually is—we have no clear idea how biased the literature is, which is the most surprising gap.Brian Nosek on Truth, Science, and Academic Incentives
- Because of hype, engineers often fail to state their assumptions and technical limitations, producing not merely a replicability problem but a reproducibility problem—an inability to even rerun the same code on the same training data with the same folds—which is a very low bar that nonetheless frequently fails.301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
- Academia's incentives are inverted relative to open source: in open source bugs are expected, everything is transparent and published first then reviewed, and reproducibility comes from executable source code, whereas in academia retractions are career-damaging blemishes and many papers are not transparent in methods or data—so academia should move toward an open-source-like culture, which can only happen when grant incentives are realigned.Balaji Srinivasan
- The cure for medical reversal and treatments deployed on little evidence is to be more evidence-based: instead of basing decisions on one small study, machine learning lets us continuously gather feedback from treatment results as they roll out, dropping things that fail at scale and identifying the specific 5% of patients for whom a treatment actually works.Pedro Domingos on Machine Learning and the Master Algorithm
- Milton Friedman anticipated the credibility/reproducibility concern early: on hearing about Heckman's computationally complex work, he bluntly remarked it 'lacks a room for fraud,' pointing to the lack of reproducibility in such methods—an insight Heckman recognized as correct.James Heckman on Facts, Evidence, and the State of Econometrics