normative

Generating and testing hypotheses on same data is invalid

You cannot confidently generate and test a hypothesis using the same data: digging into data to find moderating influences after seeing the result means the analysis is shaped by the data, so any pattern found must be confirmed with new data rather than treated as a valid test.

normativepending

Speaker

Brian Nosek

Evidence Quote

“we've now looked at the data, and the data itself has shades, how we analyze that data. So we are both simultaneously generating and testing the hypothesis, with the same data. And that's a no-no.”

Source

Brian Nosek on the Reproducibility Project— EconTalk
Created: 6/13/2026, 12:25:28 AM

Your notes

Loading notes...