by Kevin Lang argues in NBER wp 31666: When economists analyze a well-conducted RCT or natural experiment and find a statistically significant effect, they conclude the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? The answer depends on the proportion of tested nulls that are true and the power of the tests. I model the distribution of t-statistics in leading economics journals. Using my preferred model, 65% of narrowly rejected null hypotheses and 41% of all rejected null hypotheses with |t|
Topics:
Dirk Niepelt considers the following as important: Credibility, Notes, Power of test, Statistics, t-statistic, Validity
This could be interesting, too:
Dirk Niepelt writes SNB Annual Report
Dirk Niepelt writes Banks and Privacy, U.S. vs Canada
Dirk Niepelt writes Bank of England CBDC Academic Advisory Group
Dirk Niepelt writes Panel on “Will the digital euro take off?,” CEPR, 2023
Kevin Lang argues in NBER wp 31666:
When economists analyze a well-conducted RCT or natural experiment and find a statistically significant effect, they conclude the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? The answer depends on the proportion of tested nulls that are true and the power of the tests. I model the distribution of t-statistics in leading economics journals. Using my preferred model, 65% of narrowly rejected null hypotheses and 41% of all rejected null hypotheses with |t|<10 are likely to be false rejections. For the null to have only a .05 probability of being true requires a t of 5.48.