To me, one of the most interesting things about the p-hacking controversy is that the entire history of p<=0.05 as the "once in a blue moon" standard for statistical significance, as Joseph Kaldane noted in a JASA article on forensic statistics back in the 90s, rests on absolutely no statistical theory whatsoever. It's a convention, simple heuristic and rule of thumb that started with R.A. Fisher and has since been reified or consecrated into its present "unquestioned" status. Bayesian or not, the time is long overdue to challenge this metric standard or at least give it the skepticism it deserves.

That said, my interpretation of Gelman's point is that, as is well known, the peer review process rewards positive statistical significance and punishes insignificant results by not publishing those papers. This is irrespective of whether or not publishing an insignificant finding would have potentially large impact on the thinking and theorizing for a given domain. Gelman, Simonshohn and others have repeatedly pointed to the abuse of the 0.05 significance level in peer-reviewed and published research by holding up examples of ridiculous, yet statistically significant findings in paranormal, social and psychological research. One of the most egregious was the statistically significant finding that pregnant women were more likely to wear red dresses. Gelman maintains that, in the absence of logical challenges to statistical results, the mere fact that an analysis is "statistically significant" is a potentially meaningless explanation. Here, he's referring to the industry's occupational hazard with overly technical and abstruse arguments that do little or nothing to advance a debate among a lay audience.

This is a point Gary King makes vehemently when he practically begs quantitative political scientists (and, by extension, all quants) to stop mechanistic, technical reportage such as "this result was significant at a p<=0.05 level" and moving towards more substantive interpretations. Here's a quote from a paper by him, "(1) convey numerically precise estimates of the quantities of greatest substantive interest, (2) include reasonable measures of uncertainty about those estimates, and (3) require little specialized knowledge to understand. The following simple statement satisfies our criteria: 'Other things being equal, an additional year of education would increase your annual income by 1,500 dollars on average, plus or minus about 500 dollars.' Any smart high school student would understand that sentence, no matter how sophisticated the statistical model and powerful the computers used to produce it."

King's point is very well taken and maps out the direction the debate needs to take.

Making the Most of Statistical Analyses: Improving Interpretation and Presentation, King, Tomz and Wittenberg, 2002, Am Jour of Poli Sci.

To me, one of the most interesting things about the p-hacking controversy is that the entire history of p<=0.05 as the "once in a blue moon" standard for statistical significance, as Joseph Kaldane noted in a JASA article on forensic statistics back in the 90s, rests on absolutely no statistical theory whatsoever. It's a convention, simple heuristic and rule of thumb that started with R.A. Fisher and has since been reified or consecrated into its present "unquestioned" status. Bayesian or not, the time is long overdue to challenge this metric standard or at least give it the skepticism it deserves.

That said, my interpretation of Gelman's point is that, as is well known, the peer review process rewards positive statistical significance and punishes insignificant results by not publishing those papers. This is irrespective of whether or not publishing an insignificant finding would have potentially large impact on the thinking and theorizing for a given domain. Gelman, Simonshohn and others have repeatedly pointed to the abuse of the 0.05 significance level in peer-reviewed and published research by holding up examples of ridiculous, yet statistically significant findings in paranormal, social and psychological research. One of the most egregious was the statistically significant finding that pregnant women were more likely to wear red dresses. Gelman maintains that, in the absence of logical challenges to statistical results, the mere fact that an analysis is "statistically significant" is a potentially meaningless explanation. Here, he's referring to the industry's occupational hazard with overly technical and abstruse arguments that do little or nothing to advance a debate among a lay audience.

This is a point Gary King makes vehemently when he practically begs quantitative political scientists (and, by extension, all quants) to stop mechanistic, technical reportage such as "this result was significant at a p<=0.05 level" and moving towards more substantive interpretations. Here's a quote from a paper by him,

(1) convey numerically precise estimates of the quantities of greatest substantive interest, (2) include reasonable measures of uncertainty about those estimates, and (3) require little specialized knowledge to understand. The following simple statement satisfies our criteria: 'Other things being equal, an additional year of education would increase your annual income by 1,500 dollars on average, plus or minus about 500 dollars.' Any smart high school student would understand that sentence, no matter how sophisticated the statistical model and powerful the computers used to produce it.

King's point is very well taken and maps out the direction the debate needs to take.

Making the Most of Statistical Analyses: Improving Interpretation and Presentation, King, Tomz and Wittenberg, 2002, Am Jour of Poli Sci.