From time to time, Andrew Gelman reflects on the unabated stream of published academic research marred by statistics errors. In his most recent such post1, Gelman relays a message he received from Michael Nelson2, then adds a few thoughts of his own.
Some of Nelson’s observations resonate with me:
You often say that statistics is hard. I suspect many social scientists would read that statement and think, “Yep, it takes real expertise like I got in grad school. Sure, I only took two or three stats courses, but my advisor and older labmates were able to show me exactly how to do statistics in our field.
LOL! And all too true. Sadly, this perpetuates a pathological equilibrium in which journal referees in these disciplines reject research employing sound statistical methods because it doesn’t comply with the weak, even erroneous, statistical methods they were taught.
I fear that they don’t take us seriously as experts because we literally teach them not to. I think there’s a strong argument to be made that we (statisticians) routinely teach statistics to future social scientists as a tidy package of simple techniques, and we strongly imply that that’s all they need to know. […] the professors in other departments expect us to include the methods they use (NHST3) with only the briefest of caveats. […] We never teach them that they aren’t experts. Then we get mad when they ignore our expertise!
But I find Nelson off the mark when he writes:4
It seems to me that mathematicians are the principal gatekeepers for mathematics …
Gelman responds:
. . . I’m not sure! I feel queasy about proposing that anyone be the gatekeeper of anything.
… lots of well-credentialed statistics experts make basic statistical errors. The foundations of statistics are a mess, and over and over again I’ve seen big-shot statistics professors making what I consider fundamental misunderstandings.
… the whole field is just less codified [compared to Math].
Again, all too true.
In Statistical Rethinking Winter 2019 Lecture 01 Richard McElreath claims that (30:53 to 31:04)
It [Bayesian Data Analysis] used to be really controversial. … In Stats departments this controversy is basically over. Stats departments the world over are essentially Bayesian now.
I haven’t observed Statistics departments first hand5, so perhaps one of you can let me know if McElreath’s report is accurate. If it is, then why does Statistics remain so fractured that eminent professors exhibit “fundamental misunderstandings”? Are the key conflicts among the 46656 Varieties of Bayesians sufficient to prevent codification despite McElreath’s report of Bayesian ascendancy?
I find Gelman’s post from the previous day relevant. Gelman posits that science is cumulative in areas where the researchers “do research for themselves” and are “using [published work] in their own research”.
Expanding on this, one weakness of peer review is its gatekeeping role.6
Instead, peer adoption is one of the signs of productive science. Of course, peer adoption in isolation could just indicate clique entrenchment. So we need to figure out what else is required to foster progressive research programmes.
Heh, now a year ago!
Presumably, one of Gelman’s blog readers.
Null Hypothesis Significance Testing.
Last month (February 2022), Jay Daigle wrote a much better explanation of why there’s no replication crisis in Mathematics:
in math, the paper, itself, is the “real work” […]
And that means that you can replicate a math paper by reading it.
I did have a statistician on my Ph.D. dissertation committee, but that’s my sole exposure in academia.
The positive role of peer review, not always employed, is to strengthen the research under review.