P-hacking, which he discusses in the story, is definitely a thing. My grad school statistics professor called it "fishing." Basically, it's what happens when you run multiple statistical analyses on results, looking for something significant. My dissertation director joked about doing this (not publishing) with some data on Alcatraz inmates; the only significant relationship they found was that Alcatraz inmates were significantly more likely to be Capricorns. She then looked at me very seriously, and asked, "You're not a Capricorn, right?"
Yes, I am.
Statistical results are probabilistic; we look for results that have a low chance of happening if no real relationship exists. We usually set that value at 5%. What that means is, if I run 20 tests, one those will probably be significant by chance alone. That's less of a concern if I have pre-existing (a priori) hypotheses, based on past research and/or theory, I'm testing but even if I am testing a priori hypotheses, I should apply a correction to account for the number of tests I'm running.
The problem with p-hacking is that, not only does it involve running many tests, it also usually involves only reporting the significant results. So a reader would have no idea that a person ran potentially dozens of tests based on reading the article. Unfortunately, this is one of the negative consequences of the "publish or perish" mentality. Scientists feel so much pressure to come up with results, that they'll do things they know are questionable in order to meet their publication quotas for tenure and/or funding. And that problem compounds when journals reject articles that replicate past studies. As John Oliver says in the story, "There's no Nobel prize for fact-checking."
No comments:
Post a Comment