The p-value of 0.04 used to feel like a victory. You ran the experiment, crunched the numbers, and there it was — just under the magic line. Publishable. In practice, significant. Done Which is the point..
But 2016 changed how a lot of us look at that number And that's really what it comes down to..
That year, Nature didn't just publish research. Think about it: a series of articles, commentaries, and editorials that forced the scientific community to stare at the p-value and ask: what does this actually mean? It published a reckoning. And more uncomfortably: what have we been getting wrong?
If you've ever stared at a p-value of 0.04 and felt a little uneasy — or if you've reviewed a paper where that number did all the heavy lifting — this is the backstory you need.
What Happened in 2016
The short version: the American Statistical Association (ASA) released a statement on p-values. Nature covered it heavily. And the conversation shifted from "is it significant?" to "what does significance even mean?
The ASA statement — published in The American Statistician but amplified across Nature's news and comment sections — was unprecedented. In real terms, for the first time, the largest professional organization of statisticians in the world said, publicly and plainly: **stop treating p < 0. 05 as a binary truth filter.
Nature ran multiple pieces that year. A news story titled "Statisticians issue warning over misuse of P values." A commentary from David Colquhoun arguing that p = 0.04 means you're still 26% likely to be wrong (under reasonable priors). An editorial urging journals to rethink statistical standards.
It wasn't a single article. It was a campaign.
And the p-value of 0.04 became a kind of mascot for the problem — close enough to 0.05 to feel "significant," far enough from zero to be deeply unstable.
Why 0.04 Is the Perfect Troublemaker
Let's be honest: nobody loses sleep over p = 0.04? Because of that, 06 gets rejected without a second glance. And p = 0.001. But 0.That's the danger zone.
It's the "just significant" result. The one that gets highlighted in the abstract. Which means the one that makes it into the press release. The one that feels like evidence Simple, but easy to overlook. That alone is useful..
But here's what 2016 made clear: a p-value of 0.04, in a typical underpowered study, can easily correspond to a false positive rate of 20–30% or higher.
That's not a typo. It's not a controversial take. It's basic probability Worth keeping that in mind. Still holds up..
The p-value tells you: if the null hypothesis is true, how likely is this data (or more extreme)? It does not tell you: how likely is the null hypothesis true given this data? That's a different question — one that requires prior probabilities, power, and a Bayesian framework Most people skip this — try not to..
Most researchers don't think in those terms. In 2016, Nature made sure more of us started to Small thing, real impact..
The Colquhoun Argument
David Colquhoun, a pharmacologist at UCL, wrote a blistering piece for Nature in 2016 (and a longer paper in Royal Society Open Science) showing that if you test a hypothesis with a prior probability of 10% — which is generous for many exploratory fields — a p-value of 0.04 gives you a false positive risk of about 26%.
Read that again. One in four.
And that's assuming no p-hacking, no selective reporting, no researcher degrees of freedom. In the real world? The false positive risk is almost certainly higher Simple, but easy to overlook..
This wasn't new math. But Nature gave it a platform. And suddenly, that comfortable 0.04 started looking a lot shakier Small thing, real impact..
The ASA's Six Principles
The ASA statement didn't just criticize. It offered guidance. Six principles, widely cited, widely ignored, and absolutely worth revisiting:
-
P-values can indicate how incompatible the data are with a specified statistical model. That's it. Not "the probability the hypothesis is true." Not "the probability the result is real." Just incompatibility with a model.
-
P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. This is the most common misinterpretation. Full stop Most people skip this — try not to..
-
Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. The 0.05 cliff is arbitrary. Always has been.
-
Proper inference requires full reporting and transparency. No cherry-picking. No hiding the three analyses that didn't work Simple, but easy to overlook..
-
A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. A tiny effect with a huge sample can yield p = 0.0001. A massive effect with a tiny sample can yield p = 0.2. Which one matters?
-
By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. It's one piece. Not the verdict.
Nature published explainers on each of these. They ran opinion pieces from statisticians, psychologists, geneticists, and journal editors. The message was consistent: the p-value is a tool, not a truth machine.
What Nature Actually Published That Year
It wasn't just news coverage. Nature published primary research and commentary that shaped the conversation:
- The ASA statement (summarized and contextualized in Nature News, March 2016)
- Editorial: "Statisticians' warning on p-values" — calling for cultural change, not just technical fixes
- Comment: "The fickle P value generates irreproducible results" — by Regina Nuzzo, one of the clearest explainers ever written on this topic
- Correspondence from researchers across fields — some defensive, some relieved, some demanding more
- A special collection on statistical challenges in irreproducible research
And then there was the Nuzzo piece. Consider this: if you read one thing from 2016 on this, make it that. She walked through the history of the p-value, the Fisher-Neyman-Pearson wars, the rise of NHST (null hypothesis significance testing), and the quiet disaster of treating p < 0.05 as a publishability gate.
She also introduced a lot of readers to the concept of false positive risk — the probability that a "significant" result is actually a fluke. Not the p-value. The actual risk
of being wrong. That distinction—between a p-value and the false positive risk—is where the real confusion lies, and where the reform movement gained its sharpest edge.
Nuzzo’s article didn’t just explain the problem; it offered a way forward. She urged researchers to stop treating p-values as the be-all and end-all and instead see them as one piece of a larger puzzle. But they must be reported alongside other information: effect sizes, confidence intervals, sample sizes, and study design. She argued that p-values, when properly contextualized, can still be useful. Her message resonated because it didn’t demand the abandonment of statistics—it demanded better statistics Small thing, real impact..
The response was swift. Funding agencies, too, began to ask for more nuanced statistical reporting in grant proposals. Practically speaking, others, like Psychological Science, introduced reforms such as requiring the reporting of effect sizes and confidence intervals alongside p-values. Some, like Nature and The Lancet, started explicitly discouraging the sole reliance on p-values. Journals began revising their guidelines. The tide was turning, but slowly Practical, not theoretical..
Yet, the cultural shift proved harder than the technical one. On the flip side, decades of training in NHST—where a p-value below 0. In some fields, particularly medicine and psychology, the pressure to publish in high-impact journals still incentivized the pursuit of “significant” results. So habits of mind are stubborn. 05 was a golden ticket—didn’t vanish overnight. Researchers, reviewers, and editors alike often defaulted to the old ways, even as they paid lip service to the new reforms Practical, not theoretical..
The reproducibility crisis, which had been simmering for years, exploded in 2015 with the Open Science Collaboration’s replication project. So over 200 labs attempted to replicate 100 psychological studies. The result? Only 39% of the effects held up. On the flip side, the failure rate was staggering. The replication crisis became a rallying cry for change, and the ASA’s 2016 statement, amplified by Nature, became a key document in that movement Surprisingly effective..
The official docs gloss over this. That's a mistake.
But even as the conversation grew louder, resistance persisted. Some statisticians argued that p-values were being unfairly vilified. They pointed out that the problem wasn’t the p-value itself, but its misuse. Others, particularly in fields like genomics and physics, defended the continued use of p-values, especially when adjusted for multiple comparisons. The debate was far from settled Easy to understand, harder to ignore..
What emerged, however, was a consensus: the p-value, in its traditional form, was no longer sufficient. But this approach emphasized not just the result, but the process—how the data were collected, analyzed, and interpreted. It called for preregistration of studies, open data, and replication efforts. It needed to be part of a broader, more transparent approach to statistical analysis. It demanded that science stop chasing statistical significance and start seeking meaningful understanding Practical, not theoretical..
The Nature explainers and ASA statement were not the end of the story, but a turning point. In practice, they marked the beginning of a long-overdue reckoning with how we do science. The p-value, once a symbol of scientific authority, was now being reimagined as a tool—one that, when used responsibly, could help guide discovery without dictating it That's the whole idea..
Some disagree here. Fair enough That's the part that actually makes a difference..
In the end, the message was clear: science is not about finding a magic number. It’s about asking better questions, collecting better data, and thinking more critically about what the numbers really mean. The p-value, when properly understood, can be part of that process—but only if we stop treating it as the final word The details matter here..
