What a p-value really tells you (and what it doesn't)

What counts as a significant p-value

A p-value runs from 0 to 1. By convention, a result is called statistically significant when its p-value is below 0.05, which means data this extreme would show up less than 1 time in 20 if there were no real effect. Some fields and analyses use a stricter line such as 0.01.

The word only reports which side of a line the result fell on. A p of 0.049 and a p of 0.051 are nearly the same evidence, and one gets called significant and the other does not.

Where 0.05 came from

Ronald Fisher’s 1925 book Statistical Methods for Research Workers put the cutoff into circulation. He wrote that the value for which P = .05, or 1 in 20, is 1.96 or nearly 2, and that “it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not.” Statistician Lee Kennedy-Shaffer, tracing the history, points to the arithmetic convenience: 0.05 is roughly two standard deviations from the mean of a normal distribution, which mattered before computers, when researchers looked results up in printed tables. Fisher’s tables almost always included 0.05 and 0.01 columns.

Even then it was called a convention. L. H. C. Tippett wrote in 1931 that the 0.05 threshold was “quite arbitrary” but “in common use,” and Fisher himself later wrote that “no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses.”

What significance does not tell you

In 2016 the American Statistical Association issued a statement with six principles. Three of them: P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. And a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

Its executive director, Ron Wasserstein, put it this way: “The p-value was never intended to be a substitute for scientific reasoning.” A significant result can still be tiny, and a result that misses the line can still be real but under-tested.

Nominal p-values

A nominal p-value is the one reported straight from a single test, at the usual 0.05 line, with no adjustment for how many other tests were run. Papers use the word to mark the difference from an adjusted p-value.

The adjustment matters because chances add up. Two independent outcomes each tested at the nominal 0.05 level give a 0.098 probability of at least one false positive, according to a statistical review of trials with several primary outcomes. Across 100 tests of things that are all null, about 5 will come out significant by luck alone. That is why a lone nominal p-value from one of many comparisons deserves less trust than the same number from the single test a study was planned around.

The stars: *, ** and ***

Asterisks are shorthand for p-value bands in tables and charts. A common set, used by the statistics program GraphPad Prism, is ns for p above 0.05, * for 0.05 or less, ** for 0.01 or less, and *** for 0.001 or less. Some settings add **** for 0.0001 or less, and others stop at three stars.

More stars mean a smaller p-value, which means the data would be more surprising if there were no effect. They do not mean a bigger effect. Definitions vary, so check the figure legend or table note.

What to read alongside it

The ASA notes that statisticians often supplement or replace p-values with methods that emphasize estimation over testing, such as confidence, credibility or prediction intervals. In practice, look for the size of the effect and the interval around it, then ask whether the finding has been replicated and whether the analysis was planned in advance.