The Field Guide · No. 07

Why one study is never the whole story

A finding earns trust when other scientists can repeat it, so one study is a data point, not a settled conclusion.

The most important thing to know about any single study is that it is a single study. Science works by repetition. A finding becomes trustworthy not when one team reports it once, but when other teams, with their own data, get the same answer. One study is a first draft of a fact, not the final version.

There are two related ideas here. Reproducibility means someone can take the same data and the same methods and get the same result, a check that the analysis was done right. Replicability means a fresh study, with new data collected independently, reaches a consistent answer, a check that the finding is real and not a one-off. The second is the harder and more meaningful test.

This matters because a lot of exciting single studies do not hold up. When researchers have gone back and tried to repeat headline-grabbing findings, a surprising share could not be reproduced. That is not a scandal so much as how the process is supposed to work: the weak results get filtered out over time, and the sturdy ones survive. The trouble is that headlines usually cover the first splashy study, not the quiet failure to replicate it two years later.

So when a striking single result makes the news, the calm response is to wait. Ask whether it is a first-of-its-kind finding or one that other groups have already confirmed. A result that has been replicated by independent teams is worth far more than a lone dramatic one, no matter how good the headline sounds.

What to remember

From the record

Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

National Academies of Sciences, Engineering, and Medicine Reproducibility and Replicability in Science, 2019

Asked often

What is the difference between reproducibility and replicability?

Reproducibility is getting the same result from the same data and methods, which checks the analysis. Replicability is getting a consistent result from a new, independently collected dataset, which checks that the finding itself is real.

Why can't I trust a single study?

Any one study can be thrown off by chance, a small sample, or a quirk of its methods. Findings earn confidence when independent teams repeat them. Until then, a single result is provisional.

Read the news better, every morning.

We send you the day's real progress in science, medicine and beyond, told plainly and sourced. Free, forever.