Statistical power: why a small study can miss a real effect

Statistical power is the probability that a study will detect a real effect, assuming one actually exists. A study with high power is likely to find a true effect if it is there. A study with low power can easily come back empty-handed even when the effect it was looking for is real, simply because it did not have enough participants or observations to see it clearly through the noise.

Power depends mainly on three things: how big the true effect is, how much the measurements vary from person to person, and how many people or observations the study includes. Small samples are the most common reason a study ends up underpowered, since researchers can rarely make an effect bigger or make measurements less noisy, but they can, within budget and time limits, recruit more participants.

Two traps follow. First, a small study that finds no significant effect has not shown the effect is absent, only that the study lacked the power to detect it, a distinction headlines routinely blur. Second, when a small, underpowered study does report a significant result, that result tends to overstate the true effect size, because only an unusually large, and partly lucky, measured effect could clear the bar for significance in such a small sample.

When a study reports a finding, especially a surprising one from a small sample, check whether it states its power or its sample size relative to similar studies in the field. A field with a history of underpowered studies, such as early brain imaging research, tends to produce results that later fail to replicate at the same size. Pair a striking small-sample finding with a look at whether a larger, better powered study has tried to confirm it.