Multiple comparisons: how testing enough things finds something

The multiple comparisons problem is that testing many things at once raises the odds that at least one test will look statistically significant purely by chance, even if nothing real is going on. A single test run at the conventional 5 percent significance threshold has a 1 in 20 chance of a false positive. Run 20 such tests on random noise and, on average, one will come back looking significant anyway.

The field's most memorable illustration came from neuroscientists who placed a dead Atlantic salmon, bought whole from a supermarket, into a functional MRI scanner and showed it photographs of people in social situations. Scanning roughly 130,000 tiny brain regions, or voxels, without correcting for the sheer number of tests produced a cluster of voxels in the dead fish that appeared to respond to the images. Applying the standard corrections for multiple testing made that apparent brain activity disappear completely.

Two traps follow. First, an analysis that scans many outcomes, subgroups, or brain regions for anything significant, without adjusting the threshold for how many tests were run, is set up to find noise and report it as a discovery. Second, correcting properly for multiple comparisons is a known, standard step in fields like brain imaging and genetics, so a striking result that skipped it should be treated as unproven rather than simply cautious.

When a study reports a significant finding among many things it measured, such as one gene among thousands or one brain region among many, check whether it corrected its statistical threshold for the number of tests run. A result that survives a proper multiple comparisons correction is far more trustworthy than one plucked from a long list of comparisons. Pair any single striking finding with a look at how many other comparisons the same study quietly ran and did not report.