Natural experiments: when circumstance does the randomizing

A natural experiment is a study that takes advantage of a real-world circumstance, a policy change, an accident of geography, a rule with an arbitrary cutoff, that sorts people into different conditions without the researcher intervening. It sits between a true randomized trial, where the researcher controls assignment, and an ordinary observational study, where people choose their own exposure and confounding runs loose.

The founding case is physician John Snow's investigation of the 1854 London cholera epidemic. Two water companies, the Southwark and Vauxhall Company and the Lambeth Company, ran competing pipes down the same South London streets, so that neighboring houses, sometimes a single building, could draw water from either one depending on an old, arbitrary choice by a landlord. Crucially, the Lambeth Company had moved its intake upstream of London's sewage outfalls in 1852, while Southwark and Vauxhall still drew from the contaminated tidal Thames. In the first four weeks of the 1854 epidemic, 286 fatal cholera attacks occurred in the 40,046 houses supplied by Southwark and Vauxhall against only 14 in the 26,107 houses supplied by Lambeth, a rate of 71 deaths per 10,000 houses against 5, cholera fourteen times as fatal on the fouler supply.

Two traps follow. First, the comparison only works if the dividing circumstance really is unrelated to everything else that affects the outcome. Snow spent months verifying this before trusting his numbers: he confirmed that the two companies' pipes ran down the same streets and courts, that a single house might get its water from either company depending on an old landlord's choice, and that rich and poor customers were mixed on both sides. A natural experiment that has not been checked this carefully can hide a confounder as easily as an ordinary observational study can. Second, a natural experiment answers a narrower question than it looks like it does: it tells you the effect for the people actually exposed to the split, in that place and time, not for every population everywhere.

So when a headline credits a policy change, a lottery, or an accidental boundary with causing an outcome, ask what actually assigned people to each side of the comparison and whether that assignment could be tangled up with anything else that matters. The strongest natural experiments, like Snow's, describe the mixing in detail and show the two groups were alike in everything except the one variable in question. Pair a natural-experiment claim with a look at how the researchers checked that the assignment really was independent of the outcome, the way regression discontinuity and difference-in-differences designs do more formally today.