How to read an election poll

What the margin of error covers

A margin of plus or minus 3 points at the 95 percent confidence level means that if the same survey were run 100 times, the result would land within 3 points of the true value about 95 of those times, in Pew's wording. AAPOR adds the other half: about five times in a hundred it will not, which is one reason even the best poll should be read cautiously, particularly when it differs sharply from similar polls fielded at about the same time.

The table works one example with made-up numbers: a poll of 1,000 likely voters in which Candidate A has 48 percent and Candidate B has 44 percent. Each share gets the usual 3-point margin. The gap does not. If A's share is too high by chance, B's is likely too low, and the two errors stack, which is why Pew puts the margin for the difference at about twice that for one candidate.

The range for the lead runs from B ahead by about 2 to A ahead by about 10, so it includes zero. Two more limits apply. The margin belongs to the whole sample, and a subgroup has fewer people and a wider margin: in AAPOR's example, 200 people out of 1,000 carry a margin of plus or minus 6.9 points. And a margin of sampling error applies only to probability samples, where everyone has a known chance of being picked. For opt-in online polls AAPOR says it does not apply, and the plus or minus printed on one is a modeled credibility interval that AAPOR urges caution with.

Sample size, and why more helps less each time

The margin shrinks as the sample grows, but slowly. The American Statistical Association (ASA) gives the benchmarks for a poll estimating a proportion: a sample of 100 gives a margin of no more than about 10 points, 500 about 4.5, and 1,000 about 3. Getting down to 1.5 points would take a sample of well over 4,000. AAPOR makes the same point from the other end: doubling a sample from 1,000 to 2,000 trims the margin by only about a point.

Size is also not quality. AAPOR's journalist guide says a larger sample is not necessarily better, because other factors may matter more. A huge sample drawn from the wrong people still lands on the wrong answer, and no margin of error warns you.

Registered voters, likely voters and the screen

Election polls face a problem that ordinary opinion polls do not. Pew's methods team describes it as producing a model of a population that does not yet exist at the time the poll is conducted: the future electorate. People who say they will vote often do not, and some who say they will not end up voting. A poll of registered voters counts everyone on the rolls. A poll of likely voters tries to count only those who will turn out, so the two can report different numbers from the same race.

The sorting is done with a likely-voter screen. Pew reports that most pollsters ask several questions that together estimate each person's chance of voting: intention to vote, past voting, knowledge of the voting process and interest in the campaign. A deterministic screen applies a cutoff, so each respondent is a likely voter or is not. A probabilistic screen gives each respondent a probability of voting and weights answers by it. In Pew's comparison every method forecast better than counting all registered voters or all who intend to vote, though some did better than others. Pew also notes that pollsters often rely on past turnout and on current measures of enthusiasm, which are educated guesses about a crowd no one has seen yet. When you read a poll, check which group it reports and whether a screen is described.

Weighting, and what it costs

Weighting is a statistical adjustment that makes the sample match the population on traits such as age, race and education. Pew's example: if a survey has too many college graduates compared with their share of the population, people without a degree are weighted up to the proper share. AAPOR's guide adds that weighting should use outside benchmarks, such as Census Bureau data, that fit the group sampled.

Pew's researchers report a growing view that weighting on only a few variables is not enough, and that adjusting on more of them has produced more accurate results in Pew's own studies. Weighting also has a cost. Pew's Andrew Mercer calls it a crucial step for avoiding biased results that nonetheless makes the margin of error larger, an increase statisticians call the design effect. A pollster who reports the simple margin without that adjustment is claiming more precision than the survey earned. As an illustration, with a design effect of 1.5 the plus or minus 3.1 for 1,000 people grows to about 3.8. Members of AAPOR's Transparency Initiative must disclose how they weighted and whether the margin accounts for it.

Nonresponse

Nonresponse means someone sampled for a survey does not take part. The ASA calls it nearly inevitable, because some people will refuse despite every reasonable effort. It becomes a problem when who answers differs from who does not. Pew's example is that college graduates are more likely to participate, so a raw sample can contain too many of them.

Weighting can correct the traits a pollster can measure and compare with an outside benchmark, such as education. Where responders differ from nonresponders in ways no benchmark captures, the bias stays, which is why nonresponse sits among the errors the margin leaves out. Pew separates it from noncoverage, where part of the target population had no chance of being sampled, and from measurement error, where people misunderstand a question or misreport an answer.

Why two polls of the same race differ

Start with luck. Each poll has its own sampling error, and the difference between two polls carries more uncertainty than either alone. Pew's example is two polls with a 3-point margin each: a lead that moves from 5 points to 8 has a margin of plus or minus 8 on that 3-point change, so two surveys alone cannot reliably separate real change from noise. The table shows two hypothetical polls of 900 people each. Their leads differ by 4 points, and the margin on that difference is about plus or minus 9, so sampling luck alone could produce the gap.

Method differences stack on top. Polls can sample different groups, interview by different modes, ask different questions, field on different dates and weight differently. A consistent lean attributable to a pollster's procedures is called a house effect. Stanford political scientist Simon Jackman defined it as bias specific to a polling organisation, and in his study of the 2004 Australian election polls found that a good portion of the differences across polls came from differences in methodology rather than from movements in voter sentiment. A house effect describes a procedure's tendency, and it does not say which poll is closer to the truth.

Averages beat single polls, with a catch

Pew's rule of thumb is that trends across a number of polls give more confidence than one or two. A single poll can also be an outlier by chance. Pew cited a tracker that listed 590 national polls for one past presidential race and noted that at 95 percent confidence about 5 percent of them, around 30, would be expected to miss the true value by more than the margin. Those outliers draw attention because they imply a dramatic change, which is why Pew advises waiting to see whether a surprising result shows up in other surveys.

Pooling raises precision, and the catch is bias. Jackman writes that pooling always enhances precision, but that the validity of the pooled estimate rests on the assumption that the polls are unbiased, which is often not the case. An average of many polls that share the same blind spot inherits it. This is the same logic as pooling studies in a meta-analysis, and the same limits.

What a lead inside the margin does and does not mean

It means the poll cannot rule out, at the usual 95 percent confidence, that the trailing candidate is ahead among all voters. Pew's test is stricter than many headlines: being ahead by more than the single-candidate margin is not enough. With a 3-point margin, a candidate needs a lead of about 6 points before sampling error is an unlikely explanation. AAPOR's guide puts it as ahead by 1.5 to 2 times the margin.

It does not mean a tie. The estimate still points toward the leader, and the interval is a range around it, so values near the observed lead are more plausible than values at the edge. It does not mean the poll is wrong, and it does not mean the result will repeat. The reverse also holds: a lead outside the margin is no guarantee, because the margin covers only sampling error and about 1 poll in 20 misses by more than it. As with any confidence interval, the useful move is to read the whole range.

How to find a poll's methodology statement

A credible poll publishes how it was done. Pew lists the essentials: the sponsor, the data collection firm, where and how participants were selected, the mode of interview, field dates, sample size, question wording and weighting procedures. AAPOR's guide asks the same questions in groups: what was asked, who conducted and paid for it, when it was fielded, and how. It also asks whether the respondents were real people.

Look first on the pollster's own page for the release: a methodology note, a toplines or crosstabs file, or a footnote under the chart. AAPOR runs a Transparency Initiative, and a member organization commits to release these disclosure elements when it releases findings and to provide more detail within 30 days of a request. AAPOR keeps a list of certified organizations on its website. Membership is not a grade. AAPOR says it makes no judgment about the quality of the methods disclosed, and Pew calls participation a positive signal but not a guarantee of rigor. A poll that gives no sample size, no dates and no sponsor has not given you enough to check, and it earns less weight.