Focusing On Error In Polls Isnt Sexy, But Its Necessary

It’s that point in the election cycle where if you don’t like the results of one poll, you can wait five minutes and another one will come out. People are inundated with numbers, which they might not know how to interpret.

Polls contain “error” in them, which means the numbers are estimates of what people think, rather than precise data points. Pollsters and journalists will usually provide a “margin of error” that approximates how much the numbers might change due to the random chance of who is selected to participate in the poll.

But analysts know there’s more error in a poll than this margin indicates. Some say that’s why we shouldn’t report margins of error at all. And we often don’t do a good job of explaining any type of error to audiences.

Nate Cohn provided an excellent demonstration of polling error not accounted for in a margin of error by having four different sets of pollsters re-analyze data from The New York Times Upshot/ Siena Florida poll . Their results ranged from Democratic presidential candidate Hillary Clinton  being 4 points ahead to her GOP opponent Donald Trump ahead by 1 point, using the exact same data but creating their own likely voter calculations. This is an important lesson in non-sampling sources of error: The choices an analyst makes about how to define likely voters can substantially affect the outcome of the poll.

But Cohn takes the lesson a step further and argues that the existence of these other sources of error mean margin of error wasn’t worth reporting in the original poll . “Clearly, the reported margin of error due to sampling, even when including a design effect (which purports to capture the added uncertainty of weighting), doesn’t even come close to capturing total survey error,” Cohn states. 

It isn’t actually unusual for polls not to report a margin of error. Some surveys don’t report margins of error or even mention the concept, particularly online surveys which don’t use random sampling the basis for a margin of error calculation. But the NYT Upshot/Siena poll was conducted by telephone, and a margin of error typically accompanies these types of polls.

Not reporting a margin of error brings up a big concern for those of us trying to communicate with readers about polling: If there’s not a prominently listed margin of error, do they understand that there’s error? Do they have any idea how much error there might be? 

Unlike analysts, who know there’s error and can quickly estimate a margin of error by looking at a poll’s sample size, most readers aren’t thinking about the uncertainty in polling estimates. It’s up to the experts to explain it.