Sample sizes and their limits

I often counsel activists not to worry overly much about sample sizes. They look small, but the math works out because probabilities of independent events multiply, and the sampling of every respondent is an independent, random event.

But as Ed Morrissey points out, NBC’s new poll really is tiny. The MoE is 10.

According to the standard estimate formula for a 95% confidence level – 0.98 / sqrt n – I get a Margin of Error of 9.8 for the poll. And as we can see from the image above, that makes it difficult to draw many conclusions from the poll. Their poll says that Rick Perry’s support was cut in half, from 8 to 4. The problem is, a move of 4 points has only a 58% chance of representing a real shift, according to my calculations.

Why is that so bad? Why is it so close to a coinflip, whether Perry actually moved? Because I treat the two results, the 8 and the 4, as independent events, each with their own margin of error of 10. So the combined margin of error of the difference between the two, I set to 20. A difference of 4 with an MoE of 20 is barely noticeable.

So, yes, I still believe that most complaints about sample size are unwarranted. But when the Margin of Error hits double figures, the utility of a poll does begin to disappear.