Gallup caught lying about the generic ballot trend [Updated 3PM PT]
By Neil Stevens on July 21, 2010
[Updated at the bottom at 3PM Pacific Time.]
The Gallup Generic Ballot is a trusted, widely reported resource. I’ve analyzed it extensively, and defended it to others. But yesterday, when I covered the poll’s latest release, Gallup lied. I was lied to, you were lied to, everyone who’s trusted the Gallup name got lied to.
How? Gallup is combining two different sampling methods into one so-called trend of Registered Voters (complete with captioned graph), getting different results, but pretending they show one trend all the same.
Remember on June 2 when Republicans took a big lead in the Gallup generic ballot? I used it to project conservatively a 45 seat Republican gain in the House. This was a poll of registered voters, according to Gallup’s Survey Methods notes:
Results are based on telephone interviews conducted May 24-30, 2010, with a random sample of 1,594 registered voters, aged 18 and older, living in all 50 U.S. states and the District of Columbia, selected using a random-digit-dial sampling technique.
But now on July 19 that Democrats are showing a big lead, despite the fact that Gallup’s pretty graph now is titled Candidate Preferences in 2010 Congressional Elections, Among Registered Voters, the sampling is different:
Results are based on telephone interviews conducted as part of Gallup Daily tracking July 12-18, 2010, with a random sample of 1,535 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia, selected using random-digit-dial sampling.
Catch the difference? The Republicans lead with a sample of Registered Voters, but the Democrats lead with a sample of Adults. Someone who trusted Gallup’s pretty, but lying, picture would never have noticed. Real Clear Politics noticed, and actually recorded the polls differently. Friends noticed this and alerted me.
It is terribly dishonest for Gallup to string together two different polls as one series, as Gallup does not only in their graphs, but in their write-ups as well. Here’s an example from the July 19 release:
The Democrats’ six-point advantage in Gallup Daily interviewing from July 12-18 represents the first statistically significant lead for that party’s candidates since Gallup began weekly tracking of this measure in March.
Notice, they call the series one measure, even though it’s at least two different kinds of polls with two different kinds of sampling pools. You cannot pretend that a poll of all adults and a poll filtered by registered voters are part of the same series, even if the same questions are asked. That’s Polling 101, and whoever’s responsible for the Gallup release should have known this, and certainly whoever’s responsible for oversight of the Gallup releases would know this.
Somebody’s lying. I have no reason to believe that Gallup’s actual polling is fabricated, but it’s easy to see that how Gallup packages its results is designed to mislead readers into seeing trends that just aren’t there. Observe the Gallup trend, per Real Clear Politics, with polls of adults distinguished from polls of registered voters, the most honest approach:
| Date | Pool | Lead |
|---|---|---|
| 7-18 | A | D+6 |
| 7-11 | RV | D+1 |
| 7-3 | RV | R+6 |
| 6-27 | RV | D+1 |
| 6-20 | RV | D+0 |
| 6-13 | RV | R+5 |
| 6-6 | RV | D+0 |
Sorted this way the trend is clear: Registered Voters oscillate between near-ties for the Democrats, and big leads for the Republicans. But once we switch to Adults, Democrats suddenly take a big edge of their own.
Is this credible? No, but you’d never know it if you took Gallup’s releases at face value. Even if the truth was there if you dig deep enough, it’s still a lie to misrepresent the data as Gallup has done.
As a final note, I have a prediction: Somebody, perhaps even Gallup, will claim that the Survey Notes were mistaken in this last release, and that the poll was of Registered Voters, but was just recorded wrongly. If this is the case, then that still suggests a serious lapse of quality control at Gallup leading to misinformation being published, and still calls into question every poll release Gallup makes. So I don’t buy it as a defense.
Besides, Gallup noted that the latest Generic Ballot was included as part of a daily tracking poll. It is vastly more likely that this change is a result of cost-cutting, combining one poll with another to save money over the summer, when they hope nobody will notice. But really, when you change the secret sauce, somebody’s going to notice.
Update: Gallup has now issued a correction alongside the bare changing of ‘adults’ to ‘registered voters’ in the Survey Methods:
Editor’s note: The original version of this story inadvertently referred to national adults rather than registered voters in the survey methods statement. The results reported here and in all Gallup generic ballot trends so far this year are based on registered voters; the survey methods statement now correctly reflects that.
A commenter at RedState.com pointed out a significant flaw in this story: the sample size didn’t change. Originally the post read 1,535 adults, now it reads 1,535 registered voters. However the overall daily Obama Job Approval tracking poll also surveys “approximately 1,500 national adults.” are we to believe that approximately 99% of the adults in daily tracking sample were registered to vote? MSNBC reported in 2008 that about 80% of American adults were registered, and that number will decline from 2008 to 2010 as voters are purged and don’t re-register for the midterm. The odds against this are astronomical, making this poll an incredible outlier for so many to be registered.
My conclusion: that this was a simple typo just doesn’t add up.