What went wrong in New Hampshire?
Posted: Tuesday, March 31, 2009 3:24 PM by Lauren Vicary
From NBC’s Domenico Montanaro
The polls in the 2008 presidential campaign were largely spot on -- with one glaring problem: the New Hampshire Democratic primary. A panel of polling specialists released a report yesterday that tried to figure out just what went wrong.
As Boston.com reports, it “didn't come to any definitive conclusions, but said that the polling probably ended too early to take into account late movement among voters. Also, Clinton supporters were harder to reach and some pollsters did not try more than twice, skewing the sample toward pro-Obama voters, said the committee organized by the American Association for Public Opinion Research. The panel discounted other possible explanations, such as the so-called ‘Bradley effect,’ in which some white voters say they will support a black candidate, but don't vote that way in the privacy of the polling booth.”
The group makes this point: “Polling in primary elections is inherently more difficult than polling in a general election.” And: “The estimation errors in the polls before the New Hampshire Democratic primary were of about the same magnitude as in the Iowa caucus. ... But the majority of the polls before New Hampshire suggested the wrong winner, while only half in Iowa did.”
Here are some highlights from the 123-page report:
Factors that may have influenced the estimation errors in the New Hampshire pre-primary polls include:
1. Given the compressed caucus and primary calendar, polling before the New Hampshire primary may have ended too early to capture late shifts in the electorate there, measuring momentum as citizens responded to the Obama victory in the Iowa caucus but not to later events in New Hampshire.
2. Patterns of nonresponse, derived from comparing the characteristics of the pre-election samples with the exit poll samples, suggest that some groups that supported Senator Hillary Clinton were underrepresented in the pre-election polls.
3. Variations in likely voter models could explain some of the estimation problems in individual polls. Application of the Gallup likely voter model, for example, produced a larger error than their unadjusted data. While the “time of decision” data do not look very different in 2008 compared to recent presidential primaries, about one-fifth of the voters in the 2008 New Hampshire primary said they were voting for the first time. This influx of first-time voters may have had an adverse effect on likely voter models.
4. Variations in weighting procedures could explain some of the estimation problems in individual polls. And for some polls, the weighting and likely voter modeling were comingled in a way that makes it impossible to distinguish their separate effects.
5. Although no significant social desirability effects were found that systematically produced an overestimate of support for Senator Obama among white respondents or for Senator Clinton among male respondents, an interaction effect between the race of the interviewer and the race of the respondent did seem to produce higher support for Senator Obama in the case of a black interviewer. However, Obama was also preferred over Clinton by those who were interviewed by a white interviewer.
Factors unlikely to have contributed to the estimation errors in the New Hampshire pre-primary polls include:
1. The exclusion of cell phone only (CPO) individuals from the samples did not seem to have an effect. However, this proportion of citizens is going to change over time, and pollsters should remain attentive to its possible future effects.
2. Using a two-part trial heat question, intended to reduce the level of “undecided” responses, did not produce that desired effect and does not seem to have affected the eventual distributions of candidate preference.
3. The use of either computerized telephone interviewing (CATI) techniques or interactive voice response (IVR) techniques made no difference to the accuracy of estimates.
4. The use of the trial heat questions was quite variable, especially with regard to question order, but no discernible patterns of effects on candidate preference distributions were noted. While the names of the (main) candidates were frequently randomized, the committee did not receive data that would have permitted an analysis of the impact of order.
5. Little compelling information indicates that Independents made a late decision to vote in the New Hampshire Republican primary, thereby increasing estimate errors.
Factors that present intriguing potential explanations for the estimation errors in the New
Hampshire polls, but for which the committee lacked adequate empirical information to thoroughly assess include:
1. The wide variation in sample frames used to design and implement samples – ranging from random samples of listed telephone numbers, to lists of registered voters with telephone numbers attached, to lists of party members – may have had an effect. Greater information about sample frames and sample designs, including respondent selection techniques, would facilitate future evaluations of poll performance.
2. Differences among polls in techniques employed to exclude data collected from some respondents could have affected estimates. Given the lack of detailed disclosure of how this was done, it is not possible to assess the impact of this procedure.
3. Some polls combined weighting to adjust for nonresponse among demographic groups with weighting that reflects likely voter models into a single set of weights for a study. This complicates the analysis of whether or how much sampling issues or likelihood of voting models are contributing to estimation error.
Finally, factors that appeared to be potential explanations for estimation errors, but for which the committee lacked any empirical information to assess include:
1. Because of attempts by some states to manipulate the calendar of primaries and caucuses, the Iowa and New Hampshire events were rescheduled to the first half of January, with only five days between the events, truncating the polling field period in New Hampshire following the Iowa caucus.
2. The order of the names on the ballot – randomly assigned but fixed on every ballot - may have contributed to the increased support that Senator Hillary Clinton received in New Hampshire.
On The Bradley effect specifically: A 2008 analysis "indicates that while the Bradley effect did undermine some state-level polls in previous decades, there is no evidence for such an effect in recent years. In the 2008 general election, the very accurate final poll estimates of Barack Obama’s fairly decisive victory over John McCain dispelled suspicion that the Bradley effect was at play during the final weeks of the fall contest. There is also a conspicuous lack of evidence for a Bradley effect in the primary contests outside of New Hampshire. ... However, it is still possible that intentional misreporting occurred during the lead up to the New Hampshire Democratic primary because of the interaction between the race of the interviewer and the race of the respondent."
