What To Make of the Selzer Poll?

Pollster Ann Selzer lobbed the equivalent of a hand grenade into X (nee Twitter) over the weekend when she dropped a poll of Iowa showing Vice President Kamala Harris up 3 points. Needless to say, this would be a great result for Harris if it holds up.

But will it hold up? There are reasons to take the poll seriously, but also reasons to doubt it. Here are some thoughts for and against the finding. On balance I think the arguments against are stronger, but we’re really past the point of speculation at this point. We will find out in a few days.

Pro #1: Selzer is a great pollster. This is probably the strongest argument for taking this poll very seriously. Her track record is beyond reproach. 

More importantly, she has gotten it right when other pollsters have gotten it wrong. For example, in 2020 almost all of the publicly available Iowa polling showed a race in the low single digits, some with Biden leading. Selzer, on the other hand, showed Donald Trump with 10- and 7-point leads, which is roughly what ended up happening. Likewise, in 2016, Selzer was again the pollster who showed Trump with a large lead in the Hawkeye state. In 2014, Selzer showed Joni Ernst pulling away from Bruce Braley.

You get the point. Selzer knows her state better than just about anyone else, and the proof is, as they say, in the pudding. This is not a poll that anyone should dismiss.

Con #1: Outliers happen. On the other hand, outliers happen, even to great pollsters. They’re inevitable. The fact that drawing a relatively small random sample can nevertheless give you useful information about a population is one of the greatest advances in social sciences in the last 100 years.

At the same time, we are still sampling at random, which means that we will get weird outcomes from time to time. What Selzer’s error margin means is that 95% of the time, we’d expect the true population value to fall for Trump support to be captured by a given range. In Selzer’s poll, that range is between 41.5% and 47.5% support for Trump and between 43.5% and 50.5% support for Harris.

Obviously that’s a pretty large range of outcomes; if Trump wins by 4 points it’s still consistent with the poll and social science standards. But what about the other 5% of the time? The true population levels of support will fall outside of the error margin.

Is this one of those 5%? We have no way of knowing. That’s part of why we average polls at RealClearPolitics – it reduces the impact of outliers. 

Here’s the most important takeaway: That 5% chance of a poll just being flat-out wrong is inherent in the fact that a poll is a sample. There’s nothing that even the world’s greatest pollster can do about it. One-time-in-20, a poll is just going to be wrong. 

There are other sources of error (technically “bias”), such as not reaching all relevant sub-populations or weighting the data incorrectly, where pollster skill does come into play. But that is in addition to, and largely separate from, sampling error. 

But can we get a better idea if this is wrong? Let’s move onto Con #2.

Con #2: The Harris and Trump campaigns aren’t acting this way, and it isn’t showing up in other polls. One reason we average polls is because the random sampling errors will tend to cancel out, moving the poll average closer to the true population value as the number of polls increase. This is due to something called the Law of Large Numbers (yes they named it that). 

So what do other surveys show? In Iowa we have a contemporaneous result from Emerson College Polling – also a good organization – showing Trump ahead by 10 points. This is probably more consistent with the overall storyline the polls are showing us. After all, it is difficult to imagine Iowa going blue while Wisconsin is close, which is what all the polling there is showing us. 

Moreover, if this were happening the results likely would have shown up in the regional polling that the campaigns are relentlessly conducting in Wisconsin. The Driftless Region – southeastern Wisconsin – has some important demographic similarities with northeastern Iowa, and it is hard to imagine Harris winning Iowa without overperforming there. This, in turn, would surely invite campaign visits, or at least press leaks.

Or maybe not! There are two counter-arguments here:

Pro #2: Idiosyncratic results do happen. In 2000, the idea of Bush winning West Virginia while the rest of the nation was close struck many as lunacy. After all, the state had just gone for Clinton/Gore by 15 points in the preceding election.

More on point: In 2004 the Bush/Cheney campaign made multiple last-minute stops to odd states – Hawaii and New Jersey – after public polls showed close results. The states weren’t particularly close in the end. John Kerry won Hawaii by 9 points and New Jersey by 7. But they were closer than you might have expected given the overall closeness of the presidential election.

Maybe that’s Iowa in 2024: a state that just moves oddly relative to expectations (some have pointed to the state’s abortion ban as one reason for this, but I don’t find that storyline terribly compelling given the results in 2022). Or maybe there’s something in Selzer’s methodology …

Pro #3: Selzer doesn’t weight much. Ann Selzer is famously old school in her approach to her state. She uses random digit dialing and weights the results lightly. 

In this respect, her methods stands in contrast to other polls, which are heavily weighted and modeled, and arguably aren’t even traditional polls anymore. One of the big concerns with this election is that pollsters have begun weighting by “recalled vote.” That is, they ask people whom they voted for in the previous presidential election, and try to make the sample reflect those results.

This is highly controversial. It is also widespread. It isn’t clear which side this helps (if either). One problem is that respondents sometimes lie or misremember whom they voted for, tending to recall that they voted for the winner. In an era of polarization, this may be less of an issue. On the other hand, if the electorate truly changes, you might end up weighting this away.

Because of this, an anomalous poll finding from Selzer “counts” more in my mind than a curious result from some other pollsters. I’m wary of weighting by recalled vote, which makes an odd result from a pollster who doesn’t use it more likely to be correct than I otherwise would conclude.

Con #3: Selzer doesn’t weight much, and other dissimilar polls don’t weight. At the same time, the New York Times/Siena polls don’t weight by recalled vote either, and their results are also inconsistent with Selzer’s results. They show, like most other pollsters, a very close race in Iowa. So even within the universe of pollsters who don’t weight by recalled vote, Selzer’s looks like an outlier. The recalled vote in Selzer’s poll really does show a heavily Democratic sample as well (Biden won the state among voters in her sample). That doesn’t mean she should weight by recalled vote, but it does perhaps suggest that the sample is in the “pro-Dem” tail of the distribution. Perhaps.

Con #4: It has Harris winning older voters. I am generally opposed to “crosstab diving” because crosstabs have small samples and huge error margins. However, I will confess my ears pricked up a bit when I heard that the sample had Harris carrying older voters, and that she was up 19 with older women.

The reason this concerned me was twofold. First, this would be a pretty substantial shift from 2020, where Trump won older voters by 10 points in the state. Or even from 2016, where he won by 4. The younger cohort, many of whom would have moved into the “65+ age group” in the intervening years, was even more heavily Republican.

The problem is that there isn’t a clear reason for older voters to have moved into the Harris column. There have been no real advertising campaigns in the state, and the entire pro-Harris storyline has been that she excites younger voters. To the extent that we are willing to rely on impressions and age stereotypes, mine certainly aren’t that baby boomers are a suddenly progressive age group.

The more important reason is that in retrospect, this was one of the signs from 2020 that the polls were off. They showed Biden running very well among older voters. This was accepted in poll analysis, and actually made some sense: Biden was an old white dude and COVID was disproportionately threatening to older voters. In the end, though, Trump won nationally among older voters by about 5 points, roughly the same as his 7-point win from 2016.

What likely happened is that older voters, particularly in the Midwest, weren’t talking to pollsters. They were part of the reason for the poll error that year. Does that mean we are having the same result this year? Not necessarily. It’s just when you see an odd result replicated, your ears should perk up. (Again, though, Selzer managed to avoid this problem in 2016 and 2020, so it isn’t clear why she would have it in 2024.)

How I evaluate it: First, you should take this poll result seriously. No one with Selzer’s track record should be dismissed out of hand. There are also methodological reasons to prefer Selzer’s results to other results.

At the same time, it’s dangerous to set aside a mountain of evidence that this is going to be a close election on the basis of one poll.

So what should you do? Toss it in the average, as we do with other polls. It still shows a close race in Iowa, which is good news for Team Harris. This has to be viewed against the backdrop of close races across the country, however, which is less good news. Perhaps this opens another path to victory for Harris-Walz, which campaigns always welcome. But the expectation should probably remain that this is a close race, and we really don’t know what is going to happen on Tuesday.