(Credit: Picpedia/ Nick Youngson CC BY-SA 3.0 Alpha Stock Images)

A recent Fox poll found record support for Trump impeachment: “A new high of 51% want Trump impeached and removed from office, another 4% want him impeached but not removed from office, and 40% opposed impeachment altogether.”

These figures did not sit well with Trump and his supporters. The next day, the New York Post claimed the pollster for Fox misrepresented the results. The poll, the article argued, included too many Democrats. Supposedly, if the right number of Democrats had been included, the real figures would have shown support for impeachment by just 44.9% to 44.4% — essentially a tie.

The Post’s Mistake

The Post’s claim that the Fox poll contained too many Democrats was based on a Gallup poll reporting 31% Democrats, 29% Republicans, and 38% independents. The Fox poll, according to the Post, included 48% Democrats – way more Democrats than what the Gallup poll showed. Thus, the Post concluded, the Fox poll was invalid.

The problem is that the Post reporter apparently didn’t understand how party affiliation is measured. Typically, pollsters ask two questions. The first asks whether people are Republicans, Democrats or independents.

Many people say they are independents, but in practice they really behave (and think) more like Republicans or Democrats. Therefore, most pollsters ask a second question, only of independents: Do they lean to one party or the other?

For example, the Gallup poll cited by the Post found the following:

In this poll, Gallup reports that the 38% independents break down into 15% who lean Republican, 18% who lean Democrat, and 5% who lean to neither party. 

As do most pollsters, Gallup combines the “lean Republicans” with Republicans, and the “lean Democrats” with the Democrats. In the Gallup case, that means there are 44% Republicans and leaners (29% + 15%), and 49% Democrats and leaners (31% +18%).

The Fox poll also asked two questions to measure party affiliation. Combining the leaners with the appropriate party identifiers, Fox reported 48% Democrats and 40% Republicans. The mistake the Post made was to compare the 48% Democrats in the Fox poll (based on two questions) with the 31% in the Gallup poll (based on one question).

There is, nevertheless, a small difference between the two polls on party affiliation, even when comparing the results based on two questions in each poll. The Fox numbers show an 8-point advantage for Democrats (48% to 40%), compared to Gallup’s 5-point advantage (49% to 44%). That 3-point difference in the Democrats’ advantage, however, hardly affects the overall impeachment numbers.

Applying the Gallup party affiliation numbers to the Fox poll, I calculate that Fox would have shown support for impeachment by 50% to 43% — compared with the 51% to 43% actually reported.

Every good scientific (probability) poll has a margin of error, so there is no reason to believe that Gallup’s numbers are any more precise than the Fox numbers. Both polls show a margin of error for party affiliation of around four to five points, so their differences are well within that range.

Epilogue: That the Fox poll’s results are not an outlier is attested to by a subsequent CNN poll with virtually identical results: 50% believe “President Trump should be impeached and removed from office,” while 43% disagree.

The Quality of the Fox Poll

In response to the mistaken article by the Post, President Trump tweeted: “From the day I announced I was running for President, I have NEVER had a good @FoxNews Poll. Whoever their Pollster is, they suck.”

Despite these comments, the Fox polling unit deserves to be viewed as one that genuinely strives for non-partisan reporting – regardless of any other controversies about the objectivity of the network’s on-air programs.

To insure such non-partisanship, the network employs two research firms – one that is Republican, the other Democratic. The methodology statement, for example, notes:

The Fox News Poll is conducted under the joint direction of Beacon Research (D) (formerly known as Anderson  Robbins Research) and Shaw & Company Research (R). It was conducted by telephone (landline and cellphone) with live interviewers October 6-8, 2019 among a random national sample of 1,003 registered voters.

According to a 2018 rating of polling organizations by Nate Silver’s FiveThirtyEight website, “Fox News/Anderson Robbins Research/Shaw & Co. Research” obtained an “A” rating. No rating system is perfect, of course, and one can argue about the criteria. But such a high rating by an objective observer should nullify President Trump’s vulgar characterization of the Fox poll – which says more about him than the polling unit.