In order to evaluate the quality of polls, we need to find out a lot of information. The ten questions below, framed within the “Five Ws and One H” guideline for reporters (Who, What, When, Where, Why and How), provide a guide for obtaining needed information.

Who

1.Who conducted the poll?
2. Who sponsored the poll?

What

3. What was the wording (and wording context) of the question? (Among other things, did the question include measures of “non-opinion” and intensity?

When

4. When was the survey conducted?

Where

5. Where is the population that is being sampled?

Why

6. Why was the poll conducted?

How

7. How large is the sample (and what are the corresponding margins of error and response rates)?
8. How was it obtained (probability or non-probability sampling)?
9. How was it modified, if at all (Likely voters, registered voters, all adults, any other screening, etc.)?
10. How was the survey administered (live interviewer telephone, recorded telephone interviewer, cell phone, landline phone, Internet/web survey, in-person visit, mail, hand-written, etc.)

Explanation for these questions:

There are really two major areas of information we need about polls to evaluate their potential quality: The purpose for conducting the poll, and the mechanics or methodology by which the poll was conducted.

In examining the purpose for the poll, we want to know who sponsored it and who conducted it to see if there is any obvious conflict of interest, or axe to grind, by those paying for the poll. For many major media polls, the purpose will be to provide information about public opinion as it relates to ongoing news stories. For many other polls, the purpose may be to make an argument for some special interest group that the public supports the position of the interest group. And, of course, there could be many other reasons for polls to be conducted.

The three questions that relate to the purpose of the poll are:

1. Who conducted the poll?
2. Who sponsored the poll?
6. Why was the poll conducted?

It’s possible the poll was conducted by a reputable polling firm, but still was sponsored by an interest group with very great concern about the outcome of its results. In the Top Ten “Dubious Polling” Awards, we have pointed to polling firms on both the right and the left who have produced results that just happen to support the views of the sponsoring organizations!

In such cases, given the ease of manipulating results by biased question wording, it’s crucial to examine all other aspects of the poll to assess if it’s biased or not.

Even in the case of polling organizations, like the major media polls, there is often reason to suspect less than objective results. Most media polling organizations most of the time want to portray the public as highly informed, rational, and opinionated – because such an image of the public supports the rationale for conducting polls in the first place. Other times, though fairly rarely, the pollsters want to portray the public as generally uninformed, because such an image happens to fit in with the news story they are supporting.

When the first purpose motivates the poll, pollsters tend to ask forced-choice questions and pressure respondents to come up with opinions, even if they have hardly thought about the issue. In some cases, pollsters will even feed respondents information, and then ask for their opinion – to give the impression that the public has an informed opinion.

When pollsters want to signal that the public is undecided on an issue, they are more likely to give respondents an explicit “don’t know” or “unsure” option in the question itself.

Thus, answers to the questions as to who sponsored the poll, who conducted it, and why it was conducted are all important factors for evaluating the potential validity of the results.

In examining the methodology of the poll, there are many questions that need to be addressed in order to assess whether the poll can legitimately be used to generalize to some larger population. If the poll is to be interpreted as a reflection of the opinion of some larger population, then various conditions have to be met. Typically, that is the purpose of polls. In such cases, we refer to the polls as being “scientific” – which means that very specific steps were taken to make the sample of respondents representative of that larger population.

There are some examples where polls are not explicitly intended to reflect some larger population, such as online polls at some website where the viewer is invited to take a poll. There is no systematic attempt to get all elements of the public to participate, and sometimes the website will admit up front that it is “not a scientific poll.”

Most media polls are explicitly intended to measure what the public is thinking about candidates or policy issues, and thus they need to meet explicit scientific requirements to make the poll’s sample of respondents representative of the larger population.

The questions listed above (#3-5, and 7-10) provide a guide to obtaining the required information about poll methodology.

For further explanation about poll methodology, there are numerous textbooks. Listed below are some, but not all, of the textbooks that might be helpful.

For a brief introduction to polls and their methodology, Herbert Asher’s books is especially relevant.

Other Recommended Reading:

• Asher, Herbert. Polling and the Public: What Every Citizen Should Know (CQ Press, 8th Edition, 2011).
• Bardes, Barbara A. and Robert W. Oldendick.  Public Opinion: Measuring the American Mind (Wadsworth Publishing, 3rd Edition, 2005).
• Dillman, Don; Jolene D. Smyth; Leah Melani Christian.  Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method (Wiley, 3rd Edition, 2008).
• Fink, Arlene. How to Conduct Surveys: A Step-by-Step Guide (Sage Publications, 4th Edition, 2008).
• Fowler, Floyd J. Survey Research Methods (Applied Social Research Methods) (Sage Publications, 4th Edition, 2008).
• Groves, Robert M.; Floyd F. Fowler, Jr.; Mick P. Couper; James M. Lepkowski; Eleanor Singer; and Roger Tourangeau. Survey Methodology (Wiley Series in Survey Methodology) (Wiley, 2nd Edition, 2009).
• Rea, Louis M. and Richard A. Parker. Designing and Conducting Survey Research: A Comprehensive Guide (Josey Bass Public Administration Series) (Jossey-Bass, 3rd Edition, 2005).
• Weisberg, Herbert F. The Total Survey Error Approach: A Guide to the New Science of Survey Research (University of Chicago Press, 2005).