Editor’s Note: David W. Moore, author of “The Opinion Makers: An Insiders Exposes The Truth Behind The Polls,” was senior editor at the Gallup Poll for thirteen years. Before that he was a professor of political science at the University of New Hampshire and the founder and director of the UNH Survey Center. Moore’s expertise and willingness to speak truth to power is what singularly qualified him to weigh in on iMediaEthics about CNN’s use of junk science during the Presidential Debates—and how it’s deceptive to say it’s polling or “giving voters a voice in network coverage.”
It’s a technology that’s about a quarter of a century old, but Southern Methodist University (SMU) professor, Rita Kirk, persuaded CNN to use the meters during the current presidential debates, because (as she told Rhonda Roland Shearer, founder of iMediaEthics, in a personal interview), “two talking heads is boring television.”
Her argument appealed to the media moguls, who apparently accept the premise that being American is never having to listen to political leaders talk about serious issues.In its never-ending efforts to entertain voters during this critical election campaign, CNN has been using electronic audience meters to measure the impact of the debates on an arbitrarily chosen group of people, who are able to express in “real time” their positive or negative feelings about what they’re watching.
Should Americans really care what 32 people from Ohio think?
The audience meter is called a Perception Analyzer Dial described by a news story from SMU as a “palm-sized electronic dial meter.”When focus group participants like what they are hearing, they turn the knob on the dial to the right; when they don’t like the candidates or their ideas, they turn the knob to the left.
The results are immediately aggregated into squiggly lines that run across the bottom of the screen, at the same time the candidates are speaking. The advantage of this simultaneous broadcasting of the debate, and the select audience’s reaction to that debate, is that voters across the country can avoid having to think about what the candidates are saying, and instead focus on what some other unknown people think about what the candidates are saying — thus reducing the strain of thinking for oneself.
It is very difficult to correlate the exact locations of candidates’ statements within the squiggly lines. The speed of spoken words occurred at a much faster rate than the slow wave of colored lines in the graphic.
If it’s not a poll, what can those squiggly lines tell us?
SMU’s Dr. Kirk, who, with her partner, Dr. Dan Schill. convinced CNN to use the dial-a-meters, told Ms Shearer that the process is not a poll. The number of people in the audience is too small for the results to be presented as a poll, she said, but that did not deter CNN’s Soledad O’Brien from suggesting that the 16 men and 16 women from Columbus, Ohio somehow represented a “microcosm” of the 200+ million men and women in the country.
Dr. Kirk noted that while most polls include a lot of people with few questions, the dial-a-meter technology uses only a few people with a lot of questions – thus, apparently, balancing everything out. Perhaps the ultimate goal is to have just one person with a jillion questions, which could reduce conflict and save a lot of money?
“Traditional polling, and that’s where you take a small number of questions, you ask it to a large number of people…Q methodology, which is the basis of this technology, is where you take a small number of carefully selected people, and then we ask them a large number of questions.” – Dr. Rita Kirk
The SMU dial-a-meters avoid such a boring activity, because each individual silently makes his or her own instantaneous rating, thus avoiding group discussion and in-depth thought, and arriving more directly at the superficial, top-of-mind opinion that the media love so well.In the meantime, CNN uses a group of 32 voters, what people in the trade call a “focus group,” to rate the debates. The problem with such groups, of course, is that they are designed to obtain in-depth responses from the participants, who actually discuss the issues with each other and arrive at more considered views than what polls normally measure.
“People react more negatively when candidates attack each other or when they are overly detailed in the explanation of their policies.” – Dr. Dan Schill
Some of the stunning insights provided by these meters are suggested by Dr. Kirk, who noted to Ms. Shearer that “Republicans tended to think favorably of McCain on the war issues and on, you know, two or three issues.” Indeed, the squiggly lines across the bottom of the screen during the first debate were color coded by whether the focus group participants were Republicans, independents or Democrats.
Unlike the sales pitch which extolls the enhancement of democracy, the “Perception Analyzer” meter, is also described on its web site, as able to exert control over forum group members. The marketing literature advises, “Circumvent dominant respondents and minimize negative group dynamics while giving more passive respondents an equal voice.” Hal, is that you?
The lines often diverged from each other, showing Democrats positive about Barack Obama and Republicans positive about John McCain, with independents in the middle. For several decades, pundits had suspected that voters responded to political leaders on a partisan basis, but now with SMU’s dial-a-meters, their suspicions had finally been confirmed.
Men versus Women in the VP Debate
In the second debate with the vice presidential candidates, the squiggly lines across the bottom were presented separately for men and for women in the focus group. This seemed appropriate, since the two veep candidates were widely perceived as being of different sexes.
Sure enough, the squiggly lines often diverged between the two groups, with women tending to be more positive toward Joe Biden and men more positive toward Sarah Palin.
One interpretation is that the divergence was caused by partisanship, because men tend to be disproportionately Republicans and women disproportionately Democrats. But an alternative interpretation is that the dial-a-meter results were much more profound, reflecting the deep gender divide in America – where men tend to like young, pretty women who smile a lot and wink at them, while women like older, well-established, successful men who can take care of them (and who are also emotionally vulnerable).
Focus group members can see numbers on their audience reaction meters. TV viewers can only see squiggly lines and must guess approximate numerical values.
Any possible value is lost due to O’Brien’s priming
Shearer points out that even that possibility, however, is tainted by CNN’s priming of the focus group before the debate started. (Ms. O’Brien said, “A couple of things that we’re interested to see is how is negativity going to play? Because we know from research, negative go negative, dial testers tend to turn down the dial. They hate it.”)It’s at least remotely possible that after the debate, researchers could analyze the data generated by the dial-a-meters to arrive at some insights about the dynamics of opinion change.
Ms. O’Brien suggestion likely influenced the participants. The net result seemed to be that she scared the participants into not being negative themselves. During the vice presidential debate, the squiggly lines across the bottom never ventured into negative territory, ranging only from neutral (50) to positive (100) .
Whatever the post-debate utility of the data to researchers (Is there any?), it is clear that a contemporaneous viewing of the debate, and the squiggly lines running under the candidates, hardly provided CNN’s viewers with any meaningful aid to understanding the issues discussed by the candidates.
The positive side of the dial-a-meters is that they diverted voters from reflecting on the depressing problems being addressed by the candidates into the less stressful activity of wondering what those squiggly lines really meant.
If that’s not a reasonable role for the media to play in a democracy, then what is?
For those interested here is the transcript of O’Brien’s comments about the focus group before the VP debate.
| CNN ELECTION CENTER
Vice Presidential Candidates Prepare to Face Off But I want to go to Soledad O’Brien. She’s in Columbus, Ohio — Ohio being a key battleground state. And we can’t emphasize how important Ohio is. No Republican has ever won the White House without carrying Ohio. All right. Soledad, explain the folks you’ve assembled in Columbus this week to watch this debate and what’s going to happen. SOLEDAD O’BRIEN, CNN SPECIAL CORRESPONDENT: Yes, we’re kind of having our own watch party here. But here, this time around, we had gathered 32 people roughly divided between a third registered Democrats, a third registered independents, a third registered Republicans. And they’ve all been armed once again with the perception analyzers, wireless technology. Depending on how you move the device, it will all have their answers registered into the computer. So move it to the left, low. That means you are not loving what you’re hearing a candidate say. You move it to the right up to 100, that means you’re really, really connecting really, you know, what they’re saying is really resonating with you. And because that information is then fed through computers, they’re able to analyze literally, Wolf, second by second if there is a connection, if there is an impact of what the candidate is saying on our panel here. It’s going to be very interesting to watch then those squiggles that will appear on your television set because you’ll be able to see Democrats, independents, Republicans at the second of phrases being said, or even a topic is being discussed. How do people in this room feel about it? And as you mentioned, a very important state, the state of Ohio. But also people have called this sort of a microcosm of the U.S., and the issues that are important here are issues that are really resonating with people across the country. So it’s pretty important to be sitting right here in Columbus. A couple of things that we’re interested to see is how is negativity going to play? Because we know from research, negative go negative, dial testers tend to turn down the dial. They hate it. But kind of the role of the VP candidate is to be the attack dog. How are they going to navigate that? How is that going to work? And of course at the end of the day, what will be the impact overall of this debate on the presidential race? I want to quickly introduce you to a couple of people, Corina Cox (ph), registered Democrat. Corina, what do you want to hear tonight? CORINA COX, OHIO VOTER: I want to hear about the energy crisis, what they’re going to do about that and the economy and health care. O’BRIEN: Specifics important. And those are the issues, Wolf, as you well know, that most Americans would tick off as their top three as well. We also have Phil Legmiller (ph). He is a registered Republican, also undecided. All the folks in this room say they are persuadable to see what happens at the end of the night. Phil, what do you want to hear and from whom? PHIL LEGMILLER, OHIO VOTER: I want to see what Governor Palin says about how she reacts under a national stage and actually international stage. And then, give me a plan. What are we going to do? Quit giving me verbiage, you know. Just say something positive for once, instead of this same old story from all the candidates. O’BRIEN: Interesting. Talking about the positive. The folks here really say when they go negative, everybody hates it. So we will see how this debate goes tonight. Everybody is poised and ready to weigh in. Wolf, I’ll send it back to you. I know you want to kind of do a dry run of the dial test. So folks, turn your dials down to the very beginning, please, if you will to one — Wolf. BLITZER: Soledad, thanks very much. O’BRIEN: Then let’s bring them up to 100 please. BLITZER: And I want to explain to our viewers, though, what they’re going to see on their screen during the course of this 90- minute debate that is about to begin. Let’s put it up on the screen and be able to show you there’s going to be lines at the bottom of the screen that are going to be moving and literally changing potentially every second. There you can see the green is what the 15 women undecided voters in Soledad’s focus group, what they’re thinking. Excuse me, the men are the green. The women, the yellow, that they’re going to have a separate line. We divided it between men and women to see how they react. Last Friday night in the presidential debate, we had Democrats, Republicans and independents. But we wanted to show you a little bit of how the men might react differently than the women, men being Democrats, Republicans and independents the same for women. Those are the lines you’re going to see and it’s going to show you how people are reacting as this debate moves along from question to answer, and answer then follow up and all of that. If you have high definition TV on the sides of your screen, you’ll see our analysts’ scorecards. You’ll see what John King and Ed Rollins and Hilary Rosen, Paul Begala, Gloria Borger, Alex Castellanos, among others, how they’re scoring the various points in the debate. When they think a candidate took advantage of an opportunity and when a candidate missed an opportunity, they’ll go plus and minus so you’ll have a running score of that. That’s going to be for those of you who are watching in high definition. Or if those of you who are watching us along with CNNpolitics.com, you could see the scoring going on from our analysts as well. So some unique features, Campbell. It’s going to be fascinating for a lot of folks to watch. |
To read post VP debate comments regarding the “Audience Reaction Meter” go to the CNN.com,
Analysis of Vice Presidential Debate, Aired October 2, 2008 – 22:30 ET
| COOPER: Let’s do that right now. Let’s go back to our focus group, Wolf, which Soledad has.
BLITZER: Anderson, Soledad O’Brien has been reporting from Columbus, Ohio, on the campus of Ohio State University. You have got 32 undecided voters, some of whom we saw earlier, Soledad, they’ve made up their minds. But I want to hear more of what they thought about what happened today. O’BRIEN: Yes, they’re roughly divided into independents, Democrats and registered Republicans. But they all said that they were persuadable. So even though they might be leaning one way or the other, they came in into the door thinking that they might change their minds. One of the things we wanted to see was what really resonated with men and women and how do they differ and in some ways, how were they the same. Take a look at how and watch the squiggles when Joe Biden talks about a timeline for troops and watch when Sarah Palin follows. (BEGIN VIDEO CLIP) SEN. JOE BIDEN, (D) VICE PRESIDENTIAL NOMINEE: Barack Obama and I agree fully and completely on one thing. You have to have a timeline to draw down the troops and shift responsibility to the Iraqis. We’re spending $10 billion a month while the Iraqis have an $80 billion surplus. Barack says it’s time for them to spend their own money, have the 400,000 military we trained for them begin to take their own responsibility and gradually over 16 months withdrawal. John McCain, this is a fundamental difference between us, we will end this war. For John McCain there is no end in sight to end this war. (END VIDEO CLIP) (BEGIN VIDEO CLIP) GOV. SARAH PALIN, (R) VICE PRESIDENTIAL NOMINEE: Your plan is a white flag of surrender in Iraq. That is not what our troops need to hear today, that’s for sure and it’s not what our nation needs to be able to count on. You guys opposed the surge. The surge worked. Barack Obama still can’t admit the surge worked. (END VIDEO CLIP) O’BRIEN: Interesting to see, if you watch those squiggles, the yellow line was the women; women always higher than men we noticed that throughout the entire debate. But on those two things you saw, for Joe Biden as soon as he started talking about timeline and troops, numbers way up. When she’s talked about a white flag and any kind of criticism always rates lowly with dial testers. However, where Governor Palin did get high marks we say in energy and domestic policy, she talked about that, the dials would go back up. I want to quickly check in with the folks whom we started the evening with, Corina Cox and Phil Wegmeir (ph). Corina, let’s start with you. You told us what you wanted to hear, how do you feel about the debate? CORINA COX, REGISTERED DEMOCRAT: I feel good about the debate. I feel that Sarah Palin did satisfy my mind somewhat on the energy crisis. I think that she would be better serving on an energy crisis than Biden or Obama would be. O’BRIEN: Interesting to hear from a registered Democrat. Let’s go to our registered Republican. What did you think of this debate? PHIL WEGMEIR, REGISTERED REPUBLICAN: I thought she hung in there. Most everything was for Joe Biden, he was, I guess, the favorite going into the debate. She hung there in. I felt Biden probably won the debate when you look at just facts and dealing with foreign affairs and trying to address what are we going to do, how we’re going do it versus more general ideas. O’BRIEN: We still have lots of undecideds, Wolf. I’m going to come right up here to the top there. Give me your name and tell me why you’re more undecided than ever. We have only have about 15 seconds. VALERIE GIBSON, UNDECIDED: I’m Valerie Gibson. I’m more undecided than ever because I came in slightly towards John McCain although I have great respect for Obama, and Sarah Palin for me, was not as substantial on the issues. She could have helped John McCain a lot more than she did. His strengths are foreign policy and I don’t think she was as strong as she could have. Now, I’m really more undecided than ever. O’BRIEN: Our undecided panel, some have made, a number actually, probably about a third have decided and will not be joining us again because they’ve now made their decision after the vice-presidential debate — Wolf. BLITZER: We love hearing from so-called real people out there as well as our panel here at the CNN elections. Soledad thank them all for us for hanging in there and for giving us their honest opinions. We love monitoring those little squiggly lines as they were going forward during that 90-minute debate. Soledad is out in Columbus, Ohio. I want to walk over to Alex Wellen because he’s checking the web to see what’s going on; getting lots of reaction from all sorts of folk. Alex, give us a sense of what we’re seeing on the web. WELLEN: There’s place we’ve been talking about a lot of different things — there’s a place called the forum where you can create a political badge. We’ve been getting so much reaction. Let me get just this out of the way. Here is one, for example. “Self-proclaimed weasel;” a supporter of Obama, Iraq, economy, energy, those are the most important issues to this political junkie. He had said or she or presumably he, Sarah may not be sticking to the facts but she expresses herself in a way that the average voter loves.” That is an Obama supporter, interesting. Here we have someone who is a Republican, self-proclaimed, Cisco 17, economy again, number one issue for that person, Iraq, social issues. They do this themselves. We have no way of vetting them. But this is how they say they would have voted. And it’s a political junkie and they say, “It looks and appears someone has done their homework, Senator Joe Biden is giving important and pertinent facts with his responses.” So we’re getting lots of reaction online, people creating their badges. You can do that of course at cnnpolitics.com. BLITZER: Alex thanks very much; lots of useful information at CNNpolitics.com. We’re going to continue our coverage. Get much more reaction. The campaigns weighing in. They obviously think they know who won this debate. |
David W. Moore is a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. His new book is The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.”
Traditional qualitative focus groups (with discussion only and no collection of quantitative data) have the inherent drawback of groupthink, where more dominant or outspoken respondents can influence the opinions of others in the group. By using the Perception Analyzer to capture each respondent’s feedback prior to group discussion, we learn how each respondent individually thinks regardless of influence from others. Then, moving to a group discussion allows for us to dig deeper into why respondents reacted the way they did. It’s that combination of quantitative data collection with qualitative discussion that makes this a more well-rounded, accurate and informative means of research than focus group discussion alone.
That is what our literature is referring to in the caption under the focus group photo in your post. It’s not about ignoring the opinions of more dominant respondents – it’s about not having those opinions influence others before their individual opinions can be shared, thus giving everyone an "equal voice."
Whatever use researchers might have for the squiggly lines, followed up by in-depth discussions with the focus group members after the debate, methinks that the utility to voters of being distracted from the content of the debate, by watching the squiggly lines produced by an unknown, non-representative group of voters, is somewhere between …oh, say, 0 and -100.