Menu

Home » Media Polls»

Woorst polls of the past year.

Every year, poll watchers are confronted with poll results and commentary that defy either logic or science, often raising question about the very utility of polls. Typically, the problems are not with the method of conducting polls, but with the pollsters themselves – as they focus on what they believe is entertaining and appealing to the audience rather than an accurate reflection of public opinion. In the process, pollsters can manipulate public opinion or write commentary that makes a mockery of what the public is really thinking.

With this article, veteran pollsters, authors and political scientists George F. Bishop and David W. Moore issue their Second Annual Top Ten “Dubious Polling” Awards. These awards are intended to mark for posterity some of the most risible and outrageous pronouncements by polling organizations during the previous year.

Each award is ranked, from a low of one set of crossed fingers to a high of five sets. Pollsters generally know in their hearts when all is not right with their polls, but they (figuratively) cross their fingers and hope that no one notices anything amiss. The five crossed-fingers icon is the ultimate in wishful thinking, perhaps the equivalent of football’s “Hail Mary pass” for the truly untrustworthy poll.

 

The “FUZZY MATH” Award

 

WINNER: Fox News Network, for its creative presentation of polling numbers, showing what 120 percent of the public was thinking – and this did not include an additional 15 percent who weren’t thinking! Who knew the public could be a third larger than itself?

BACKGROUND: Only two weeks after declaring a “zero tolerance for on-screen errors,” Fox News had to scrap this draconian policy and instead argue that one of its charts was actually an accurate representation of a Rasmussen poll.

 

 Fox News Chart of Rasmussen Poll

The chart shows 59 percent of Americans who believe it was “somewhat likely” that some scientists had falsified their research, 35 percent who said “very likely” and 26 percent “not likely” – with another 15 percent (not shown in the chart) who had no opinion.

Most third grade math students would no doubt conclude on the basis of these numbers that 120 percent of the public had an opinion, with the public constituting a grand total of 135 percent of itself. That may sound bizarre to some, but apparently not to Fox.

After Media Matters brought this error to Fox’s attention, the network responded to Politico’s Michael Calderone by denying there was a problem. Lauren Petterson, executive producer of Fox & Friends could see no error in the graphic, claiming “we were just talking about three interesting pieces of information from Rasmussen.”

Really? Host Steve Doocy on air noted the two top numbers and mentally added them, saying “you get 90 – you get a lot of people right there thinking it is likely, although 26 percent say ‘not very likely’.” He stumbled on “90” because even he could see that 90+ percent added to the 26 percent figure exceeded 100 percent. Now, how could that be?

How indeed? Better ask those math wizards at Fox.

 

“WISHFUL THINKING” Award

WINNER: Frank Newport, Editor-in-Chief of the Gallup Poll, for his persistent and admirable faith in the American public’s attention to policy details, all polling evidence to the contrary notwithstanding.

BACKGROUND: Right before Christmas, President Obama’s senior adviser, David Axelrod, appeared on Meet the Press hosted by David Gregory and on This Week with George Stephanopoulos to discuss the pending healthcare legislation, arguing – among other things – that many people are not aware of the specifics of the proposals, and that much public opposition measured by polls is thus misleading. Newport took issue with Axelrod’s assertions about public ignorance, by saying: “I’m not aware of current data measuring how well Americans understand what’s in the healthcare bill.”

Wow! He’d been polling for weeks on what people think about the healthcare proposals, and he couldn’t say what they even knew about them? If people didn’t know anything, or were confused about the issue, wouldn’t that be relevant to understanding public opinion? That, at least, was the argument Axelrod was making.

Pollster Newport, however, argued passionately in favor of assuming that people know the issues rather than asking them (now, is that any way for a pollster to act?). His argument: “Healthcare is one issue which is not highly abstract or abstruse for Americans….It is, instead, an issue that is near and dear to most Americans’ daily lives…. It would seem that Americans would be able to understand the ramifications of proposed remedies for this particular policy issue if for no other[sic].”

Well, yes, healthcare itself may not be highly abstract or abstruse for Americans, but the proposed healthcare bill is another story, as Axelrod’s focus groups had shown. Even one of Gallup’s own polls affirmed that point. Newport admitted that last July, a (rogue) Gallup poll (that actually addressed what people knew) showed “about half of Americans said they understood the issues in the healthcare debate.” (That’s a positive way of reporting that 48 percent said they understood the issues, 51 percent said they did not.)

Other polls also showed large proportions of the public confused or uninformed. In mid-September, CNN reported that while 43 percent of Americans said they knew “a great deal” or “good amount” about Obama’s healthcare proposals, 57 percent said they knew “only some” or “not much at all” about them – exactly Axelrod’s point. The same month, CBS reported 59 percent who found the proposed healthcare reforms “confusing.”

It’s a quaint, romantic notion to believe that Americans are fully informed of pending policy proposals and express their considered views through our eminent pollsters, providing what Newport calls “enlightened input” to our legislators. As Axelrod points out, however, confirmed by CNN and CBS (and even Gallup), the truth, alas, is far different. But kudos to those who deny their own polls and believe otherwise.

 

The “OOPS!” Award

  

WINNERS: The Marist, Quinnipiac, and SurveyUSA polling organizations for their relentless campaign portrayal of an easy re-election campaign for New York City’s Mayor Michael R. Bloomberg, and their final predictions of a double-digit victory for the mayor, who actually won by just 4.4 percentage points. Whew! Shades of Dewey/Truman!

BACKGROUND: For weeks leading up to the election, the conventional wisdom was that Bloomberg would win handily, reinforced by polls from these three organizations showing the incumbent with large, double-digit leads. When he actually won with 50.7 percent of the vote to William C. Thompson, Jr.’s 46.3 percent, the city was stunned. So many what ifs…the biggest one being, what if the polls had given an accurate picture of the electorate?

Ben Smith at Politico noted that Bloomberg’s close victory “left Democrats pondering what might have been if New York’s Democratic donors hadn’t turned their back on Thompson, if its politicians had worked for him, and most of all if President Barack Obama had offered anything more than the lamest words of praise.” Thompson was outspent by a 14-1 margin, the city’s top politicians kept their distance, and Obama barely acknowledged Thompson’s candidacy until very late in the campaign.

But why waste the money, the effort, and the president’s political resources when the polls say the race was essentially over before it began?

OK! So, the pollsters didn’t predict the wrong winner. But they predicted the wrong margin, and apparently mischaracterized the electorate throughout the campaign.

Hey, guys! How about a few mea culpas, along with those heavy sighs of relief?

And, by the way, what the @#$% happened?

 

“FOX IN SHEEP’S CLOTHING” Award

   

WINNER: Fox News’ favorite pollster Scott Rasmussen and his eponymous polling organization, for his consistently more negative evaluation of President Obama compared with other polls, all under the guise of being an “independent” pollster. Truth is Rasmussen is a partisan pollster (yikes!). Guess which party?

BACKGROUND: A recent comparison by the Atlantic’s Andrew Sullivan showed that Rasmussen’s reports on Obama’s approval ratings form quite a different pattern from what all other polls (collected by Pollster.com) reveal (see pictures below – the blue line is approval, the red line is disapproval):

 

President Obama’s Job Approval Ratings: Rasmussen vs. Other Polls

Rasmussen had a net negative rating for Obama way back in August 2009, compared with a substantially positive rating by the rest of the polling organizations.

At the end of 2009, the average of all polls on pollster.com, except Rasmussen, showed Obama with an approval to disapproval rating of 51.3% to 42.2% – a 9.1 point positive margin. By contrast, Rasmussen at the same time had approval at 45.2% to 54.3% disapproval, for a 10.1-point negative margin – a swing of 19.2 percentage points. You can see why Fox finds Rasmussen so pleasing to interview.

Some people, of course, have gone after the messenger for his out-in-right-field message, but the pollster argues on his website that “Scott Rasmussen, president of Rasmussen Reports, has been an independent pollster for more than a decade. Like the company he started, Scott maintains his independence and has never been a campaign pollster or consultant for candidates seeking office.”

Never?! Well, tell that to the Washington, D.C.-based Center for Public Integrity, which shows “Scott Rasmussen Inc” as a “consultant” for the Republican National Committee, which paid him $95,500 in 2003-2004; and for George W. Bush, who paid him $45,500 in the same year. (Double Yikes!)

Scott, you sure pulled the fur over our eyes.

 

“HIRED GUN” Award

     

WINNERS: Peter D. Hart Research Associates and John McLaughlin & Associates for their wildly contradictory polls on the “Card Check” bill (the Employees Free Choice Act – EFCA), polls which just happened to agree with the positions of the groups who paid for the polls. Hmmm…how about that for a coincidence?

BACKGROUND: Some critics argue that polls do not reflect so much what the public is thinking, but what pollsters want the public to think. Well, the Hart and McLaughlin polling organizations are here to prove the critics right.

Polling on the question of the “card check” bill, Hart found about three-quarters of American adults in support of allowing employees to have a union once a majority of workers signed authorization cards. By contrast, McLaughlin found three-quarters of American voters opposed. The slight difference in samples – “adults” vs. “voters” – is clearly not enough to account for this titanic difference in opinion measurements.

The important factor to understand is that Hart was polling for the AFL-CIO, which supports the card check bill, while McLaughlin was polling for the anti-card check organization, Coalition for a Democratic Workplace (CDW). No doubt the clients were satisfied. They got what they paid for – the illusion of a public overwhelmingly supportive of their positions.

And what was the public really thinking? Don’t ask. (Hint: Most people haven’t the foggiest idea what this bill entailed – which is why pollsters can make Americans appear to say whatever the clients want.)

 

“THE EMPEROR HAS NO CLOTHES” Award

 

WINNER: The Gallup Organization for its admission that its Six Decade Old Annual “Most Admired” Lists of Living Men and Women do not include the “most admired” people in the world after all. (Say it ain’t so, ghost of George! Have you been lying to us all these years?!)

BACKGROUND: The latest Gallup list of the “Most Admired” Men and Women of the past year was reported as showing Barack Obama the runaway winner among men, with Hillary Clinton edging out Sarah Palin among the women. The controversial ranking, which sent the polling world into an uproar, was conservative talk show host Glenn Beck’s fourth place finish among men, ahead of Pope Benedict in fifth place.

Perhaps no one in the world, except possibly Beck himself, while sobbing in the privacy of his mirror-lined bedroom and lamenting that he alone in the world will “fear for my country,” finds this ranking plausible. But Dana Milbank of the Washington Post appeared to take Gallup at its word, launching his scathing attack of the Fox News host by claiming, “It’s official: Americans admire Glenn Beck more than they admire the pope.”

Pollsters immediately found fault with Gallup’s methodology, claiming that any question that directly asked a representative sample of Americans who they admired more – the pope or Glenn Beck – could not possibly find Beck the winner. Even Beck and his colleagues on Fox News seemed to agree (at least publicly) that there was something fundamentally wrong with the question.

You May Also Like...

Getty Scrubs Caption Error, College Journalist NOT Occupy Wall Street Protestor, But No Correction

Shockingly, Frank Newport of the Gallup Organization concurred, essentially admitting (though in a rambling obfuscatory way) that while “there are a number of ways to measure admiration…blah…blah…blah,” the way Gallup did it, would not – despite the ranking on the Gallup website – “allow one to conclude that Americans admire Glenn Beck more than they admire the pope.”

Well, what did the poll allow one to conclude? According to Newport, “The question basically measures ‘top-of-mind’ brand awareness.” Ah! It measures salience – someone who has most recently been in the news; it doesn’t measure admiration after all.

So, does that mean that other Gallup findings can’t be trusted either? Sarah Palin may not be twice as admired as First Lady Michelle Obama? Tiger Woods’ beleaguered wife, Elin, may not be tied with Germany’s Angela Merkel as the ninth most admired woman in the world?

And what will Gallup do now with more than sixty years of its admittedly bogus data? Our guess: Next year, Gallup will once again announce its list of “most admired” men and women, pretending that this year’s admission never happened. After all, as Newport argued, the question “is based on a historic Gallup precedent.” (Aren’t some precedents just plain bad?)

(Note: Gallup’s founder, Dr. George Gallup, died more than a quarter of a century ago, so he is not forced to witness his old polling firm ‘fessing up to its sixty plus years of playful mendacity.)

 

“PHANTOM OF THE SOAP OPERA” Award

       

WINNER: Strategic Vision, LLC, a national, “Atlanta-based” polling firm whose numbers are suspect, whose location is not in Atlanta, and whose actual polling process (the interviewing) may be bogus. O Strategic Vision, Where Art Thou?

BACKGROUND: Last September, the American Association for Public Opinion Research (AAPOR) censured Strategic Vision for not releasing relevant information about polls it says it conducted in several primary states back in 2008.

The polling firm was co-founded in 2002 by David E. Johnson, described in a 2008 press release as a “veteran Republican pollster and strategist” who “worked on Bob Dole’s 1988 presidential campaign and has overseen numerous campaigns across the nation.”

The New York Times reports the firm was initially intended to be a public relations company, though two years later it branched off into election polling. Its “polls have been cited by numerous news organizations, including The Associated Press, The Washington Post, MSNBC, Fox News and, on at least three occasions, The New York Times, even though the company has repeatedly failed to provide supporting data and the methodology for its surveys.” (emphasis added)

Whoa! Are the media falling down on the job, or what?

After the AAPOR censure, Nate Silver of FiveThirtyEight.com noticed something suspicious about the polling firm’s numbers – disproportionately they ended in “8” more than “1” (such as 58 vs. 51 percent). Silver initially estimated such a pattern could occur by chance only one time in 86 million, though a guest statistician subsequently calculated better odds, but still a daunting one chance in 5,000. Silver concluded there was an overwhelming probability that Strategic Vision had simply made up the numbers without doing the polling. (Mamma Mia!)

Then the situation got mysteriouser and mysteriouser. Ben Smith of Politico reported that all of the offices that Strategic Visions listed on its website – its main office in Atlanta, and several other offices in Madison, Seattle, and Tallahassee – all “match the location of UPS stores, rather than actual offices.” (Double whoa!)

Eventually, Nate Silver, with the help of a blogger on pollster.com, tracked down the location of one office, in the small town of Blairsville (pop. 650), Georgia, about a two-hour drive and 110 miles north of Atlanta, which Silver noted “appears to match the listing for the Seasons Inn Motel & Plaza.”

Was this the “Atlanta-based” firm’s actual location? Apparently, yes. Strategic Vision’s CEO Johnson said the difference between Atlanta and Blairsville was “semantic.” (Really? Tell that to the Blairsville Falcons.)

Another question: Who did all of the polling work and where did they do it? Johnson claimed that his firm either employed or had a central call office in Florida, but declined any additional identifying information. (Why all the secrecy? Did this guy used to work for the CIA?)

To allay charges that his firm had simply made up the numbers (perhaps by issuing results that were similar to polls that had already been conducted), Johnson promised to produce crosstabs of his polling data. These are tables that show how respondents’ answers to questions compare among demographic groups, such as men and women, racial groups, people with different levels of income, and so on. While it may be easy to fake the overall answers to questions, it is difficult to fake such responses among all the demographic subgroups and not get caught.

But so far, no crosstabs for those errant polls.

So, what will happen now? Will Strategic Vision sue AAPOR and Nate Silver as Johnson intimated? Will Strategic Vision actually produce its crosstabs? How will it ever explain its one-in-5,000 numerical pattern? Will we ever discover the location of its many offices? Does it have real interviewers? Is this a phantom or a real polling firm?

Stay tuned. The soap opera isn’t over until the thin man sings!

 

“BOBBLEHEAD POLLING” Award

     

WINNERS: The CBS News, ABC/Washington Post, CNN, and USA Today/Gallup polls – for their seemingly random findings on the public’s reaction to President Obama’s health care speech in September. Like a bobblehead doll, the public appeared to nod first in one direction, then another, then back again – at least according to these esteemed media organizations. (Can we really believe these guys?!)

BACKGROUND: On Wednesday, September 9, 2009, President Obama gave a nationally televised speech before the Congress on the subject of healthcare, outlining in some detail the kinds of provisions he wanted in a healthcare reform bill. Two days later, a CBS poll declared the public had rallied in favor of the president with an astounding 21-point swing in approval rating on the issue. (But wait!)

The following Monday, an ABC/Washington Post poll came to the opposite conclusion: The “bottom line views on health reform,” according to that analysis, “failed to improve since President Obama addressed the nation.” (Wait wait! There’s more!)

At 3:00 PM that afternoon, CNN reported a 13-point favorable swing in Obama’s approval rating since the president’s speech. (Don’t go away!)

The very next day, a USA Today/Gallup poll bobbled in the opposite direction – Obama’s speech “didn’t change minds” after all and “for the first time” a majority of Americans actually disapproved of the way the president was handling health care policy.

What’s a bloke to think? Is the public a bobblehead? Or are those pollsters manipulating their respondents? (You decide – we’ve already made up our minds!)

 

“STONEWALLING/COVERUP” Award

       

WINNERS: The John Hopkins Bloomberg School of Public Health and one of its professors, Dr. Gilbert Burnham, for stonewalling in the face of serious questions about a flawed survey project, which reported more than 600,000 Iraqi deaths from 2003 to 2006. The head researcher was formally censured by the American Association for Public Opinion Research (AAPOR) for covering up his data collection efforts, but the Bloomberg School refuses to investigate the methodology. (Ah, the wisdom of the three monkeys: “See no evil, hear no evil, speak no evil!”).

BACKGROUND: In 2006, the British medical journal, The Lancet, published the results of a survey, designed and supervised by Dr. Gilbert Burnham of the John Hopkins Bloomberg School of Public Health, and his colleagues.* The survey purported to show that about 600,000 Iraqi deaths occurred in Iraq by July 2006, as a consequence of the invasion of Iraq.

A lot of people were against the war, but jacking up the body count with bad studies is not a good tactic for anyone. According to economics professor Michael Spagat of Royal Holloway College, these results were anywhere from seven to 14 times as high as other credible estimates, including those made by the non-partisan Iraq Body Count, a consortium of U.S. and U.K researchers, also concerned about the human toll of the war.

Such large differences in estimates led other researchers to question the methodology of the study. But contrary to scientific norms, Burnham refused to provide details about how the survey was conducted. When a complaint was lodged with AAPOR, its standards committee also tried to obtain such details, but was rebuffed. That led to the censure.

What exactly were John Hopkins Bloomberg School, and Burnham, et. al., hiding? AAPOR asked for the kind of information that any scientist doing this type of work should release: a copy of the questionnaire, the consent statement that interviewees have to see, a full description of the selection process, a summary of the disposition of all sample cases, and how the mortality rate was calculated.

John Hopkins Bloomberg School initially stood behind the study, but then eventually concluded that Burnham had made some unauthorized changes in his methodology, and thus “the School has suspended Dr. Burnham’s privileges to serve as a principal investigator on projects involving human subjects research.”

But the Bloomberg School has not come clean with the problems of the research project. Their press release admitted that their internal review “did not evaluate aspects of the sampling methodology or statistical approach of the study.” Instead, Bloomberg asserts, “It is expected that the scientific community will continue to debate the best methods for estimating excess mortality in conflict situations in appropriate academic forums.”

Let’s see: The Bloomberg School will not attempt to evaluate what experts believe is almost certainly a faulty methodology, saying the scientific community should make the evaluation. But then the school advises Burnham not to release details about his methods, so the scientific community can’t have the information it needs for a definitive assessment.

Sounds like a cop-out and a Catch 22, all rolled into one!

And we thought Richard Nixon was tricky.

* Burnham G, Lafta R, Doocy S, Roberts L. 2006a. ‘Mortality after the 2003 invasion of
Iraq: a cross-sectional cluster sample survey’. The Lancet 368:1421-1428. It can be accessed online at http://brusselstribunal.org/pdf/lancet111006.pdf.

 

“MANUFACTURED OUT OF WHOLE CLOTH” Award

       

WINNERS: Zogby, Ipsos/McClatchy, ABC/Washington Post, Associated Press/Stanford University, Pew Research Center and CNN/Opinion Research Corporation polls, for their valiant efforts to manufacture a “public opinion” on “cap and trade” legislation, though most people haven’t a clue.

BACKGROUND: Recent poll suggest that so few people are aware of what “cap and trade” programs are, it is pure fantasy to think of a “public opinion” on the issue. Yet, somehow, many pollsters have been able to fabricate the illusion of a public that is so highly engaged and informed, the vast majority of people have a position with respect to the issue.

But let the people guess what “cap and trade” means among three possible answers – healthcare, banking reform, or the energy and environment – and less than one in four can accurately point to the latter. Or ask them how much they’ve heard about “a policy being considered by the president and Congress called ‘cap and trade’ that would set limits on carbon dioxide emissions,” and only one in seven can say “a lot,” while more than half say “nothing at all.”

So, how do pollsters create the illusion of an engaged public? Simple: They feed their respondents information, and then immediately ask what respondents think, using “forced-choice” questions to get an answer.

The key point here is that none of these polls can any longer represent the American people! Once pollsters give information to their respondents, their samples no longer represent the larger population, which has not been fed that same information.

Of course, different polling organizations feed different information to their respondents, which is how the polls run the gamut of opinion. At one end of the spectrum is a Zogby poll (conducted for a global warming skeptic), showing almost a two-to-one opposition to cap and trade legislation, while at the other end of the spectrum is a CNN poll finding about a two-to-one level of support. Other polls fall somewhere in-between. So, fellow pollsters, fabricate away! Just stop pretending you report what the American public is thinking. You know better. We (should) know better, too.

 

UPDATE: 03/02/2010: 4:20PM EST: For those of you in Ohio who would like to hear David Moore speak, he will be giving give a lecture, 2:30 p.m., Thursday, March 4, 2010, sponsored by the University of Cincinnati, Charles Phelps Taft Research Center and the Department of Political Science in the McMicken College of Arts and Sciences (A&S).

David Moore, a senior fellow and political scientist at the Carsey Institute of the University of New Hampshire, will speak on the virtues and foibles of public opinon polls. His title? “How Public Policy Polls Undermine Democracy.” The talk is free and open to all and will be held at the Taft House at Stratford Heights, 2625 Clifton Ave. The focus of Moore’s lecture will be “on how mass media-sponsored public opinion polls on  health care reform and ‘cap and trade legislation’ often undermine the democratic process in American politics and society” according to a university press release.

 

George Bishop is Professor of Political Science and Director of the Graduate Certificate Program in Public Opinion & Survey Research at the University of Cincinnati. His most recent book, The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls (Rowman & Littlefield, 2005) was included in Choice Magazine’s list of outstanding academic titles for 2005 (January 2006 issue).

David W. Moore is a Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of  The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008; trade paperback edition, 2009). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout.”

Submit a tip / Report a problem

2010 Top Ten ‘Dubious Polling’ Awards: Year’s Problematic Polls

Share this article:

13 Responses

  1. Bob says:

    Wow, what a surprise. Almost all the "awards" go to polls or polling or news organizations that dare to show Obama and/or his policies in a negative light. Because, obviously, the majority of Americans agree with him and love him and any poll that doesn’t demonstrate such must be suspect. Those polls that ridiculously interview 15% to 23% more Democrats and then breathlessly tell us how popular Obama is (was is more accurate now) or claim that only 17% of the country calls itself Republican? Not even a mention.

  2. David Moore says:

    Bob, perhaps if you look again, you’ll see that the partisan findings of the poll are not our concern, just the validity of the findings or comments. The Manufactured Out of Whole Cloth Award, for example, criticizes poll findings that happen to support the Obama administration’s approach to dealing with global warming (the so-called "cap and trade" approach), but that we think are mostly "manufactured" opinion. The Stonewalliing/Coverup Award takes issue with a study widely criticized by conservatives on substantive grounds, though our only concern is that the Bloomberg School of Public Health and Dr. Gilbert Burnham refused to come clean with the report’s methodology. The Bobblehead Polling Award is critical of the various pollsters for their wildly different findings about the effect of Obama’s health speech to the nation, not for the partisan leanings of the findings. We skewer Fox News for its Fuzzy Math in reporting a poll, but we skewer all the other major media polls as well — and in every case, our concern is the flakiness of the polling or the poll reporting, not the results per se.

  3. Michael Alao says:

    Bob, you just don’t get it, do you? This isn’t a "liberal media" attack, it’s academics pointing out polls that are clearly flawed.

  4. George Bishop says:

    Bob,

    Dave’s comments reflect my thinking as well. To take another example, last year we nailed Peter Hart Research Associates, a well-known Democratic polling firm, for conducting an advocacy poll that showed the public in favor of bailing out the auto industry. Big surprise, eh? So I think we’ve been pretty even-handed in finding flawed polls and inflicting comfortable pollsters of whatever stripe. Judge us by our full record.

    George Bishop

  5. Roy says:

    I am just plain not buying that EVERY poll on "cap-and-trade" was like that. It’s not really all that complicated: which do you put a higher priority on, lower energy prices or taking action on global warming? You can have lots of good laughs at what the public thinks "cap-and-trade" means, but I’m pretty sure plenty of people have asked more specific questions about global warming, gas prices, etc etc, and those with a vested interest in accurate answers – politicians – have drawn their conclusions.

    This site may not be Media Matters(or to be bipartisan, the Media Research Center), but that "award" is pretty non-rigorous and sure does sound to me as if somebody wishes the public’s opinion weren’t what the polls say it is.

    Or to put it another way, I’d be willing to put down a fair sum that the one who decided on it is in favor of anti-global warming legislation.

  6. David Moore says:

    Roy,

    The issue is not whether there can be good polls on the issue of cap and trade, but whether the pollsters cited have provided such polls. Go to http://www.pollingreport.com and do a search under "cap and trade" to see how the issue is addressed by the various polling organizations. Our concern is that the pollsters refused to measure both opinion and non-opinion, giving the impression that 90+ percent of the public has a developed opinion on the matter.

    That pollsters have to give information to the respondents before asking the questions is a clue to the fact that a large segment of the public simply has not focused on the issue enough to have an idea about whether or not cap and trade is a viable policy. When pollsters give information, they taint the sample and essentially manipulate their respondents toward one direction or another.

    You suggest the issue is "not all that complicated" — just a matter of whether one puts a "higher priority on lower energy prices or taking action on global warming" — but I would argue even that trade-off is not simple. How higher would the energy prices be (and which "energy" — gasonline, heating oil, natural gas?) for how much lower global warming? This is a tricky trade-off with no precise figures, and it’s folly to assume that the vast majority of the people have thought about such issues.

    That many people have not thought about the issue is reflected in the ability of pollsters to get diametrically opposed results (see the Zogby and CNN poll results we cite in the explanation of the award). Many respondents are influenced by the questionnaire itself, because they don’t have a developed opinion of their own.

    You say you’d be willing to bet "a fair sum" that the person who decided to include this award "is in favor of anti-global warming legislation." You’d lose that "fair sum." It’s the illusion of public opinion we oppose, not efforts to stop global warming.

  7. Rick says:

    Talk about manufactured public opinions! The sad thing is that when you present the public with hard numbers of what the "public" thinks or believes, then you create a leveling effect, so that more people will actually align their beliefs with what they think others think. This is actually a propaganda technique called the "bandwagon" theme.

    So, this isn’t simply a matter of inventing public opinion, but it is also a matter of producing that opinion.

  8. frizzled says:

    “STONEWALLING/COVERUP” Award

    The claims made about the Lancet survey are nonsense. In fact, anyone who reads the linked Johns Hopkins press release will see that Dr Burnham’s study violated procedures by recording the names of subjects. This doesn’t bring into question any of the conclusions of the study, which have been available to outside researchers for years – again this is on the Johns Hopkins site.

    The independent Iraq Body Count study counted only those names which have been reported in print newspapers and is in no way a substitute for an independent statistical survey. Furthermore, yet another statistical survey conducted in 2007 by ORB in Iraq confirmed that about 1,000,000 Iraqis have died in the conflict. Considering that 5 million refugees have been created by this war these figures are not surprising. It is also not surprising that there have been constant politically motivated attacks on reporting these facts by neocons and other pro-war types. But that doesn’t alter the truth.

  9. David Moore says:

    >To Frizzled —

    You are correct in saying that the John Hopkins Bloomberg School of Public Health suspended Dr. Burnham from acting as a principal investigator in projects involving human subjects because he recorded names of subjects, a violation of the study procedures that had been approved.

    You are also correct in saying that this particular violation does not itself detract from the study’s findings. We never made such a claim in our description of the award.

    The STONEWALLING/COVER-UP award we granted to Burnham and to the Bloomberg School was earned because of Burnham’s and the Bloomberg School’s refusal to release details of the methodology of Burnham’s study. As we noted in the background to the award, the American Association for Public Opinion Research censored Burnham specifically for that refusal. What kind of information did Burnham and Bloomberg refuse to release?

    "AAPOR asked for the kind of information that any scientist doing this type of work should release: a copy of the questionnaire, the consent statement that interviewees have to see, a full description of the selection process, a summary of the disposition of all sample cases, and how the mortality rate was calculated."

    Scientists who make claims about their scientific findings, but who then refuse to reveal their basic methodology, cannot expect the public to take their claims seriously.

    Some people claim that the procedures used by Burnham led to a much higher estimate of Iraqi deaths than is warranted. The only way to address those criticisms is if Burnham were to reveal exactly what his procedures were, so that other scientists can made an objective assessment. His refusal to release such methodological detail obviously leads to this question: What is he hiding?

    This is not a political issue, but a scientific one. People who both supported the war, and many others who opposed the war, are equally critical of Burnham’s refusal to reveal his methods, as any scientist would be expected to do.

  10. frizzled says:

    Thank you for your reply David. Still, I maintain that you are distorting the truth. You wrote,

    "A lot of people were against the war, but jacking up the body count with bad studies is not a good tactic for anyone."

    Since you have admitted the ethical problems Burnham was censured for were not relevant to the validity of his study’s conclusions, how does it follow that his study was "jacking up the body count"?

    You wrote,

    "According to economics professor Michael Spagat of Royal Holloway College, these results were anywhere from seven to 14 times as high as other credible estimates, including those made by the non-partisan Iraq Body Count, a consortium of U.S. and U.K researchers, also concerned about the human toll of the war."

    But the usual method for estimating country-wide excess mortality is to perform a statistical survey. Far from being a legitimate statistical survey, the Iraq Body Count project maintains a collated list of deaths reported in print news. We would not expect this count to be a credible estimate of total nation-wide excess mortality at all. Why are you claiming that it is?

    In fact there has been a more recent statistical survey of civilian deaths in Iraq that corroborates the Lancet study remarkably well. The recent independent ORB survey found that 1.2 million Iraqi civilians have died during the war, consistent with Burnham’s figures. See http://www.latimes.com/news/printedition/asection/la-fg-iraq14sep14,1,1207545.story?coll=la-news-a_section&ctrack=1&cset=true . Why did you not mention this surely relevant fact in your article?

    Finally you wrote,

    "People who both supported the war, and many others who opposed the war, are equally critical of Burnham’s refusal to reveal his methods, as any scientist would be expected to do."

    But it says on the linked Johns Hopkins website "The original protocol explicitly stated that no names of study participants or living household members would be collected…. These numerical databases have been available to outside researchers and provided to them upon request since April 2007."

    In other words, Burnham has made his full data available to other scientists already after 2007. While I’m also critical of Burnham violating some details of his protocol and not being more forthcoming it seems that he corrected this problem several years ago. Why then this criticism well after the fact in 2010?

  11. David Moore says:

    To Frizzled –

    I appreciate your interest in this discussion, but perhaps there is a misunderstanding.

    In the previous post, I acknowledged that the steps taken by the John Hopkins Bloomberg School of Health against Burnham were not relevant to the validity of his study. But I noted a second censure by the American Association for Public Opinion Research, which asked Burnham to outline in detail the method by which he collected his data and made his estimates of the body count. He refused to do so, and for that reason, AAPOR issued its censure.

    I believe that this AAPOR censure (which was announced in 2009, the year for which our “Top Ten ‘Dubious Polling’ Awards” are intended) is in fact relevant to the credibility and validity of Burnham’s estimates.

    As stated in AAPOR’s announcement of the censure, “AAPOR holds that researchers must disclose, or make available for public disclosure, the wording of questions and other basic methodological details when survey findings are made public. This disclosure is important so that claims made on the basis of survey research findings can be independently evaluated. Section III of the AAPOR Code states: ‘Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted.’”

    The release also noted that “AAPOR found that Burnham, a faculty member at the Johns Hopkins Bloomberg School of Public Health, repeatedly refused to make public essential facts about his research on civilian deaths in Iraq. In particular, the AAPOR inquiry focused on Burnham’s publication of results from a survey reported in the October 2006 issue of the journal Lancet. When asked to provide several basic facts about this research, Burnham refused.”

    The release went on to quote AAPOR’s President, Richard A. Kulka, “When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research.”

    To this day, neither Burnham nor the Bloomberg School of Public Health has released the specific information requested by AAPOR.

    This lack of transparency by Burnham is the crux of the issue in our STONEWALLING/ COVER-UP Award. Yes, we implied that Burnham was “jacking up the body count,” but we are quite willing to be proven wrong. Let Burnham provide the requested details as to how he counted deaths in Iraq, and if other scientists agree this was a reasonable approach, we’re more than willing to admit we were wrong.

    In the meantime, we believe that his continued stonewalling on this matter more than justifies our initial award.

    Updated: 3-19-2010. 1:01pm

  12. David Moore says:

    To Frizzled, I appreciate your interest in this discussion, but perhaps there is a misunderstanding.

    In the previous post, I acknowledged that the steps taken by the John Hopkins Bloomberg School of Health against Burnham were not directly relevant to the validity of his study. (Perhaps, even here, I was hasty, because it can reasonably be argued that if Burnham was not being forthright about recording the names of respondents, in contravention to the protocol he said he had been following, then it is difficult to trust any of his numbers.)

    Nevertheless, in the original post, I noted a second censure, which you have ignored altogether. That one was by the American Association for Public Opinion Research, which asked Burnham to outline in detail the method by which he collected his data and made his estimates of the body count. He refused to do so, and for that reason, AAPOR issued its censure.

    I believe that this AAPOR censure (which was announced in 2009, the year for which our “Top Ten ‘Dubious Polling’ Awards” are intended) is in fact directly relevant to the credibility and validity of Burnham’s estimates.

    "As stated in AAPOR’s announcement of the censure, “AAPOR holds that researchers must disclose, or make available for public disclosure, the wording of questions and other basic methodological details when survey findings are made public. This disclosure is important so that claims made on the basis of survey research findings can be independently evaluated. Section III of the AAPOR Code states: ‘Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted.’”

    The release also noted that “AAPOR found that Burnham, a faculty member at the Johns Hopkins Bloomberg School of Public Health, repeatedly refused to make public essential facts about his research on civilian deaths in Iraq. In particular, the AAPOR inquiry focused on Burnham’s publication of results from a survey reported in the October 2006 issue of the journal Lancet. When asked to provide several basic facts about this research, Burnham refused.”

    "The release went on to quote AAPOR’s President, Richard A. Kulka, “When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research.”

    When Burnham’s findings were published, many critics expressed skepticism about their validity because they diverged so radically from other credible estimates. As evidence for this assertion, we cited this article, which also cited Michael Spagat. You have argued that another study, the ORB survey, “corroborates” Burnham’s study, but that is missing the point. We are not arguing which study is correct, but rather that if Burnham wants his study to be viewed seriously, he must make available to the scientific world the methods by which he obtained his estimates. And that he refuses to do.

    To this day, neither Burnham nor the Bloomberg School of Public Health have released the specific information requested by AAPOR.

    This lack of transparency by Burnham is the crux of the issue in our STONEWALLING/ COVER-UP Award. Yes, we said that “jacking up the body count with bad studies is not a good tactic for anyone,” because it appears to us that in fact Burnham did just that. His refusal to come clean with his method of making estimates is clearly suspicious, regardless of what other studies show (and here there is much conflict among the various studies’ results).

    The only way to allay suspicion is for Burnham to provide the requested details as to how he counted deaths in Iraq, and to let other scientists assess the utility of those methods.

    In the meantime, we believe that his continued stonewalling on this matter more than justifies our initial award.


  13. captaingrumpy says:

    Look.lets leave a dead horse to die quietly. He was withholding poll information that was requested. He did NOT give up the information when asked. So,he fixed it YEARS after he was told to. Gives him time to rewrite the whole bloody report, doesn’t it !

Comments Terms and Conditions

  • We reserve the right to edit/delete comments which harass, libel, use coarse language and profanity.
  • We moderate comments especially when there is conflict or negativity among commenters.
  • Leave a Reply

    Your email address will not be published. Required fields are marked *