Editor’s Note : David Moore, a former Gallup pollster and a iMediaEthics poll columnist, criticized Quinnipiac in March 2011 for its polls on New Yorkers and bike lanes. Moore specifically criticized the poll for asking a biased question and for not measuring how intensely people feel about the issue – i.e., whether or not they genuinely care about the issue. To illustrate how an objective survey should be conducted, in early August, Art Science Research Laboratory, which publishes iMediaEthics , contracted with survey company SurveyUSA to poll New Yorkers with questions Moore regards as appropriate for understanding what the public is really thinking. The report below includes Moore’s analysis of the ASRL findings compared with those bike lane polls from Quinnipiac and Marist. See below Moore’s bi-monthly Poll Skeptic column, and see here links to the ASRL’s methodology and the poll’s Full Report.
Should we believe two polling organizations that claim New York City residents are solidly supportive of the city’s efforts to expand bicycle lanes?
While Quinnipiac and Marist both have reported this year that New Yorkers are in favor of bike lane expansion (59% and 66%, respecitvely) Art Science Research Laboratory found that most New Yorkers simply don’t care.
Using rigorous polling methods, Art Science Research Lab (ASRL, the publisher of iMediaEthics.org*) determined that a majority of New Yorkers (56%) are indifferent toward bike lane expansion — not caring whether it occurs.
The rest of the residents are about evenly divided – approximately one fifth (23%) intensely supportive and another fifth (21%) intensely opposed. Our methods differs from Quinnipiac and Marist by using objective questions to measure opinion, including how intensely residents feel about the issue.
Quinnipiac Poll vs. Reality
Quinnipiac was the first to poll on the issue back in March when it announced that New Yorkers supported the expansion of bike lanes in the city by a 54% to 39% margin.
At the time, I took issue with the findings, suggesting that Quinnipiac was stacking the deck by the wording of their questions. I also suggested that they needed to measure how many people have even heard of the issue, what experience they personally have had, if any, with the new bike lanes, and how intensely they feel about the issue.
Since then, Quinnipiac has conducted two more polls: one in May and another in July. In response to my criticisms, the Quinnipiac pollsters now include a question that measures how much people have heard about the expansion of bike lanes.
(When asked why they didn’t include such a question the first time, Mickey Carroll, Director of the Polling Institute, responded via e-mail: “We didn’t ask an ‘are you aware’ question because it would be superfluous about something that had been all over the New York news.” Apparently, it wasn’t a superfluous question after all, because when they did ask it in their next poll, they found only about a third (37%) of New Yorkers who had heard “a lot” about the issue. Moreover, support declined the more people knew about the bike lanes. In fact, even with their biased question, a majority of those who had heard “a lot” about the issue actually opposed the expansion.)
Still, Quinnipiac has ignored my other criticisms, continuing with the use of their biased question about the expansion of bike lanes, and refusing to measure intensity of opinion. Research has shown that many people will offer an opinion in a poll, even though – if given the option – they will readily admit they really don’t have a preference one way or another. To understand the public, it is crucial to measure whether people are firmly anchored to their opinion or not.
With the continued use of their biased question and refusal to measure intensity, Quinnipiac reported an “increase” in public support in July, with 59% saying the expansion of lanes was a “good thing,” compared with 54% who said that the previous March.
ASRL Poll vs. Quinnipiac Poll
It’s one thing to hypothesize that the Quinnipiac Poll is biased. It’s another thing to demonstrate it in practice. In order to illustrate the principles of polling we think are important, the Art Science Reserach Laboratory (ASRL) contracted with SurveyUSA to conduct a poll of New York City residents on the subject of bike lane expansion, following a methodology that we believe produces a more realistic picture of the public.
SurveyUSA interviewed 898 city adults from Aug. 4-8 using voice-recorded interviewers for 66% of the interviews and live interviewers for the rest.
Half the sample of respondents in the ASRL poll were asked the Quinnipiac question, and the ASRL results were virtually identical with what Quinnipiac reported in its July poll. Among registered voters, Quinnipiac reported that 59% felt the expansion of bike lanes was a “good thing,” and 35% a bad thing, while ASRL found comparable figures of 60% to 35%.
What these results demonstrate is that both the ASRL poll and the Quinnipiac poll reached approximately the same types of New York City residents. Though two-thirds of the ASRL respondents were interviewed by recorded voice and a third by live interviewers, while all of Quinnipiac’s interviews were conducted by live interviewers, the mode of interviewing apparently had no effect. ASRL got virtually the same results as Quinnipiac when using the same question.
For the other half of the sample, ASRL asked its own question about bike lanes. Unlike Quinnipiac, ASRL offered an explicit “no opinion” option. The Quinnipiac question used a “forced choice” format – accepting a “no opinion” response only if volunteered, but not offering one explicitly. This approach subtly “forces” (pressures) the respondent into accepting one of the two acceptable options, thus underestimating the percentage of people with genuine non-opinions.
The net result is that respondents asked the ASRL question (which gave an explicit option to say they had “no opinion”) were much more likely to admit they didn’t have an opinion — 28% compared with just 4% who were asked Quinnipiac’s forced-choice question.
Though “no opinion” was offered in the question, it’s still essential to measure “intensity” of opinion. Many people may express an opinion, but often they do so without any real conviction. Especially in the context of a survey, where people are expected to have an opinion (that’s the whole point of the poll, after all), many respondents may come up with an opinion to meet interviewer’s expectations, but still not really care about the issue one way or the other.
To determine if that was the case, we asked respondents who said they favored the expansion of bicycle lanes how upset they would be if there were no expansion. And we asked people who opposed the expansion how upset they are because of the expansion.
These follow-up questions were asked of the full sample of respondents who indicated a positive or negative view of the bicycle lane expansion – including respondents who were asked the Quinnipiac question as well as respondents asked the ASRL question. (The questions weren’t posed to respondents who stated they had no opinion on the bike lane expansion.)
The scale for the upset question was: very upset, somewhat upset, not too upset, or not at all upset. People who said “very” or “somewhat” were classified as “upset,” while people who said “not too” or “not at all” were classified as “not upset.”
Respondents who say they are “not upset” with the opposite policy from the one they just said they preferred are classified as people with essentially “non-opinions.” The logic here is straightforward: People who say that they “oppose” the expansion of bike lanes, but then readily admit that they aren’t upset that there is an expansion, are apparently not concerned about the issue one way or the other. Their initial opinion, we can surmise, was a top-of-mind response, with no real impact on their way of thinking.
The same logic holds for people who say they “favor” the expansion of bike lanes, but then say they wouldn’t be upset if the expansion did not occur. For practical purposes, they essentially don’t care one way or the other – whether bike lanes are expanded or not.
The net results are as follows:
Note the similarity in results using the Quinnipiac question and the ASRL question – once the intensity questions have been asked. A little more than a fifth (23%) of New Yorkers are positive about the expansion of bike lanes and say they would be upset if there were no expansion. Another fifth (22% or 20%) express negative opinions about the bike lanes and are upset that there is an expansion
The rest of New Yorkers aren’t upset that there is an expansion, but they also wouldn’t be upset if there were no expansion. Essentially, they don’t care one way or the other.
After the ASRL poll had already begun, Marist published the results of its July poll, showing “Two Thirds Favor NYC bike lanes.”
How did Marist get figures even higher than Quinnipiac? Two factors help explain the difference: 1) Marist’s results are based on all residents, not just registered voters (that accounts for one or two percentage points), and 2) Marist didn’t ask about the expansion of bike lanes, but about bike lanes in general: “In general, do you support or oppose bike lanes in New York City?” It clearly does not address the issue of expansion of bike lanes.
Marist’s subsequent question did address that issue: Only 27% wanted to “add more bike lanes,” while 23% wanted to see them reduced. Still, the headline out of Marist was the two-thirds support for bike lanes in general.
Like Quinnipiac, Marist asked a forced choice question about bike lanes and failed to measure intensity of opinion. And like Quinnipiac, Marist wildly overestimates the percentage of residents with a genuine opinion.
Other major findings of the ASRL poll:
People who have heard “a lot” about the expansion of bike lanes oppose the expansion by a 12-point margin (35% oppose, 23% favor), with 42% indifferent. Residents who have heard only “some,” “not much,” or “nothing at all” tend to favor the expansion, by an average of about 8 percentage points (23% to 15%), with 62% indifferent.
New Yorkers who are most upset with the expansion are people who drive cars every day along streets and roads with the bicycle lanes. They represent about a fifth (19%) of city residents. They oppose the expansion by 37% to 22%, with the rest (41%) not upset one way or the other.
The strongest supporters of bicycle lanes, of course, are frequent bike riders. They represent about 7% of adults in the city – people who report riding in the bicycle lanes at least a few times a week or more. They support the expansion of lanes by 58% to 16% (with the rest indifferent).
Infrequent riders support the expansion of bicycle lanes by 33% to 16%, with just over half not caring one way or the other.
People who never ride bicycles are mostly unconcerned about the issue (61%), while the rest of the non-riders lean a bit more negatively (23%) than positively (16%).
The frequency of walking along streets with bicycle lanes shows no correlation with attitudes about the expansion of those lanes.
New Yorkers who are engaged in the issue (those who would be upset if their preferences were not to be adopted) are about evenly divided as to the future: About 39% would like to see more bicycle lanes, while 41% would like to see either fewer lanes (26%) or none at all (15%). Another 20% are satisfied with the current number.
One argument against bike lanes is the high potential for accidents. The poll finds that three quarters (77%) of New Yorkers have witnessed a dangerous situation in the city involving a bike at least once in the past year – such as the bicycle going the wrong way on a street, a bicycle riding on a sidewalk, or a bicycle coming close to striking a pedestrian, car or cyclist.
Almost a third (31%) have witnessed such a situation more than five times in the past year, and among opponents of bike lanes, that number rises to 46%.
Does it matter which view of the public that pollsters present? The view presented by Quinnipiac and Marist is of a public that is widely engaged with clear opinions. That’s simply unrealistic. Of all the problems and issues New Yorkers face on a daily basis, the notion that virtually all of them are immersed in the issue of bicycle lanes defies credulity.
Marist and Quinnipiac also suggest that a large majority of the public is firmly behind the city’s efforts to expand the bike lanes. But that “support” is largely an illusion created by biased poll questions, where respondents are pressured to come up with opinions, no matter how shallow they may hold them.
The view presented by the ASRL poll is far more realistic – a public that is mostly unengaged in the issue of bicycle lane expansion, but among people who are engaged, about an equal number intensely in favor and intensely opposed to the expansion policy.
The first view gives the impression that the policy of bike lane expansion is legitimate because a majority of residents are in favor. It allows supporters to deflect criticism of the policy overall, and the way it’s being implemented, by pointing to alleged widespread public support.
The second version points to a realistic view of where the public stands, but it also identifies where the points of contention are. Most policies are probably not supported by majorities of the public – the average citizen hasn’t the time or expertise to be engaged on all issues. Nor are most policies opposed by majorities, despite what the polls show. We are deluded about public support and opposition, because most media pollsters refuse to measure and take into account how intensely people feel about the issues.
The issue of expanding bike lanes is one that needs to be addressed by all parties concerned — as it is in New York City. But polls that manufacture consent should not be used to intrude into the process and detract from those who have legimitate concerns about whether, and how, the bike lane policy should be implemented.
The public, the government, the media — we all — need a realistic assessment of public opinion on the issues, not the kind of superficial poll-manufactured public opinion that is too often treated as the real deal.
This Poll is part of iMediaEthics’ PollCheck project to fact check media polls.
More iMediaEthics polls found here
*Art Science Research Laboratory, www.asrlab.org, is not-for-profit, co-founded with the late Harvard professor and scientist, Stephen Jay Gould, and his wife, sculptor, art historian and journalist, Rhonda Roland Shearer. It has a non-partisan journalism ethics program in which students and young journalists work with professional researchers to promote the media’s use of scientific methods and experts before publication. ASRL also publishes investigations of factual errors and ethical breaches by media outlets on www.iMediaEthics.org. Alexa ranks iMediaEthics.org among the Top 20 most visited news media watchdogs.
David W. Moore is a Senior Fellow with the Carsey Institute at the University of New Hampshire. He is a former Vice President of the Gallup Organization and was a senior editor with the Gallup Poll for thirteen years. He is author of The Opinion Makers: An Insider Exposes the Truth Behind the Polls (Beacon, 2008; trade paperback edition, 2009). Publishers’ Weekly refers to it as a “succinct and damning critique…Keen and witty throughout