Mainstreet to release findings of investigation into 'big polling failures' during Calgary election

For the latest update on this story, seeMainstreetsays Calgary election polling failures caused by 'perfect storm'

Also check out this story:Mainstreet Research considers legal action against Nenshi's campaign pollster

The polling firm that consistently and wrongly predicted Calgary Mayor Naheed Nenshi would lose in October's election willrelease the findings of an internal investigation into its "big polling failures" on Monday.

Before the Oct. 16 vote that saw Nenshi re-elected to a third term, Mainstreet Research released three surveys on behalf of Postmedia, which owns the Calgary Sun and Calgary Herald. All suggested Nenshi was headed for defeat at the hands of a relatively unknown challenger, Bill Smith.

• Poll puts Bill Smith in the lead over Naheed Nenshi in Calgary's mayoral race
• Bill Smith leads Nenshi by 17 points in latest poll
• Smith leads Nenshi by 13 points in final Mainstreet poll before Calgary election

Many long-time political watchers and polling experts questioned Mainstreet's results during the campaign. After the vote, Mainstreet admitted its polling failed to connect with some younger voters and those who don't speak English as their first language.

Mainstreet also announced its head of analytics, Joseph Angolano, would conduct a review of the firm's polling methods.

In an interview last week, Quito Maggi, the president and CEO of Mainstreet Research, told CBC News that a panel of five independent experts including a mathematician, a political scientist and a retired pollster have also reviewed Angolano's investigation.

Maggi expects some of the experts' comments about the internal investigation to be released on Monday.

Report found firm wasn't reaching potential Nenshi supporters

Maggi says he learned something from reading Angolano's report.

He says his firm wasn't reaching people who were inclined to vote for Nenshi, meaning the company's results underestimated the incumbent mayor's voters.

"That non-response bias was so significant," said Maggi. Non-response bias happens when there is a difference between those who participated in the poll and those who were excluded.

"We've never seen something where the non-response bias is so much skewed in one direction."

Mainstreet conducted Interactive Voice Response (IVR) polling, which is also known as robo-call or push button polling.

Bruce Cameron, the president of the polling firm Return on Insight, said he believes the IVR method is good for identifying supporters, but does not adequately predict voter intentions for a large population.

"It's not good for polling," Cameron told CBC News.

Cameron says IVR has a low response rate and tends to over-represent older, more conservative-leaning voters.

Beyond the methodological critiques of IVR, Mainstreet Research's polling data especially when analyzed by sub-groups such as younger voters also raised many eyebrows among long-time political watchers and polling experts.

University of Calgary political scientist Melanee Thomas was perplexed by Mainstreet's data, that suggested Nenshi was losing amongst young and female voters to the conservative-learning candidateSmith.

Thomas said that flies in the face of established voter preferences. On average, women and young people tend to vote for progressive candidates over conservative ones.

"Everything we know about how gender affects voting behaviour," said Thomas, "would lead us to predict that women would be more likely to support somebody like Naheed Nenshi over Bill Smith."

Other polls contradicted Mainstreet

Other polls late in the race contradicted Mainstreet Research's polling results. One survey, conducted on behalf of a non-profit citizens group dedicated to light-rail transit expansion in Calgary, had Nenshi way out in front of Smith.

Another poll, conducted for a group of academics by Forum Research, also suggested that Nenshi was cruising to victory.

Both of these public opinion surveys revealed demographic breakdowns more in line with the tendency for younger voters and female voters to support Nenshi.

Political watchers think Mainstreet's "big failures" had a big impact on the campaign, influencing the tone and tenure of public debate.

"It was shaping up to be a total snoozer of an election," said Thomas. "What made the campaign interesting was a bad estimate that made the campaign."

Undecided voters especially those looking to vote against the incumbent often look to polls to see who has the best chance of winning and back them, she said. "That narrative matters a lot for some voters figuring out their minds," said Thomas.

Janet Brown, a Calgary-based independent pollster and political commentator, thinks Mainstreet's polls had a profound impact. She believes Mainstreet's first poll effectively ended long-time city councillor Andre Chabot's bid to be mayor.

Mainstreet's first poll had Chabot well behind with just seven per cent, compared to Smith at 42 per cent and Nenshi at 33 per cent of decided voters.

Brown thinks anti-Nenshi voters factored the polls into their decision-making and coalesced around Smith. Brown also believes Mainstreet's polls shaped what she calls the narrative about the campaign.

"The campaign became about polls and it became about horse race numbers and it wasn't about issues that mattered to people," Brown told CBC News.

Brown worries that public policy debates from reviving the city's sputtering economy to whether Calgary should build a new arena for the Flames or host the Olympics in 2026 didn't get the prominence they deserved.

"All of those things kind of got pushed aside and the media coverage was just very much about the horse race. 'Who's ahead?' 'Who's behind?'"

Lessons for the news media?

The news media also bears part of the blame for the way it covered Mainstreet's polls, says Brown.

Brown wonders why the news media seemed to put so much stock in the polls given that many political watchers had raised red flags and pointed out that some of Mainstreet's data just didn't make sense such as the predictions about young and female voters.

She says she and other independent experts warned journalists at the time. Brown says journalists need to be much more critical when it comes to public opinion polls.

"In the case of Postmedia, I think they had lots of credible people telling them to question those polls. But they chose to ignore that well-intended advice to ignore these polls."

Postmedia, which owns dozens of newspapers across the country including many of Canada's largest, put its relationship with the polling firm on hold after the election.

In an email, Postmedia executive Phyllise Gelfand told CBC News the company plans to meet with MainstreetResearch in January to review the results of the polling firm's internal investigation. Postmedia declined CBC News' request for further comment.

Brown is also frustrated with the CBC's reporting, which she feels should have provided more critical analysis.

As a rule, CBC News always checks the methodology of polls with its in-house research experts.

But the public broadcaster's editorial leader in Calgary admits CBCNewscould have been more critical about polling numbers that did not make sense.

Recent polling failures elsewhere

Public opinion polling has come under intense scrutiny in recent years.

There have been some big failures, including pollsters' failure to predict the outcome of last year's Brexit vote in the U.K and the results of the country's 2015 general election.

The polls were also askew during the Alberta provincial election in 2012. Polls had the Wildrose Party ahead for most of theelection campaign, but the Progressive Conservatives ended up winning.

According to recent academic research, political journalism is obsessed with public opinion data but incapable of comprehending it.

Benjamin Toff, a professor with the Hubbard School of Journalism and Mass Communication at the University of Minnesota, researched the so-called "Nate Silver Effect."

It's named after the American writer, statistician and founder of FiveThirtyEight, a website that focuses on data journalism about polling, politics, economics and sports.

Toff contends that "conventional news organizations appear ill-equipped" to interpret and judge public polling data. Toff has also concluded that the news media's traditional role as an independent "gatekeeper," policing the standards and release of polling data, is eroding.

'Polls are important for democracy'

Despite concerns about journalists needing to apply a more critical eye to polling results, Mount Royal University political scientist Duane Bratt thinks polls are ultimately a good thing.

"I think polls are important for democracy," he told CBC News.

"There are those who say we should have no polls during a campaign, except the campaigns would have polls. And why should they have that sort of information and the public not? I think there's value to a poll.

"I think polls are necessary in a democratic society but bad polls really hurt democracy."