Inside Saint John's response to a 'devastating' cyberattack

It was Nov. 26, 2020, and the municipal computer network in Saint John, N.B., had been dark for almost two weeks taking down the city's website, costing the city thousands of hours in lost work and affecting its emergency dispatch system.

It was the work of cybercriminalswho unleashed a ransomware attack that forced the city to disconnect itself from the rest of the online world.Saint Johnhired a Toronto-based company to navigate negotiations with them.

But the criminals weren't very communicative.

"Wanted to update you to let you know that the Ryuk Threat Actors have not reached out since they decrypted the sample files, on November 20th," Jason Kotler, president and CEO of a company called CYPFER (Cyber Security, Payment Facilitators, Emergency Response), wrote in an email to city lawyers and outside counsel on Nov. 26.

"Ryuk is patient and will likely not respond until we reach out again. We might hear from them within the week. Nonetheless, we will continue to monitor."

The city hasn't said much publicly about its response to the cyberattack, the after-effects of which are still affecting some of its operations a year later. More than 160 pages of records that offer a peek inside the chaos that ensued after the attack, but the documents were only turned over after CBC News filed an access-to-information complaint.

The city ultimately decided not to pay a ransom, estimated by one councillor at between $17 million and $20 million worth ofBitcoin, and instead opted to rebuild its network from scratch.

It was a decision that would have serious consequences for the citizens of the foggy Atlantic port city.

Saint John's experience may offer lessons for Newfoundland and Labrador, which has been hit with a cyberattack that has wreaked havoc, cancelling medical procedures and cancer treatments.

• The cyberattack throttling N.L's health-care system

While officials in that province have released few details about how its attack happened, last week it confirmed both employee and patient data was stolen.

Cyberattacks can take 'many years' to recover from

More than a year haspassed, but the city still hasn't fully recovered from what Saint John Mayor Donna Reardon described as "a devastating attack."

As of this summer, employees in city offices still couldn't print, Reardon said, though that functionality has now returned.

That was perhaps a more benign issue compared to the city police force's struggle. It couldn't generate statistics on crime occurrences, such asthenumber of mental health crisis calls,or access some police reports.

"It's taken a long time to get things back up and running, to unlock all of their tools," Reardon said.

When asked when the city is expected to be fully recovered from the attack, a city spokesperson didn't provide an exact timeframe, saying recovery from cyberattacks can take "many years."

"Many systems that were in place prior to the attack are operational," city spokesperson Lisa Caissie wrote in an emailed statement.

"The city continues to collaborate with all service areas, including the Saint John Police Force, on priorities for restoration. Remaining work relates mostly to automation for efficiency."

The city has spent nearly $3 million recovering from the attack, though that number may increase since the process isn't complete. All but $400,000 spent so far is estimated to be recovered through insurance.

Network breached 2weeks before attack

The problems began on Oct. 28, 2020, when the city's network was breached through a phishing email, councillors learned at a briefing on Nov. 16, 2020. A virus attack hit the city's systems a few days later, on Nov. 3 and 4.

On Nov. 13, 2020, around 9 p.m., the city discovered a ransomware attack was underway. One record describes the attack as being triggered through an Excel file. The federal goverment'sCanadian Centre for Cyber Security (CCCS) describes ransomware as"a type of malware that ultimately denies a user's access to files or systems until a sum of money is paid."

In the early hours after the attack was discovered, records show the city disconnected "all information technology infrastructure and devices" to try and contain it.

"The end result of this action was all network services across the municipality are currently shut down, including email and computer aided dispatch to name only two," according to a security event report issued by New Brunswick's Office of the Provincial Security Advisor early on.

An hour after the attack was discovered, the city's Public Safety Answering Point, its emergency call centre, lost connectivity, including access to "their computer aided emergency services dispatch system and mapping tools." A contingency plan saw 911 calls rerouted through Fredericton.

"The City of Saint John does not yet know how bad the damage is, that work continues," a security event report says.

The records don't indicate when the city became aware of the ransom request or realized it was a Ryuk attack.

The CCCS saysRyuk is "a ransomware variant known to target large enterprises, hospitals and critical infrastructure and demand extremely large ransoms."

Active since August 2018, the report says Ryuk "is affiliated with multiple Russian-speaking cybercriminals."

Attack group not interested in selling info on dark web

According to minutes from a briefing councillors received from Saint John city manager John Collin on Nov. 16, 2020, Ryuk was described as "a Russian Mafia group that are ransom oriented and will provide de-encryption codes if paid."

But they are not interested in "personally identifiable information" to sell on the dark web, the minutes say.

"Most finance files are not touched. The city is safe, 911 calls are re-routed through Fredericton. The restoration plans are underway to re-establish the network."

The update says councillors were told not to discuss the attack, and to refer requests to the city's communications director.

It also says money would be available "at the federal and provincial level to rebuild rather than pay ransom," though the city has not received any funding from the provincial or federal governments to date.

Thousands of hours of work lost

By Nov. 20, 2020 CYPFER had created a negotiation strategic plan that spelled out how Saint John would negotiate with the cybercriminals who were looking for payment. The details of that strategy are redacted in the copy provided to CBC News.

More than a week after the attack began, the records suggest the city still wasn't entirely sure what information could be at risk.

"I would suggest that they haven't shown us anything that speaks to the sensitivity of the data they may have," Stephanie Rackley-Roach, the city's chief information officer, wrote in an email on Nov. 22, 2020, parts of which were redacted.

In an update to council the next day, the city manager described how the city was slowly rebuilding from scratch, saying "progress restoring the network destruction is slow and deliberate."

Most city services were continuing as usual, Collin said, including waste management, water and sewer services..

But according to a Nov. 25, 2020, briefing to the provincial government, thousands of hours of work had been lost on servers and devices.

Keeping secrets

One year later, it's not clear what systems or capabilities the city still doesn't have back.

For the last year, the Saint John Police Force has been unable to answer access to information requests that ask for crime data and police reports, but Caissie, a spokesperson with the city, suggested this functionality has recently returned.

"As of this week, we can confirm that the Saint John Police Force has been provided with the capability to run a number of reports," Caissie said.

The attack also impacted provincial court proceedings, but the province hasn't tracked how many might have been delayed. The provincial government referred questions about that to the police, which referred questions to the province.

"Anecdotally we are aware that there were changes including the providing of disclosure documents," Department of Justice spokesperson Geoffrey Downey wrote in an email.

The city initially refused to provide most of its records about the cyberattack, citing a number of exemptions in the province's access to information legislation. But additional records were turned over earlier this year, following CBC's access-to-information complaint.

The Saint John Police Force is still investigating the cyberattack, according to spokesperson Jim Hennessy, but no update on whether any progress has been made was offered.

The agency consulted with the RCMP, but the RCMP has never initiated an investigation into the attack, a spokesperson for the Mounties confirmed.

Lessons for Newfoundland and Labrador

While Caissie confirmed a forensic report found no direct evidence of data theft, the attack on Newfoundland and Labrador's health care system has compromised patient data, the province confirmed, on top of delaying life-saving treatment. Caissie said the city has not received a request to provide advice to its Atlantic counterpart.

• Officials confirm patient data stolen in N.L. cyberattack

But if there's one thing Newfoundland and Labrador can learn from Saint John's experience, it's to not pay ransom should the province be asked, according to Dima Alhadidi, who has spent years researching topics such as data privacy.

"Regardless of the consequences, we should not pay," said Alhadidi, who is an assistant professor of computer science at the University of Windsorin Ontario.

"Because if we pay, this will motivate them to target other victims and we will end up having the same problems."

The decision not to pay a ransom was made by Saint John council, and the city's mayor believes it was the right one.

"Even if you decide you had the money and you pay for it, is there any guarantee you're actually going to get everything back? I mean, you're dealing with criminals," Reardon said.

Alhadidi also believes that governments hit by cyberattacks should be open with the public about the attack and what led to itto help protect other public agencies.

She would also like to see mandatory training for all employees about how to deal with suspicious emails, and for all agencies to have a contingency plan on what to do should they be hit with a cyberattack.