Ministry of the
Solicitor General

Investigation into the emergency alerts sent on January 12, 2020

Investigation into the emergency alerts sent on January 12, 2020

February 18, 2020

Office of the Provincial Security Advisor
Ministry of the Solicitor General

Table of Contents

Executive summary
Introduction
Background
Investigation and findings
Conclusion


Provincial Emergency Operations Centre Action Plan

Executive Summary

Pursuant to the Emergency Management and Civil Protection Act, Emergency Management Ontario (EMO) supports municipalities, First Nations and ministries in implementing their emergency management programs by providing them with advice, assistance, guidelines, training, and other tools. EMO administers the Provincial Nuclear Emergency Response Plan (PNERP) and oversees its implementation. In particular, EMO’s Provincial Emergency Operations Centre (PEOC) is mandated to plan and coordinate the offsite response to a nuclear or radiological emergency.

The PEOC monitors actual or emergent situations that have the potential to impact Ontario. The PEOC is activated during a provincial or large-scale municipal emergency and its key function is to coordinate a Government of Ontario response.

Canada’s National Public Alerting System (NPAS), also known as the Alert Ready – Emergency Alert (Alert Ready) system, provides emergency management organizations the capability to rapidly warn the public of imminent or unfolding hazards to life. Public alerts are issued through radio, television, and on LTE-connected and compatible wireless devices. Alert Ready is a public-private partnership between federal, provincial, territorial (FPT) governments and the broadcasting and telecommunications industries. Alert Ready is owned by Pelmorex Corp., a private company licensed under the Broadcasting Act to broadcast emergency alert messages and other public safety information.

On January 12, 2020, the PEOC issued an emergency alert via the Alert Ready system to Ontarians reporting an incident at the Pickering Nuclear Generating Station (PNGS). A second alert was issued 108 minutes later advising that the previous alert had been sent in error. Later that day, the Solicitor General announced a full investigation into the alerts.

On January 15, 2020, the Deputy Solicitor General – Community Safety (DSG-CS) directed the Provincial Security Advisor (PSA) to immediately investigate the circumstances surrounding the issuance of the January 12, 2020 alerts. The DSG-CS directed the PSA to present findings from the investigation with the goal of assisting the Ministry of the Solicitor General (SOLGEN) in preventing future incidents and maintaining public trust in the emergency alert process.

The scope of the investigation was to include the following areas:

  1. Determining the sequence of events that occurred in relation to the alert issued on January 12, 2020
  2. Protocols and procedures for issuing alerts via the Alert Ready - Emergency Alert System
  3. Protocols and procedures for cancelling such alerts
  4. Review of pre-scripted messages in the Alert Ready - Emergency Alert System
  5. EMO training on the use of the Alert Ready system
  6. Technological capabilities and gaps of the Alert Ready system
  7. Review of EMO lines of communication with government and stakeholders.

In the course of the investigation, the Office of the Provincial Security Advisor (OPSA) worked with partners in SOLGEN, including SOLGEN investigators and other emergency management professionals.

Key Findings

  1. Sequence of Events that Occurred in Relation to the Alert Issued on January 12, 2020
    • On January 12, 2020 at 7:23 a.m., the Provincial Emergency Operations Centre issued a nuclear emergency alert to Ontarians via the Alert Ready system as a result of human error on the part of a PEOC Duty Officer (DO) scheduled on duty that morning.
    • At each shift change, the DO tests both the live and training Alert Ready systems. First, the DO logs in to the live system and then logs out to ensure system access. Second, the DO logs into the training system, drafts a message, sends an alert on the training system, and then logs out to practice issuing an alert. The DO can draft their own message or load a saved template.
    • On January 12, 2020, the DO logged in to the Alert Ready live system, believed they had switched to the training system, selected a pre-scripted nuclear template for the PNGS and accidentally sent a live alert.
    • The DO was not acting on any information concerning a nuclear incident.
    • The DO immediately recognized the error and proceeded to seek guidance on corrective action from EMO supervisors.
    • Despite a series of consultations with on-duty, on-call and off-duty EMO supervisors at different levels, no clear instructions were provided to the on-duty DOs in the immediate aftermath of the first alert. Supervisors were uncertain about whether or how to issue a second alert as broadcast intrusive, meaning that it would be broadcast in the same manner as the first alert.
    • Unrelated information about PNGS circulated and added confusion to senior OPS executives’ understanding of the situation.
    • As the morning progressed, senior OPS executive involvement in determining corrective action became necessary.
  2. Protocols and Procedures for Issuing an Alert on the Alert Ready System
    • EMO Alert Ready policy outlined procedures for creating and issuing an alert.
    • There was a common understanding amongst EMO staff on internal approvals for issuing an alert; however, they were not built in to the Alert Ready system and not documented in the EMO Alert Ready policy.
    • The Provincial Nuclear Emergency Response Plan details the notification protocols to be implemented in the event of a nuclear emergency. As the alert was issued in error no such process occurred.
  3. Protocols and Procedures for Cancelling an Alert
    • There was an EMO “End Alert” procedure which addressed an alert sent in error; however, communications from January 12, 2020 cited no reference to this procedure.
  4. A review of Pre-scripted Messages in the Alert Ready System
    • The PEOC only maintained templates with pre-scripted messaging for nuclear alerts. The Provincial Nuclear Emergency Response Plan (PNERP) required that nuclear emergency bulletins be pre-scripted for each reactor facility and notification category, as far as practical. The alert issued in error included a pre-scripted template maintained by the PEOC for the Pickering Nuclear Generating Station.
    • The PEOC did not have a pre-scripted template for an “End Alert” message on January 12, 2020.
    • Both PEOC alerts sent on January 12, 2020 were issued in English only. The PEOC did not have any French language templates. The PEOC did not initiate emergency translation procedures to issue either alert in French.
    • Identical pre-scripted templates were used on both the Alert Ready live and training systems. No distinctive labelling such as “EXERCISE, EXERCISE, EXERCISE…” was used on the training system.
  5. EMO Training on the Use of the Alert Ready System
    • PEOC DOs are expected to complete a four-week entry training program that includes Alert Ready training; however, November 2019 training for the DOs on duty the morning of January 12, 2020 was shortened due to PEOC activation to coordinate a provincial response for an emergency flood evacuation.
    • A review of the Alert Ready training system logs revealed that DOs were not consistently sending alerts on the training system as per PEOC shift change procedures.
    • Most EMO supervisors did not have access to or training on the Alert Ready system.
  6. Technological Capabilities and Gaps of the Alert Ready System
    • The Alert Ready system allows a single user to draft and send an alert.
    • The Alert Ready training and live systems can be opened simultaneously.
    • The training system is similar to the live system in order to simulate issuing a real alert.
  7. A Review of the EMO Lines of Communication with Government and Stakeholders
    • Government officials and stakeholders are to be notified in advance of a nuclear emergency alert as per the PNERP. The lack of notification caused significant concern and confusion.
    • EMO communications with government and stakeholders between the first and second alert were reactive and handled on a case-by-case basis using inconsistent messaging.
    • Pelmorex’s role on January 12, 2020 exceeded advice on the technical use of the Alert Ready system and included advice on how and whether to issue alerts.

The information reviewed in the course of the investigation confirmed that on January 12, 2020 at 7:23 am, the PEOC issued a nuclear emergency alert to Ontarians via the Alert Ready system as a result of human error on the part of a PEOC DO scheduled on duty that morning.

The findings revealed EMO procedural gaps, lack of training, lack of familiarity with the Alert Ready system and communication failures. These findings can provide context to the DO error and the length of time – 108 minutes – that elapsed between the alert issued in error and the second clarifying alert.

The investigation was conducted with the full cooperation and support of EMO, the PEOC and relevant stakeholders. The findings are provided to assist the DSG-CS and EMO in preventing a similar incident in the future.

Introduction

On January 12, 2020 at 7:23 am, EMO Provincial Emergency Operation Centre issued an emergency alert via the Alert Ready – Emergency Alert System (Alert Ready) reporting an incident at the Pickering Nuclear Generating Station. At 9:11 am the PEOC issued a second alert to inform the public there was no incident at PNGS and that the prior alert was issued in error. Later that day, the Solicitor General announced a full investigation.

On January 15, 2020, the Deputy Solicitor General - Community Safety directed the Provincial Security Advisor to immediately investigate the events surrounding the issuance of the two alerts.

The DSG-CS indicated that the scope of the investigation would address, but not be limited to:

  1. determining the sequence of events that occurred in relation to the alert issued on January 12, 2020
  2. protocols and procedures for issuing alerts via the Alert Ready - Emergency Alert System
  3. protocols and procedures for cancelling such alerts
  4. review of pre-scripted messages in the Alert Ready - Emergency Alert System
  5. EMO training on the use of the Alert Ready system
  6. technological capabilities and gaps of the Alert Ready system
  7. review of EMO lines of communication with government and stakeholders

The investigation was informed by reviewing relevant government correspondence, reviewing procedures and policies, conducting interviews, observing a demonstration of the Alert Ready system and consulting other emergency management professionals.

The DSG-CS directed the PSA to present findings from the investigation with the goal of assisting the ministry in preventing future incidents and maintaining public trust in the emergency alert process.

Background

Emergency Management Ontario

Pursuant to the Emergency Management and Civil Protection Act, Emergency Management Ontario (EMO) supports municipalities and ministries in implementing their emergency management programs by providing them with advice, assistance, guidelines, training, and other tools.

The EMO’s Provincial Emergency Operations Centre monitors actual or emergent situations that have the potential to impact Ontario or may require a coordinated response from Ontario. The PEOC is activated during a provincial emergency and its key function is to coordinate a Government of Ontario response. This includes providing municipalities and First Nations with a single point of contact for provincial assistance in times of crisis. The PEOC is staffed 24/7 by Duty Officers. The PEOC chain of command is determined by a shift schedule which rotates EMO management staff through the positions of Duty Operations Chief (DOC) and Commander. The DOs are present in the PEOC when on duty, whereas the DOC and Commander may be offsite but on-call.

EMO leads Ontario’s offsite nuclear emergency response. EMO administers the Provincial Nuclear Emergency Response Plan and oversees its implementation. The PEOC is mandated to coordinate the offsite response to a nuclear or radiological emergency in conjunction with reactor facilities and stakeholders.

Alert Ready – Emergency Alert System

Canada’s National Public Alerting System, also known as Alert Ready – Emergency Alert (Alert Ready) system, provides emergency management organizations the capability to rapidly warn the public of imminent or unfolding hazards to life. Public alerts are issued through radio, television, and on LTE-connected and compatible wireless devices. Alert Ready can be broadcast and wireless intrusive, meaning that the messaging is accompanied by a distinct tone on wireless devices and interrupts television and radio broadcasting.

Alert Ready is a public-private partnership between federal, provincial, territorial governments and the broadcasting and telecommunications industries. Public Safety Canada’s (PSC) website provides an outline of NPAS, as shown below.

Image details

Source: Public Safety Canada, 2020

Under this framework, the PEOC is an alert issuer. Alert issuers submit their alerts to the alert aggregator and issuers are responsible for the content of alert messages. The PEOC issues alerts on behalf of Ontario municipalities, First Nations and the Government of Ontario. The Ontario Provincial Police issues AMBER alerts directly to the alert aggregator; similarly, Environment Canada issues weather alerts directly to the alert aggregator.

In Canada, the alert aggregator is called the National Alert Aggregation and Dissemination system. This system collects public safety messages from authorized government authorities and distributes those messages by satellite and through the internet to alert distributors. The NAAD system is owned by Pelmorex Corporation (Pelmorex), a private company licensed under the Broadcasting Act to broadcast emergency alert messages and other public safety information. In contrast, alert aggregation in the United States is the responsibility of the (FEMA), rather than a private company.

The Canadian Radio-television Telecommunications Commission (CRTC) has a mandatory distribution order for broadcast of The Weather Network and MétéoMédia, two Pelmorex services. The CRTC orders television providers to broadcast The Weather Network or MétéoMédia at a fixed rate, in part because Pelmorex funds and operates the NAAD system. There is no procurement contract between Government of Canada or the Government of Ontario and Pelmorex for the Alert Ready system. The Government of Ontario has a User Agreement with Pelmorex as an alert issuer.

The Governance Council is responsible for the oversight of the Pelmorex NAAD system and includes representation from the Alert Ready community, including FPT governments, Pelmorex and alert distributors.

Investigation and Findings

The findings of the investigation are organized according to the scope outlined in the Introduction.

  1. Sequence of Events
  2. The Alert Ready System
  3. Alert Protocols and Procedures
  4. Pre-scripted Messages
  5. Provincial Emergency Operations Centre and Training on the Alert Ready System
  6. Lines of Communication with Government and Stakeholders.

Sequence of Events

Determining the sequence of events that occurred in relation to the alert issued on January 12, 2020

The following timeline includes information obtained from interviews, e-mails, staff notes, PEOC audio logs, cellphone call logs and text messages. The timeline describes events involving EMO staff, other SOLGEN staff, wider OPS staff and external stakeholders.

In reference to the timeline, Employees 1-3 and Supervisors 1-6 are EMO staff. Employees 1 and 2 were scheduled to be on duty on January 12, 2020. Supervisors 1 and 5 were the on-call management team. Supervisors 2, 3 and 4 and Employee 3 were off duty, but self-deployed in response to the alert. Supervisor 6 is required to be available as needed.

While not included in the timeline, Employees 1 and 2 were inundated with calls from concerned and impacted stakeholders following the issuance of the alert the morning of January 12, 2020.

Sequence of events that occurred in relation to the alert issued on January 12, 2020
Time Events
7:00 a.m. Employees 1 and 2 began shift change protocol at the PEOC. This included logging in and out of the Alert Ready live system and logging in and sending an alert on the training system. Employee 1 logged into the live system, believed they had switched to the training system, selected a pre-scripted nuclear message and sent a broadcast-intrusive public alert.
7:23 a.m. The public received a nuclear emergency alert. A screenshot of the mobile version is displayed below.
Emergency Alert about an active nuclear situation at the Pickering nuclear plant. There was no release of radiation. Emergency staff are responding. No need to take action. People within 10 kilometres of the plant should monitor local media for further information.
7:25 a.m. Employee 1 contacted off-duty Supervisor 2 and explained that the alert was intended for the training system but was sent in error from the live system. Supervisor 2 advised the employee to follow the cancellation procedure and notify on-call Supervisors 1 and 5. Supervisor 2 said they would call Supervisor 6.
7:27 – 7:33 a.m. Employee 2 notified on-call Supervisor 1 that the alert was sent in error. Supervisor 1 asked Employee 2 how the cancellation alert would be sent out and Employee 2 was unsure of the procedure. Supervisor 1 then followed the chain of command to notify on-call Supervisor 5 who then notified Supervisor 6.
7:37 a.m. Cabinet Office queried SOLGEN and Ministry of Energy, Northern Development and Mines (ENDM) executives requesting an explanation for the alert.
7:39 a.m. PEOC sent an e-mail to an internal group of emergency management staff advising that “A NAAD System alert was sent out in error. Retraction Process has begun!”
7:40 a.m. Off-duty Supervisor 3 e-mailed Employee 1 regarding the alert and requested any details that could be shared with stakeholders. Employee 1 advised off-duty Supervisor 3 that the alert was sent in error. 
7:41 a.m. Supervisor 6 sent a text message to DSG-CS advising that the alert was sent due to human error or malfunction and that there was no real event at the Pickering Nuclear Generating Station.
7:41 –7:47 a.m. A senior executive at Pelmorex sent a text message to off-duty Supervisor 4 advising that a broadcast intrusive alert was sent via the NAAD system and inquired whether the alert was meant to be broadcast to all of Ontario. A Pelmorex employee contacted Employee 2 to advise that the alert expired and that no further action was required through the Alert Ready system. Employee 2 asked how to communicate the error to the public and the Pelmorex employee suggested not using the Alert Ready system for that purpose.
7:47-7:54 a.m. Multiple calls between Employee 1 and various supervisors confirmed that the message was sent in error and intended only for the training system. Off-duty Employee 3 returned to PEOC to help field calls and provide updates to supervisors. 
7:54 a.m. The Pelmorex employee told Employee 2 that no action is required, including a cancellation. The Pelmorex employee confirmed that The Weather Network issued a broadcast message advising that the alert was sent in error. The Pelmorex employee asked about sending out their own Twitter message to the public. Employee 2 advised that they would confer with a supervisor and call Pelmorex back.
8:00 a.m. Employee 2 called the Pelmorex employee and advised that supervisors had approved Pelmorex sharing information on the incident internally. Another Pelmorex employee on the call asked again about a Twitter message. Employee 2 advised that public messaging is under review.
8:03 a.m. An e-mail circulated amongst senior OPS executives indicating that a power outage occurred at PNGS, that a call was taking place with OPG and that more details would be forthcoming.
8:04 a.m. The DSG-CS sent an email to senior OPS executives advising that there would be a conference call shortly with EMO supervisors. The DSG-CS also noted: “Preliminary info is that alert was sent out in error for a NON REPORTABLE EVENT at Pickering. Working on Comms to be sent out ASAP.”
8:05 a.m. Supervisor 1 arrived at the PEOC.
8:06 a.m. OPG tweeted that the alert was sent in error.
8:08 – 8:35 a.m. Supervisors 4 and 5 communicated by text message about the content of a second alert and whether to issue it as broadcast intrusive.
8:10 a.m. SOLGEN Communications Branch (Comms) sent Supervisor 2 draft content for the second alert.
8:10 a.m. Under the direction of Supervisor 1, Employee 3 called Supervisor 5 to provide an update. Employee 3 conveyed that Supervisor 2 approved an email to Community Emergency Management Coordinators (CEMCs) advising them that the alert was sent in error. Supervisor 5 asked why The Weather Network notified the public that an error had occurred, and Employee 3 responded that Pelmorex may have shared the information provided to them by the PEOC. Supervisor 5 advised that Supervisor 4 was recommending a non-intrusive alert.
8:15 a.m. A teleconference was held between DSG-CS, EMO Supervisors 5 and 6 and Comms, it was confirmed that the alert was sent in error and that it was unrelated to any activity at PNGS. DSG-CS advised EMO that DSG-CS and Comms would need to approve the content of future communications and alerts with respect to the alert sent in error.
8:24 a.m. Cabinet Office sent an email to senior OPS executives querying whether a second alert was planned noting concerns that other communication channels would not have the same reach as the first alert.
8:30 a.m. In response to a Global News reporter’s inquiry, Comms advised the reporter that the event was an error.
8:33 a.m. Supervisor 2 discussed potential options with the Pelmorex employee for issuing a non-intrusive alert. The Pelmorex employee advised non-intrusive alerts will not be distributed directly to the public by television, radios or mobile devices. The Pelmorex employee advised that Pelmorex had already notified alert distributors 15 minutes earlier of the error, based on Pelmorex management approval. Pelmorex advised against sending a second alert and to allow alert distributors to handle messaging to the public.
8:34-8:40 a.m. Supervisor 5 communicated with Supervisor 6 by text message about issuing a non-intrusive alert and issuing a press release, based on Pelmorex advice.
8:43 a.m. The DSG-CS advised Supervisor 6 that draft alert messaging was approved.
8:43 a.m. Comms staff communicated internally about a potential reportable incident at the PNGS.
8:48 a.m. An employee of the Solicitor General’s Office emailed the DSG-CS and Comms to request a minor edit to the draft alert messaging in order to more accurately reflect that there was not an incident at PNGS.
8:49 a.m. An OPG representative called PEOC and spoke to Employee 2. Employee 2 informed OPG that the second message will not be sent as an alert. Supervisor 2 took over the call and advised OPG that PEOC is still awaiting approval on messaging and method of dissemination. OPG advised that they wished to coordinate messaging to the public with the PEOC in a timely manner, noting that  media had begun to arrive at the Pickering site.  
8:50 a.m. Supervisor 5 e-mailed Supervisor 1 with the approved wording to be released immediately as an emergency alert.
8:53-9:12 a.m. The DSG-CS instructed Supervisor 6 to incorporate the clarifying change requested by the Solicitor General’s Office in the draft alert messaging. Supervisor 6 advised that the wording was already sent to the PEOC. The DSG-CS asked Supervisor 6 why the second alert had not been sent.
8:55 a.m. Supervisor 5 confirmed with Supervisor 6 by text message that an alert will be sent broadcast intrusive, as instructed by Supervisor 6.
9:11 a.m. Employee 1 sent the second alert to the public, under supervision of employees and supervisors in the PEOC. A screenshot of the mobile version is displayed below.
Emergency Alert stating there is no active nuclear situation at the Pickering nuclear plant. The previous warning had been issued in error. There is no danger to the public. No further action is required.

Box 1: Planned Maintenance Outage at Pickering Nuclear Generating Station

On January 12, 2020, Unit 1 at Pickering Nuclear Generating Station was on a planned maintenance outage and was therefore out of service in a shutdown position. At 1:21 am, there was a momentary loss of power to some equipment on this Unit resulting in an internal notification. OPG provided a courtesy update to the Canadian Nuclear Safety Commission (CNSC) regarding this loss of power.

Findings from the investigation confirmed that the above planned outage and brief loss of power was initially misconstrued in senior OPS executives’ correspondence as possibly being related to the alert sent in error. This temporarily added confusion to their understanding of the situation. Subsequent discussions with stakeholders, however, confirmed that the minor issue at Pickering was unrelated to the alert sent in error from the PEOC.

Findings

  • The first alert was issued as a result of human error. Employee 1 accidentally issued an alert on the live system, instead of the training system.
  • Employee 1 was not acting on any information concerning a nuclear incident.
  • Employees 1 and 2 did not know how to take corrective action and immediately sought direction from supervisors.
  • Through a series of consultations with supervisors at different levels, no clear instructions were provided to Employees 1 and 2 in the immediate aftermath of the first alert.
  • The supervisors were uncertain about whether or how to issue a second broadcast intrusive alert.
  • At senior and working levels, Pelmorex staff consistently advised EMO against sending a second alert via the Alert Ready system.
  • Unrelated information about PNGS circulated and added confusion to senior OPS executives’ understanding of the situation (see Box 1).
  • As the morning progressed, senior OPS executive involvement in directing corrective action became necessary.

The Alert Ready System

Technological capabilities and gaps of the Alert Ready system

The following information describes the Alert Ready – Emergency Alert (Alert Ready) system used the morning of January 12, 2020 at the Provincial Emergency Operations Centre (PEOC).

The Alert Ready system is a public-private partnership between FPT governments and the broadcasting and telecommunications industries (See Background for more details). All authorized alert issuers have access to the same Pelmorex Corp. (Pelmorex) operated system; however, some alert issuers have invested in custom interfaces to tailor the user experience to their needs. EMO's PEOC does not have a tailored interface, rather it uses the system provided by Pelmorex.

The Alert Ready system users can access the system from different devices (computer, smartphone, etc.) with credentials managed by Pelmorex and issued to PEOC Duty Officers (DOs) through EMO. EMO is responsible for ensuring access credentials are kept up-to-date, and that credentials are cancelled when an employee quits, is terminated or is on extended leave.

Creating an Alert

A single user creates an alert by completing a form with event category, alert onset time, time of alert expiry, geographic location, severity, urgency, certainty and alert text. The form can be completed by the user or populated by loading a saved template. The alert issuer is responsible for the content of all messages, including the templates.

An alert can be broadcast and wireless intrusive, meaning that the messaging is accompanied by a distinct tone on wireless devices and interrupts television and radio broadcasting. The broadcast and wireless intrusive criteria are determined by the event severity, urgency and certainty values. If the broadcast intrusive criteria are met, the user will be notified on the interface while creating the alert.

Live and Training Systems

The Alert Ready system has a training system and a live system. As the names suggest, the former is for users to practice sending alerts; the latter is for real events. The process for issuing an alert on the training system closely mimics the live system, however the training alert terminates in the system after the final prompt.

Login credentials for the live and training systems are the same. Both training and live systems can be open simultaneously on the same device. The user interface is visually similar for both systems. They are differentiated largely by font and background colour and the words ‘training’ and ‘live.’ Prompts are pushed to the user throughout the alert issuance process on both systems, with the prompts on the live system indicating that a live alert will be issued should the user continue. The prompts on the training system largely mimic those on the live system. 

Alert Approval

A single user can draft and issue an alert by re-entering their password as a final step on the live system. There is no system requirement to forward the alert to a supervisor, designate or peer for review. There is no ability to preview an alert in its entirety before it is sent.

Findings

  • The Alert Ready live system allows a single user to draft and send an alert without managerial approval.
  • The training and live systems can be opened simultaneously on any device.
  • The training system is similar to the live system in order to simulate issuing a real alert.
    • The process and prompts are nearly identical with the exception of a final confirmation prompt on the live system.
    • Despite different font and background colours, the live and training system interfaces are similar.
    • The login credentials are the same for both training and live systems.

Alert Protocols and Procedures

Protocols and procedures for issuing an alert via the Alert Ready System
Protocols and procedures for cancelling such alerts

The protocols and procedures described here were in effect at the PEOC on January 12, 2020.

Procedures

The EMO Alert Ready policy in effect on January 12, 2020 outlined the procedure for the timely and accurate issuance of alerts by PEOC staff. The procedure required PEOC Duty Officers (DO) to document and verify incoming information from alerting authorities such as municipalities, First Nations and provincial ministries. The alerting authority is designated the approver of the alert; however, the PEOC DO verifies the alert text with the requesting official and ensures that alert threshold and content criteria are met. The DO is required to send a ‘Heads Up’ message to the PEOC Duty team (including supervisors) upon receiving a request for an alert.

According to the EMO Alert Ready policy, there was no requirement for a PEOC supervisor to review or approve an alert. However, in interviews, EMO staff demonstrated a common understanding of authority and approval levels for issuing an alert during a real emergency. The DO has the authority to compose and send an alert, but the Duty Operations Chief has the authority to approve it. The approval process can include verbal consultation and e-mail communications.

An “End Alert” procedure was in effect on January 12, 2020, according to the EMO Alert Ready policy.  It notes that all broadcast intrusive alerts “must be ended” by the PEOC DO (emphasis in original), meaning an “end alert” is sent to the public. The procedure explicitly states that: the end alert procedure applies when “an alert was issued in error”, among other situations; and the DO “shall draft an end message in the same format that the BI [broadcast intrusive] Alert was issued.” The end procedure also states: “The alert will also include ‘END – ‘ in the subject line.”

In correspondence during the event and during subsequent interviews, PEOC staff did not cite the “End Alert” procedure.

Terminology

Pelmorex Corp. (Pelmorex) and EMO documentation and communication referred to varying terminology to describe potential corrective action for an alert sent in error.

  • PEOC staff and Ministry of the Solicitor General management referred to “retracting” the alert. During this investigation, a retraction definition or process could not be found.
  • The EMO Alert Ready policy referred to “ending” an alert, which is a required step following all alerts, whether issued in error or not.
  • Draft EMO internal documentation refers to issuing an alert “correction (update).” It is unclear whether this referred to a second alert broadcast to the public or an update to the system and stakeholders.
  • The NAADS Issuer User Guide provides instructions for “cancelling” an alert which advises “all parties” that an alert was cancelled and no longer in effect. It was not clear to whom “all parties” referred. The User Guide does not address an alert sent in error.
    • EMO staff mentioned pulling the cancellation procedure; however, it is unclear whether these were the instructions being referenced.
  • EMO staff and Pelmorex discussed the “expiry” of the first alert, but there was disagreement about whether allowing an alert to expire was sufficient corrective action.
    • The expiry time is set by the issuer and determines when the alert will stop broadcasting.
  • EMO staff asked Pelmorex to “pull it [the alert] down”; however, there was no reference to “pull it down” in EMO and Pelmorex documentation.

Findings

  • The EMO Alert Ready policy outlines procedures for creating and issuing an alert.
  • There was a common understanding amongst EMO staff on internal approvals for issuing an alert; however, they were not built in to the Alert Ready system and not documented in the EMO Alert Ready policy.
  • There was an internal “End Alert” procedure that applied to the January 12, 2020 incident, but EMO staff did not refer to it.
  • There was no consistent terminology in use between EMO and various stakeholders in describing potential corrective action for an alert sent in error.

Pre-scripted Messages

Review of pre-scripted messages in the Alert Ready - Emergency Alert System

The following information describes the templates with pre-scripted messages in the Alert Ready – Emergency Alert (Alert Ready) system available to the PEOC the morning of January 12, 2020.

The PEOC only maintained templates with pre-scripted messaging for nuclear alerts. The Provincial Nuclear Emergency Response Plan required that nuclear emergency bulletins be pre-scripted for each reactor facility and notification category, as far as practical. There were no templates for other alerts as the event type, location and impact can vary. There were no templates to issue an “End Alert.”

The PEOC did not have any French language templates. Procedures existed for submitting an emergency French translation request. The service standard for emergency French translation was to respond within 30 minutes. On January 12, 2020, as the first alert was sent in error, translation was not considered. The PEOC did not submit an emergency French language request for the second alert.

EMO staff explained that character limitations within the Alert Ready system may impact the ability to issue alerts in a bilingual format. Wireless Intrusive alerts are limited to 600 characters and Broadcast Intrusive alerts are limited to 900 characters. Some of the current English nuclear templates contain in excess of 600 characters. EMO was exploring possible solutions to this issue, but a resolution was not documented as of January 12, 2020.

The Alert Ready training and live systems access identical templates for an alert. Duty Officers were free to create an alert or choose a template when issuing an alert on the training system. On January 12, 2020, Employee 1 selected a pre-scripted nuclear template alert during shift change procedures.

Findings

  • The PEOC only maintained templates with pre-scripted messaging for nuclear alerts. The PNERP required that nuclear emergency bulletins be pre-scripted for each reactor facility and notification category, as far as practical. The alert issued in error included a pre-scripted template maintained by the PEOC for the Pickering Nuclear Generating Station.
  • The PEOC did not have a pre-scripted template for an “End Alert” message on January 12, 2020.
  • Both PEOC alerts sent on January 12, 2020 were issued in English only. The PEOC did not have any French language templates. The PEOC did not initiate emergency translation procedures to issue either alert in French.
  • Identical pre-scripted templates are used on both the Alert Ready live and training systems. No distinctive labelling such as “EXERCISE, EXERCISE, EXERCISE…” is applied to the training system.

Provincial Emergency Operations Centre and Training

EMO training on the use of the Alert Ready system

Provincial Emergency Operations Centre

Duty Officers are shift-based positions within the PEOC.  Employees 1 and 2, the DOs who worked the daytime shift (06:30/07:00 – 18:30/19:00) on January 12, 2020, were hired on November 4, 2019.

There are currently 13 active DOs employed within EMO and of those 13, five are contract staff who began working as DOs in November 2019. DOs can be scheduled for on -duty eight-hour shifts during regular business hours and on-duty 12-hour shifts outside of regular business hours as well as on-call 14- and 24-hour shifts.

Duty Operations Chiefs are managers from across EMO that are scheduled across two shifts, weekday business hours of 08:00 to 17:00 or evening on-call for 17:00 to 08:00. During regular business hours, the DOC is the program manager for the DOs and is generally onsite at the PEOC. The night-shift consists of on-call managers from various areas within EMO. The Duty Commander (DC) role is filled by the Director and senior managers within EMO on a 24-hour basis.

Interviewees consistently noted resource constraints, high turnover of DO staff and gaps in the DO shift schedule. Some noted that the creation of the shift-based 24/7 Duty Officer position and associated job classification in 2018 resulted in lower pay for those who worked in similar positions prior.  It was further noted that the lower pay may be impacting staff retention and hiring. New DOs regularly worked alone or with another new DO and often volunteered for additional shifts. Some DOs were on-duty upwards of 55 hours a week and over 200 hours a month, not including on-call time. Employees 1 and 2 had worked 56 and 48 hours respectively in the seven days preceding January 12, 2020. January 12, 2020 was the seventh consecutive day and the third consecutive 12-hour shift that Employee 1 worked. Between December 11, 2019 and January 11, 2020, Employee 1 was on duty for 216 hours. The expected four-week schedule is 160 hours.

Training on the Alert Ready System

New DOs are scheduled to complete a four-week entry training program. The program consists of two weeks of in-class training focussing on policies, procedures and the Alert Ready – Emergency Alert (Alert Ready) system and two weeks of job shadowing. Following the training program, DOs are added to the PEOC schedule and should be paired with a senior DO for on-the-job training.

At each shift change, the DO tests both the live and training Alert Ready systems. First, the DO logs in to the live system and then logs out to ensure system access. Second, the DO logs into the training system, drafts a message, sends an alert on the training system, and then logs out to practice issuing an alert. The DO can draft their own message or load a saved template. Review of training system logs indicated that some DOs compose their own test message, but many regularly choose nuclear emergency templates. No procedures requiring supervisors to review training system logs and training system alerts were found.

Employees 1 and 2 completed the two-week in-class training, however they only completed one of two weeks of job shadowing, due to PEOC activation to coordinate a provincial response for an emergency flood evacuation. Interviewees indicated that the 2018 Hawaii false ballistic missile alert (see Box 2) was raised by trainees during November 2019 Duty Officer training; however, no specific instructions on how to prevent or respond to a similar incident were provided.

There is no specific training for the DOC or DC positions. While DOCs and DCs are familiar with the Alert ready system, many mentioned not being trained or having access to the system. Only one management level staff had credentials to access the system on January 12, 2020.

Box 2: 2018 Hawaii False Ballistic Missile Alert

On January 13, 2018 at 8:07 am local time, the Hawaii Emergency Management Agency (HI-EMA) issued an emergency alert to Hawaii residents that read “Emergency Alert: BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL”.  At 8:45 am, a second alert was issued stating that the first alert was false and that there was no missile threat or danger to Hawaii residents.

A subsequent investigation of the incident revealed that the first alert was sent during routine shift change drills at HI-EMA, and that the employee who sent the alert claimed that they were unaware that they were participating in a drill.

Notably, following the results of investigations into the false alert, the United States FEMA agreed to implement recommendations requiring vendors of alert system software to incorporate critical functions such as previewing and cancelling alerts, as well as provisions for training.

Findings

  • DOs are expected to complete a four-week training program that includes Alert Ready system training; however, November 2019 training for Employees 1 and 2 was shortened due to PEOC activation to coordinate a provincial response for an emergency flood evacuation.
  • A review of Alert Ready training system logs revealed that DOs were not consistently sending alerts on the training system per PEOC shift change procedures.
  • On the morning of January 12, 2020, Employees 1 and 2 were scheduled to work together and both had approximately six weeks of work experience at the PEOC.
  • Interviewees noted resource constraints, high turnover of DO staff and gaps in the DO shift schedule. DOs regularly work overtime.
  • Most managers did not have access to or training on the Alert Ready system.
  • A previous false alert investigation in Hawaii, raised by DO trainees in November 2019, resulted in recommendations requiring vendors of alert system software to incorporate critical functions such as previewing and cancelling alerts as well as provisions for training.  
    • There is no evidence that EMO, the NAAD Governance Council or Pelmorex Corp. implemented changes as a result of the Hawaii incident.

Lines of Communication with Government and Stakeholders

Review of EMO lines of communication with government and stakeholders

The review of EMO lines of communication is based on information obtained from interviews, e-mails, staff notes, PEOC audio logs, cellphone call logs and text messages.

EMO Internal Communications

Internal EMO communications from January 12, 2020 are reconsidered here insofar as they impacted external communications with wider government and stakeholders. Reference Sequence of Events for additional context.

In the immediate aftermath of the first alert, on-duty PEOC Duty Officer communications centered around telephone calls to a mix of on-call and off-duty EMO supervisors. Six supervisors were engaged during this period, three of whom were off-duty. These calls to supervisors were both to advise of the situation and to seek direction on corrective actions. The supervisors in turn emailed, called and texted each other about the alert and possible next steps.

Communications with Government and Stakeholders

As the first alert was sent in error, the PEOC could not notify EMO management and stakeholders in advance.  EMO staff experienced a high volume of queries from federal, provincial and municipal officials seeking information and clarification regarding the first alert. Stakeholders were variously told that the alert was issued in error, that there was a technical malfunction and that a retraction was underway. Between the issuance of the two alerts, some stakeholders re-contacted the EMO staff, growing increasingly frustrated with the lack of corrective action and related messaging for the public.

Communications with Pelmorex Corp.

After the first alert, Pelmorex Corp. (Pelmorex) staff informed EMO staff at the managerial and DO levels that the alert had expired and advised against issuing a second alert. Pelmorex suggested to an EMO supervisor that Pelmorex and EMO coordinate communications to the public. Pelmorex then shared information on the first alert to alert distributors.

In subsequent interviews, Pelmorex maintained that issuing a second broadcast intrusive alert undermined public trust in the Alert Ready system. EMO staff commented that Pelmorex was concerned about “alert fatigue.”

Findings

  • Government officials and stakeholders are to be notified in advance of an alert as per the Provincial Nuclear Emergency Response Plan. The lack of notification caused significant concern and confusion.
  • EMO communications with government and stakeholders between the first and second alert were reactive and handled on a case-by-case basis using inconsistent messaging.
  • Pelmorex consistently advised EMO against issuing a second alert. 
  • Pelmorex’s role on January 12, 2020 exceeded advice on technical use of the Alert Ready system and included advice on how and whether to issue alerts.

Conclusion

The information reviewed in the course of the investigation confirmed that, on January 12, 2020 at 7:23 am, the PEOC issued a broadcast intrusive nuclear emergency alert to Ontarians via the Alert Ready system as a result of human error on the part of a PEOC Duty Officer scheduled on duty that morning.

During routine shift change procedures, the DO logged in to the Alert Ready live system, believed they had switched to the training system, selected a pre-scripted nuclear message for the Pickering Nuclear Generating Station saved by the PEOC in a system template folder, and accidentally sent a live alert. The DO was not acting on any information concerning any nuclear incident. The DO immediately recognized the error and proceeded to seek corrective action from PEOC supervisors.

The findings revealed EMO procedural gaps, lack of training, lack of familiarity with the Alert Ready system and communication failures. These findings can provide context to the DO error and the length of time – 108 minutes – that elapsed between the alert issued in error and the second clarifying alert.

The investigation was conducted with the full cooperation and support of EMO, the PEOC and relevant stakeholders. The findings are provided to assist the DSG-CS and EMO in preventing a similar incident in the future.


Provincial Emergency Operations Centre Action Plan

On the morning of January 12, 2020, the Provincial Emergency Operations Centre issued an emergency alert regarding the Pickering Nuclear Generating Station. The alert was sent in error. The Ministry of the Solicitor General conducted an investigation into the incident that identified a number of concerns with planning, procedures, training, communications and staffing.

The following chart outlines corrective actions taken and planned to help prevent future errors in emergency notifications and to strengthen the public alerting system in Ontario.

Immediate actions implemented since January 12
Area of concern Actions
System verification/ testing process
  1. Modified the Alert Ready System verification sequence to minimize the potential for human error.
  2. All messages used to test the Alert Ready system now clearly indicate they are test messages.
  3. Supervisors have been assigned the task to verify that duty officers are following shift change procedures.
  4. The Alert Ready system verification process now requires two duty officers to conduct the test.
  5. Distinct login credentials are now in place for the live and test platforms of the Alert Ready system to prevent an operator from logging into the live system by mistake.
Staff Training All duty officers have completed refresher training on all procedures related to the Alert Ready system.
“End Alert” Procedure A new procedure is now in place to ensure an “end alert” message is sent immediately should an emergency alert be issued in error.
French Resources
  1. All alert notification template messages, including “end alert” messages are now available in French.
  2. All staff at the Provincial Emergency Operations Centre (PEOC) have been provided refresher training on emergency translation procedures.
Actions underway
Staff Training
  1. All supervisors at the PEOC will undergo training on the Alert Ready system and will be granted appropriate access to the system.
  2. Emergency exercise planning will include elements of critical thinking and atypical situations to test the PEOC decision making process for effective emergency alerts and response.
Provincial Emergency Operations Centre Staffing The ministry will conduct a review of staffing at the Provincial Emergency Operations Centre.
Engagement with partners
National Alert Aggregation and Dissemination Governance Council (NAAD)
  1. The ministry will request clarification from the NAAD governance council on the roles and responsibilities to eliminate any confusion surrounding who is responsible for issuing or ending emergency alerts, including those issued in error.
  2. The ministry will request that the NAAD governance council address the Alert Ready system issues identified in the report.
  3. The ministry will recommend to the NAAD governance council that the Alert Ready system be modified to require two-person verification for all live emergency alerts.