Loss of EMS Events

Loss of EMS Events

2015 System Operator Seminar

Training Objectives

•System Operators will be able to identify the ERO EventAnalysis Process Category associated with a loss ofEMS event

•System Operators will be able to demonstrate through apaper exercise the steps to take in response to a loss ofEnergy Management System (EMS) Event

NERC Event Analysis of EMS Event

ERO Event Analysis Process EMS Event Categories

1f - Unplanned evacuation from a control center facilitywith BPS SCADA functionality for 30 minutes or more.

1h - Loss of monitoring or control, at a control center,such that it significantly affects the entity’s ability to makeoperating decisions for 30 continuous minutes or more.

2b - Complete loss of SCADA, control or monitoringfunctionality for 30 minutes or more.

Examples of 1h Events

Examples include, but are not limited to the following:

i. Loss of operator ability to remotely monitor, control BulkElectric System (BES) elements, or both.

ii. Loss of communications from SCADA RTUs

iii. Unavailability of ICCP links reducing BES visibility

iv. Loss of the ability to remotely monitor and controlgenerating units via AGC

v. Unacceptable State Estimator or Contingency Analysissolutions

Loss EMS Events

Category

% NERC EMS Eventto All Events

24%

81%

All

37%

Data for all qualifying events from October 2010 throughDecember 2014.

Loss of EMS Events

Are loss of EMS events impactful to System Operators?

Have loss of EMS events been impactful to the BES?

Loss of EMS Events

Reasons for EMS events not having BES impact

•Most commonly occur during maintenance

•Planned for off peak hours and weekends with fairly flat loads

•EMS staff is nearby

•Heightened SO awareness

•Preparation and good SO action

•Processes and Procedures

•Training (Practice makes perfect)

•Interconnected Systems

FRCC RC Conservative Operations

Purpose:

•Implemented to maximize the FRCC’s ability to operatethe BES reliably during limited or complete loss of EMSmonitoring capabilities

•Identifies triggers to implement Conservative Operations

•Identifies actions and communications the RCSOshould take during partial or complete loss ofEMS/SCADA or RTCA

Procedure title: FRCC Reliability Coordinator Conservative Operations

Procedure FRCC-MS-OPRC-029

FRCC RC Conservative Operations

•RCSO should declare Conservative Operations (Ops) if RC’sEMS, Data Acquisition, or RTCA is lost and ETR is >30 minutesor if RCSO has a reliability concern during loss of functionality

•In this presentation we will refer to this as a loss of functionality

•RCSO verify on duty supervisor is notified, request ETR, and (ifapplicable) ask about a plan to failover

•RCSO notify SE RC of loss of functionality and potential call forConservative Ops

Estimated Time of Repair (ETR)

Southeastern (SE)

FRCC RC Conservative Operations

Operating Entities (OEs) as referenced in the RCConservative Operations Procedure

•Balancing Authorities (BAs)

•Distribution Providers (DPs)

•Generator Owners (GOs)

•Generator Operators (GOPs)

•Load Serving Entities (LSEs)

•Transmission Operators (TOPs)

•Transmission Owners (TOs)

FRCC RC Conservative Operations

•RCSO to request SE RC notify RCSO of any new contingenciesand schedule discrepancies of >200 MWs until functionality isback

•RCSO to notify all OEs via hotline and FTMS of loss offunctionality and Conservative Ops could be called

•OEs to acknowledge message over hotline

•RCSO to instruct all BAs to maintain ACE as close to zero aspossible and notify RCSO of deviations >±50MWs

FRCC RC Conservative Operations

•OEs to notify RCSO of any new contingencies

•OEs to notify RCSO if they get nominal voltages exceed or ifthey have any real-time overloads on their transmission facilities

•For nuclear switchyards, provide notification to the RCSO ifthe local transmission station voltage is outside ofestablished limits.

FRCC RC Conservative Operations

•RCSO to notify the MRCOO (or designee) of loss of functionalityand Conservative Ops may be called

•RCSO to periodically check the RTCA internet sharing site toview other TOP’s CA output

•RCSO should log all pertinent data and OE reports received in atime sequenced manner

FRCC RC Conservative Operations

If the loss of functionality ETR is unknown or >30 minutes, the RCSOshould declare Conservative Ops.

Actions are similar but the RCSO will notify that Conservative Ops is beingimplemented

Additional steps for SE RC:

•In addition to previous notify SE RC that Conservative Ops isimplemented

•RCSO to contact SE RC for updates on actual vs. scheduled netinterchange (ANI vs. SNI) in 15 minute intervals

•RCSO update SE RC on status of functionality every 60 minutesuntil Conservative Ops is terminated

FRCC RC Conservative Operations

Additional steps for OEs:

•RCSO to notify OEs via hotline and FTMS of loss of functionalityand that Conservative Ops is being implemented

•Request acknowledgement

•OEs to provide ACE on the hotline

•RCSO to review ACE data against the data provided by SE RC

•RCSO request off schedule BAs to correct ACE and instruct FPL toredispatch to make up remaining discrepancies

•RCSO to update the OEs on the status of the EMS every 60minutes until the Conservative Operations have been terminated

FRCC RC Conservative Operations

RCSO to notify MRCOO (or designee) of loss offunctionality and implementation of Conservative Ops

RCSO should:

•Verify with Next-day planner that outaged lines have beenreturned to service

•Defer any planned outages that haven’t been taken untilConservative Ops is terminated

FRCC RC Conservative Operations

RCSO should:

•Assess the need to reduce imports/exports with OEs

•Coordinate any interface reductions with interface owners

•Coordinate with OEs and assess the need to return toservice any planned outage

FRCC RC Conservative Operations

When EMS/SCADA or RTCA is back:

•RCSO to terminate Conservative Ops and notify SE RC,OEs and MRCOO

“Conservative Operations”

All OEs need to understand what it means when the RCcalls for Conservative Operations

“Conservative Operations”

FRCC RC requests:

- OEs acknowledge the message on the hotline

- BAs maintain ACE as close to zero as possible andnotify FRCC RC of any deviations > 50 MWs

- OEs notify FRCC RC of any new contingencies

- OEs notify FRCC RC if nominal voltages exceed alarmlimits, or if any real time overloads exist.

- OEs notify FRCC RC if the local transmission stationson nuclear switchyards is outside of established limits

“Conservative Operations”

•Internally each company may define ConservativeOperations differently

•It’s also important to understand what it means withinyour company

Loss of EMS Exercise

•An automated planned disk management activity beganto run on HPC EMS server

•EMS was operating under normal conditions

•A file system error occurred on the primary EMS serverwhich causes a critical process to stop and the serverbecomes hung

•This causes the replication of data to the backup EMSservers to be unavailable

•EMS-2, EMS-3, and EMS-4 become disabled

Loss of EMS Exercise

Physical Relocation Was not an Option!!!

•Primary EMS server remained enabled

•Therefore no automatic fail-over took place

•Gateway server lost it’s connection with the primary server

•Memory replication services application on the primaryEMS server losses it’s connection to the three otherservers that are designed to step in and take over

•The three backup EMS servers become invalid andunavailable for failover (status disabled)

Loss of EMS Exercise

•FRCC RC and HPC share an EMS

•When HPC losses EMS functionality it usually means the RCwill too

•HPC and the RC lost monitoring and control of theirEMSs for over 1 hour

•HPC and the RC both called for Conservative Ops

•RC worked with Southeaster RC to monitor the stateACE

Loss of EMS Exercise

Assessment Instructions

•Three roles Reliability Coordinator, HPC BalancingCoordinator and HPC System Operator

•Checklist for each

•Sequence of Events provided

•Complete your work package and make sure to get asignature before you break

Questions?