New Tricks For Training Grid Watchdogs

In control rooms across the United States, technical experts and sophisticated electronic equipment watch over the nation’s vast network of electric power transmission lines. Twenty-four hours a day, they’re looking for any problem that might interfere with the reliable flow of electricity to your home or business.

But what if the information those grid operators get from their computers is wrong?

As it turns out, that’s not such a far-fetched question.

The final report on the 2003 electricity blackout in the northeast United States put some of the blame for the massive disruption on “inadequate situational awareness.” That’s a fancy way of saying gauges in the control room didn’t show what was really happening.

How could such a mistake happen? The answer and the solution may lie in the equipment used to train grid control room operators.

Jeff Dagle, chief electrical engineer at the Pacific Northwest National Laboratory (PNNL) in Richland, Washington, says, “After the 2003 blackout, the U.S. Department of Energy asked us, as an independent contractor, to investigate grid operator training.

We talked with several of the leading vendors who provide simulators for training grid operators. We discovered that the computer software used in these simulators was set up to provide a lot of ‘what if’ situations that involved showing problems with the grid itself. Operators could practice opening and closing circuits and things like that to get around a problem.”

In the 140 grid control rooms throughout North America, data normally gets updated about every four seconds. Often, it’s easy for a human to recognize if there is a problem. And if something happens and the data doesn’t change at the proper interval, very quickly some sort of error indicator should appear. A computer screen might change color, or a light might begin flashing to alert a grid operator that something is wrong.

But on August 14, 2003, one of those alarm processors failed. The computers did not report the problem immediately. Dagle and his study team made a startling discovery.

“At that time, the simulators were set up only to mimic physical disruptions or actions within the real grid. The ‘what-if’ conditions did not include problems that could originate with the computers themselves in these simulator exercises.”

Simulator designers told Dagle’s team that no one had ever asked them to include faulty computer data in their training scenarios.

Teams from PNNL and AREVA T&D, a company that provides computer software for grid operations and grid simulations, worked on a project to improve the training. Fabrice Hudry, manager for Power Systems Application Engineering, says, “We used our existing software to create specific training scenarios that included bad information.”

Next, PNNL and AREVA T&D invited real grid operators from the Bonneville Power Administration to test these new simulations. Hudry says, “These experienced grid operators told us the scenarios we created were realistic and lifelike. At the end of the practice session, they told us the simulation with bad data gave them a new level of awareness, something they could use when they returned to their real control rooms.”

New national reliability standards in effect with the force of law now require certification for grid operators. Continuing education program hours are a part of the requirement, not just passing a test, so grid operators must continue to learn throughout their careers.

AREVA T&D offers their improved simulator training two ways. For utilities with big budgets, AREVA T&D will sell the software to them directly. For utilities with smaller budgets and fewer workers, AREVA T&D offers a different service. Hudry says, “We offer a remote connection so that grid operators can use the simulator to train from their own office. This cuts out the time and expense of travel.”

The next step in improving grid operator training will involve helping controllers develop better awareness of what is going on not just in their own territory, but in sections of the grid bordering their own area or even farther away.

To find out more about grid operator training, visit the Web site of the North American Electric Reliability Council at www.nerc.com/about.