An insight on the set-up and uses of Hydro Generator Expert Systems

1. GE Energy
Experience with
Hydro Generator
Expert Systems
As presented at the Iris Rotating Machine Conference
June 2008, Long Beach, CA
Peter Lewis, Iris
John Grant, GE Energy
J. Evens, NYPA

2. GE Energy | GER-4488 (07/08)

3. Experience with Hydro Generator Expert Systems
Current technological advances in condition monitoring are monitors suitable for this hydro-generator monitoring [1]. In the case
employing an increasing number of complex sensors and of partial discharge monitoring, where no solution existed, a
advanced monitors to diagnose the operating status and condition cooperative R&D effort between NYPA and Iris Power led to the
of hydro generators and turbines. Advanced systems routinely development of a cost effective on-line PD monitor called
employed may include bearing vibration, air gap, partial discharge, HydroTrac™ [2].
and flux monitoring. Proper interpretation of this often complex
Knowledge Base
information can lower operating and maintenance expenses, in
One of the first diagnostic expert systems for on-line turbo-generator
addition to reducing unscheduled outages and catastrophic
monitoring was developed in the 1980s by EPRI and was called
failures. However, the volume of available data from these
GEMS [3]. Although a later attempt to create a commercial system
monitors, and the extensive interpretation necessary to evaluate
based on this research prototype failed, many of the machine
the complex waveforms and spectrums, can overwhelm plant
behavior models developed for GEMS were later very relevant to
personnel and resources. Sophisticated software and algorithms
HydroX. In addition, this system clearly demonstrated the need for
are often necessary to correlate and interpret this data to establish
some form of probabilistic reasoning as complex machine monitoring
the overall generator and drive train condition.
is never fully deterministic. The technical success of GEMS spawned
HydroX™ (for Hydro Expert) is a knowledge-based expert system follow on work by EPRI and others in the area of hydro generator
program for on-line monitoring of hydro-generators. Working with monitoring [4].
the New York Power Authority, the system was developed over five
Recognizing the loss of machines expertise in the hydro industry,
years by Iris Power and GE's Bently Nevada* team. After a further
in the late 1990s NYPA initiated a research project to interview
two years of prototype evaluation at NYPA’s St. Lawrence Power
generator design, operations, and maintenance personnel to
Project on two 60 MVA generators, the validated system is now
document a diagnostic rule set for an expert system like HydroX.
commercially available.
Although at the time, suitable monitors and sensors were still under
The successful development of HydroX was predicated by several key development, and no suitable software platform existed, it was felt
factors, including: that documenting the rules was a critical first step. This was a multi-
1. Available and cost effective on-line monitors for critical year effort using experts from OEMs, industry, academia, and utility
components of the turbine and generator. engineering and operations staff. The result of this project was the
2. Expertise in the form of hydro-generator design, operation, knowledge base that was used later to create HydroX.
and maintenance knowledge that could be codified into expert
system rules. Expert System Tools
3. A suitable commercial software platform or expert system shell. Since the 1980s, expert systems have been a topic of research
aimed at automating monitoring and diagnostics for complex
4. A site where the system could be deployed and evaluated.
industrial equipment. Early attempts involved the use of specialized
Each of these factors is discussed below in greater detail. computer hardware and software which were not robust or ready
for industrial applications. With the growth in popularity and
On-line Monitors
capabilities of desktop PCs, it became possible to develop
As part of an upgrade and life extension project of their hydraulic
distributed client-server applications. During the 1990s a prototype
fleet which began in the late 1990s, NYPA identified several key
system called ACMS (Advance Condition Monitoring System) was
technologies necessary to more completely monitor a large hydraulic
fielded on such a platform but proved too unreliable, slow, and
turbine and generator. In some cases, although on-line monitors were
difficult to configure to be commercially viable. Other vendors
available, their cost or complexity made them prohibitive for inclusion
developed expert system shell programs [5] however, these systems
into an expert-based monitoring system like HydroX. As well as the
suffered from a lack of standard interfaces to sensing and
normal process data, specialized monitors that were considered
monitoring systems. During this period vendors tended to create
critical to the expert system diagnostics include on-line air-gap,
bearing vibration, stator partial discharge and core temperature and islands of technology which were incapable of communicating
vibration. Over time, competition in the market place led to several with each other.
GE Energy | GER-4488 (07/08) 1

4. Only in the past few years have practical PC-based tools been HydroX Features
available for development and commercial deployment of expert
HydroX is a condition-based diagnostic system for hydro-turbine/
system based plant monitoring systems. System 1* is such a tool,
generators. The system is based on a commercial PC-based asset
and contains standard interfaces such as OPC clients/servers
management tool called System 1. System 1 is a distributed software
which allow it to communicate with external third party monitors
product based on a SQL Server database and contains components for
and sensors. In addition, it contains a rule based inference engine
data collection from remote systems via OPC, a production rule engine for
and provides tools for users to develop decision based logic.
processing user defined rules, and a design tool for developing and testing
System 1 also provides a number of analysis and visualization tools
rules and developing custom user interfaces. The rules are the basis for the
that enhance the rule engine by allowing end-users to view data
HydroX System and represent the knowledge base of the expert system.
(historical and current) and rule results in a variety of ways.
Individual rules were created to process input data into more useful relevant
data. Processed data is than fed through various analysis algorithms
Evaluation Site
embedded in rules again, or to provide decision support. Often multiple rules
An ideal time to install the sensors/monitors necessary to support
are created and grouped into “Rulepaks” that are meant to provide specific
a system like HydroX is during a plant refurbishment/upgrade. At
analysis functionality. In this manner, an expert system can be created that
the St. Lawrence Power Project, NYPA was undertaking a plant life
can encode expert knowledge into an automated analysis system.
extension project sequentially on 16 units and this project provided
the perfect platform for evaluating the HydroX rule-set. During Utilizing the knowledge base developed earlier with NYPA, a
each unit’s upgrade, additional sensors were installed to support modular set of HydroX Rulepaks were created in System 1.
the expert system and interfaces to the plant control and Encoding each major sensor group in its own Rulepak facilitated
monitoring systems were created. Using the acquisition portion of the customization of HydroX for the available machine sensor data
HydroX, data was collected from these systems over time on at different sites. If a particular monitor such as PD is not available,
several units, making it possible to identify machine specific then the rules dealing with those inputs can be easily removed,
behavior and characteristics. The generalized rule-set created leaving the rest of the system functional. Some Rulepaks
during the knowledge base development was then customized incorporate corroboration algorithms that can communicate with
through a “tuning” algorithm. These tuning rules were created to other Rulepaks in order to raise confidence in a diagnosis. In this
account for specific generator behaviors due to subtle differences manner HydroX offers a comprehensive system that can draw
in manufacture or external factors such as seasonal changes in upon multiple data paths to reinforce its diagnostic accuracy. The
ambient conditions. addition of more monitoring systems often will lead to a better
diagnosis.
Software Components
One particular challenge in any expert system is dealing with
uncertainty in the data analysis. System 1 has built-in mechanisms
HydroX RulePak System 1 Config HydroX Display
for indicating the severity of a problem. In HydroX this was
extended to utilize a Mycin like uncertainty scheme [6] to combine
System 1 Platform and Database facts from various sensor inputs into a diagnosis with a certainty
factor. As sensor readings vary further from expected values, or
multiple indications of a problem become apparent, the certainty
System 1 Data Acquisition
in the diagnosis of a fault condition increases.
Where possible, the prediction of “expected” value for sensors is
Bently 3500 made based on mathematical models of machine parameters that
are then tuned for the specific unit. These predicted values are
GCS Computer SCADA Computer HydroTrac Controller then compared to the actual measured values and deviations are
Figure 1. System 1 components analyzed by the rules to compute a diagnosis. For example, the
2 GE Energy | GER-4488 (07/08)

5. predictions of thrust bearing pad temperatures are made based on be expected depending on the machine state. HydroX uses this
the thrust bearing oil temperature and the MW load of the information to set mode-specific thresholds for alarms making the
machine. This basic equation is then customized to account for system very sensitive to small variations in readings.
heating/cooling time constants of the machine with load, and to
The machine mode is also used in several instances to calculate
the actual readings obtained at full load for each sensor which
and alarm on the trend of sensor values. The trend of nominal air
vary due to sensor location and other physical properties.
gap, during field flashing for example, can indicate a specific type
of problem that trending at nominal machine load would not
detect.
Figure 2. Graph showing comparison of actual and predicted bearing vibration
based on unit load and bearing oil temperature
Figure 4. Trend plot of measured air-gap changes during a startup at NYPA
For many sensors, the alarm thresholds may be significantly
different depending on the mode of the machine. HydroX has rules
Current industry trends are to move to more automated plants,
to determine the machine mode and where necessary, different
with less on-site expertise and operations staff. As described
thresholds and even rules are executed dependent on this mode.
above, HydroX can calculate and trend key features and synthesize
The specific modes HydroX recognizes are: standstill, mechanical
summary indications from complex data sets from monitors such
runup/rundown, rated-speed de-energized field, field energized but
as vibration, air-gap, PD, etc. Using these intermediate indicators,
unsynchronized, synchronized unloaded, load transient and loaded
along with diagnostic rules, an expert system like HydroX can filter
thermally stable. An example of this behavior would be air gap
and focus attention to abnormal values, and provide diagnosis of
measurements, where significantly different nominal air gaps can
specific faults as well as possible remedies. In addition, trending of
such parameters over years can indicate long-term degradation
that may otherwise go undetected until damage limits are
breached.
NYPA Installation
As part of a plant modernization project, Unit 18 at the St.
Lawrence Power Project was removed from service to be
refurbished/up-rated. During this work, additional sensors and
monitors were installed to instrument the unit for HydroX. In
addition to the conventional unit monitoring connected to the
Figure 3. Depiction of expected air gap trend for different machine states
GE Energy | GER-4488 (07/08) 3

6. Figure 5. HydroX sensor set
plant control system, additional sensors and monitors were added is that since the units are coming off a major overhaul, the number
for partial discharge, bearing vibration, core vibration, back of core of faults has been minimal. In addition, many of the long-term
temperatures, and air-gap. trending rules for conditions such as partial discharge can take
As each of the 16 units in the plant are refurbished (a 10-year years to calculate and are just now providing useful values.
program), the identical sensor set is installed and connected to
HydroX. Once completed, all 16 units will be monitored.
A group of two dedicated PC computers run the HydroX
components; the data acquisition system, the SQL Server
Database, the Diagnostic Rule Engine and the User Interface.
These computers were installed on a separate LAN, and interfaced
to the other necessary plant systems (Generator Control System to
obtain conventional unit sensor data, HydroTrac for PD data, and a
Bently Nevada 3500 rack for air gap and vibration data). The
interfaces to external systems were accomplished using an OPC
Data Interface [7].
Experience to date:
Over the past several years, the prototype HydroX has been moni-
toring Unit 18 (and now several other units as they are refurbished
and instrumented). One difficulty with this approach to deployment
Figure 6. HydroX data interfaces
4 GE Energy | GER-4488 (07/08)

7. One significant problem that only became apparent as additional The creation and testing of these rules was a significant and unan-
units were connected to HydroX related to the tuning of the rules. ticipated effort, but was clearly necessary if HydroX was to be a
The models and algorithms used to provide predicted sensor val- commercial success.
ues require substantial tuning for various constants, which can A similar problem was found with the setting of alarm limits for
only be done once the unit is in service. For the deployment of a measured values. There are a multitude of custom values that
successful commercial system, it is not practical for a Field Service must be set for HydroX to calculate malfunction certainties proper-
Engineer to be on-site waiting on a unit start-up, and for possibly ly. These values are usually known by plant personnel and used for
weeks after that to collect data for the various machine states basic alarming of critical parameters. There are still many values
needed to tune the rules. For this reason, a set of “auto-tuning” that may not be known by plant personnel and also, the sheer
rules were written. These rules track data during initial unit opera- multitude of values would make the collection of these values and
tion, and automatically calculate and enter the specific constants customization of the system extremely time consuming. In many
needed for the various predicted sensor values. The rules use linear cases these values can be based on given machine standards.
regression to determine the dependency of two independent vari- HydroX was built to address this issue by incorporating an auto-
ables on a given sensor input. This dependency is usually calculat- matic tuning system for alarm limits. For example, stator winding
ed during startup as the machine will see the greatest span of temperature limits are set according to winding insulation classes
measurements for a given input. (i.e., NEMA), such standards are used in HydroX to automatically
Figure 7. Partial sample logic of an auto-tuning rule
GE Energy | GER-4488 (07/08) 5

8. choose the proper limits based on machine construction parame- and experts alike by providing them with real-time, easy to
ters. HydroX also allows the end-user to set these values manually understand information. By providing automated data collection
and override the automatic values if required. and analysis, the system minimizes the vast volume of data that
would otherwise have to be collected and analyzed manually. This
A final lesson that can be taken from this experience concerns the
also leads to a greater wealth of data but without jeopardizing the
reliability of the system. In general, a hydro turbine and generator
speed and accuracy of analysis as can be the case when too much
is a true model of reliability with some units in continuing service
data is present. HydroX also reduces the number of annoying
after 50 years. Unfortunately the same cannot necessarily be said
“nuisance alarms” by providing a corresponding certainty with
for the components used to monitor them. It is far more likely that
each diagnosis. It is expected that an expert system like HydroX
a sensor, data acquisition system, computer or network will experi- can extend machine life, reduce forced outages, and reduce
ence a problem than a hydro generator will. Problems with some operation and maintenance expenses.
sensors failing and computer components have occurred since the
original installation of the system in 2005. Software and operating REFERENCES
system problems can also occur in any system relying heavily on 1. J.F. Lyles et al, “Using Diagnostic Technology for Identifying
computer systems and network interfaces. In particular, plant net- Generator Maintenance Needs”, Hydro Review, June 1993, p. 58.
work security has been a source of problems, as network security 2. B.A. Lloyd, S.R. Campbell, G.C. Stone, “Continuous On-line PD
becomes ever more stringent forcing frequent upgrades of soft- Monitoring of Generator Stator Windings”, IEEE Trans EC, Dec.
ware, hardware and protocols—all of which may require reconfigu- 1999, p. 1131.
ration of the various components in HydroX.
3. G.S. Klempner, A. Kornfeld, and B. Lloyd, “The generator expert
monitoring system (GEMS) experience with the GEMS prototype,”
Future Plans
EPRI Utility Motor and Generator Predictive Maintenance
Based on the successful deployment on two units at St. Lawrence,
Workshop, December 1991.
a commercial System 1 Rulepak for HydroX has been created. Over
time this system will be installed on all 16 units at St. Lawrence. It 4. A. Roehl and B. Lloyd, “A developing standard for integrating
is expected, that during future deployments at other sites, new hydroelectric monitoring systems” EPRI Motor and Generator
interfaces will be developed to sensors and monitors from other Conference, Orlando, Nov. 1995.
vendors. Standardized protocols like OPC make this a relatively 5. Nilsen, S., OECD Halden Reactor Project, Inst. for Energiteknikk;
simple effort. Obvious future extensions to the system would be to “Experiences made using the expert system shell G2, Tools for
include support for pump storage units which are often critical and Artificial Intelligence”, 1990, Proceedings of the 2nd International
highly stressed assets. IEEE Conference, 6-9 Nov 1990, page(s): 520-529
Conclusions 6. Rule Based Expert Systems: The MYCIN Experiments of the
Stanford Heuristic Programming Project, BG Buchanan and EH
The HydroX system is an advanced expert system that will help
Shortliffe, eds. Reading, MA: Addison-Wesley, 1984
utilities protect hydro turbine-generators while reducing the cost of
operation by transitioning from preventive to condition-based 7. OPC Foundation – www.opcfoundation.org
maintenance. The system combines advanced fault detection HydroX is a trademark of the New York Power Authority.
knowledge from multiple industry experts with modern data HydroTrac is a trademark of Iris Power Engineering, Inc.
* Bently Nevada and System 1 are trademarks of General Electric Company.
acquisition systems in order to empower maintenance technicians
6 GE Energy | GER-4488 (07/08)

9. Notes
GE Energy | GER-4488 (07/08)

10. GE Energy | GER-4488 (07/08)

11. GE Energy | GER-4488 (07/08)

12. ©2008, General Electric Company. All rights reserved.
GER4488 (07/08)