OpenNMS - Rule Based Event Management Presentation

1. Rule Based Event Management
Presentation 2013-03-12 / Version: 1.1.2
markus.schneider73@gmail.com
created with

2. Agenda
➢OpenNMS Event Management
 Drools Platform Overview
 Drools Rule Basics
 Activation of Drools
 More Information
OUCE 2013 2

3. OpenNMS Event Management
Event Event Event Alarm Event Alarm
Validation / Duplicate Correlation / Trouble Notification Escalation
Mapping Detection Automation Ticketing
Event Flow (Best Practice)
- Perform - Generate - Update - Open - Notify that - Incident is
validation Alarms Alarms Incident action not solved
Ticket is required in the
- Is event - Find - Clear estimated
defined? Duplicates Alarms - Point to Time
the Root-
- Run Cause
Automations
OUCE 2013 3

4. OpenNMS Event Management
Event Event Event Alarm Focus of this
Event Alarm
presentation
Validation / Duplicate Correlation / Trouble Notification Escalation
Mapping Detection Automation Ticketing
Event Flow (Best Practice)
- Perform - Generate - Update - Open - Notify that - Incident is
validation Alarms Alarms Incident action not solved
Ticket is required in the
- Is event - Find - Clear estimated
defined? Duplicates Alarms - Point to Time
the Root-
- Run Cause
Automations
OUCE 2013 4

5. What is an Event?
 Indication of something that has happend
 Two types of events:
 Internal: Management of OpenNMS
 External: Management of IT-Operations
 Events are defined in eventconf.xml
 Events can have different properties
 Events are received on port 5817 / REST
 Client scripts: 'send-event.pl' / 'send-trap.pl'
OUCE 2013 5

6. Event Anatomy
<event> Unique Universal Event Identifier:
uei.opennms.org/webserver/down
<uei/>
<event­label/>
<descr/> Defines the
7x Severities Event – Alarm
<logmsg/> Relation
<severity/>
<alarm­data/>
<operinstruct/> Runs Action
<mouseovertext/>
<autoaction/>
</event>
OUCE 2013 6

7. Node Discovery
Nodelabel: Interface
sun
$> send­event.pl ­i 127.0.0.1 ­s Discovery
­p "nodelabel sun"
uei.opennms.org/internal/capsd/addInterface
­x 4
Event Type:
Severity: Internal Event
Warning
OUCE 2013 7

8. Event View
Node ID #5
OUCE 2013 8

9. What is an Alarm?
 Alarms are generated by Events
 Reduction-key identifies the Event as an Alarm
 Alarms are processed by Alarmd
 Three types of Alarms:
 "1" - to be a problem that has a possible resolution
 "2" - to be a resolution event
 "3" - for events that have no possible resolution
 Events are linked to Alarms
 Cleared Alarms are removed automatically from the DB
OUCE 2013 9

10. Alarm Anatomy
Duplication
<event> Detection Rule
<uei/>
Clearing
...
Rule
<alarm­data
reduction­key="%uei%:%nodeid%:%parm[#2]%"
alarm­type="2"
clear­key="uei.opennms.org/dbserver/down:
%nodeid%:%parm[#2]%"
auto­clean="true"/>
<update­field field­name="severity">
Grooming
<update­field field­name="logmsg"
Rule update­on­reduction="false"/>
</alarm­data>
Change
</event> Rule
OUCE 2013 10

11. Alarm Rules
Reduction Key
 It's used for event duplication detection (repeat count)
 The granularity determines the amount of reduction
Clear Key
 Used in case of a resolution (alarm-type=2)
 Resolution alarm's clear-key has to match the problem
alarm's reduction-key
OUCE 2013 11

12. Alarm Rules
Update Field
 Allow updates to a few specific alarm fields (i.e. severity)
 lastEventId, lastEventTime, logMsg, and eventParms
are updated by default
Auto Cleaning
 All previous events matching the reduction key of the
current event will be removed from the DB
OUCE 2013 12

13. Alarm View
 Acknowledge Alarms
 Clear Alarms
 Escalate Alarms
OUCE 2013 13

14. Event Processing
„There is a default automation that deletes
unacknowledged alarms whose severity is
'Cleared', so if you want an alarm to go away,
it should be cleared and unacknowledged“
Jeff Gehlbach / OpenNMS
OUCE 2013 14

15. Event Categories
Problem Event
A problem event precedes another event in a sequence.
It is most likely the cause of an symptom event that
arrives later, assuming they are related to the same
component.
Resolution Event
A resolution event indicates the return to a typical state,
thus canceling a problem state. When a resolution event
is received, processing should clear any related problem
events.
Symptom Event
A symptom event is a symptom of some other problem.
The cause of a problem might not always be known when
a symptom event is received.
OUCE 2013 15

16. Simple Event Sequence
Problem Event
uei.opennms.org/webserver/down
Resolution Event
uei.opennms.org/webserver/up
OUCE 2013 16

17. Xzample.events.xml
Create Xzample.events.xml
$> touch $OPENNMS_HOME/etc/events/Xzample.events.xml
OUCE 2013 17

18. Add a Problem Event
to Xzample.events.xml
to Xzample.events.xml
<event>
<uei>uei.opennms.org/webserver/down</uei>
<event­label>Webserver Down</event­label>
<descr>
&lt;p&gt;%parm[subSource]% ­
Status 503 Service Unavailable&lt;/p&gt;
</descr>
<logmsg dest='logndisplay'>
&lt;p&gt;SubSource: %parm[subSource]% is down ­
Source: %parm[source]%&lt;/p&gt;
</logmsg>
<severity>Warning</severity>
<alarm­data reduction­key="%uei%:%nodeid%:%service%"
alarm­type="1"
auto­clean="false" />
</event>
OUCE 2013 18

19. Add a Resolution Event
to Xzample.events.xml
to Xzample.events.xml
<event>
<uei>uei.opennms.org/webserver/up</uei>
<event­label>Webserver Up</event­label>
<descr>
&lt;p&gt;%parm[subSource]% ­ Status 200 OK&lt;/p&gt;
</descr>
<logmsg dest='logndisplay'>
&lt;p&gt;SubSource: %parm[subSource]% is up ­
Source: %parm[source]%&lt;/p&gt;
</logmsg>
<severity>Normal</severity>
<alarm­data reduction­key="%uei%:%nodeid%:%service%"
alarm­type="2"
clear­key="uei.opennms.org/webserver/down:%nodeid%:%service%"
auto­clean="true"/>
</event>
OUCE 2013 19

20. Problem & Resolution Event
Problem Event
reduction­key="%uei%:%nodeid%:%service%"
clear-key == reduction-key
Resolution Event
clear­key="uei.opennms.org/webserver/
down:%nodeid%:%service%"
OUCE 2013 20

21. Extend eventconf.xml
Add the following line to the end of eventconf.xml
$> echo
'<event­file>events/Xzample.events.xml
</event­file>' >>
$OPENNMS_HOME/etc/eventconf.xml
Reload eventconf.xml
$> $OPENNMS_HOME/bin/send­event.pl
uei.opennms.org/internal/eventsConfigChange
OUCE 2013 21

22. Send a Problem Event
Node ID #5
Service
(in this case)
Http
$> ./send­event.pl ­n 5 ­s Http
­p "source send­event.pl"
parm[#1]
­p "subSource webserver1"
uei.opennms.org/webserver/down ­x 7
parm[#2] Severity:
Critical
OUCE 2013 22

23. Event View
OUCE 2013 23

24. Alarm View
OUCE 2013 24

25. Send a Resolution Event
Node ID #5
Service
(in this case)
Http
$> ./send­event.pl ­n 5 ­s Http
­p "source send­event.pl"
parm[#1]
­p "subSource webserver1"
uei.opennms.org/webserver/up ­x 3
parm[#2] Severity:
Normal
OUCE 2013 25

26. Event View
OUCE 2013 26

27. Alarm View
OUCE 2013 27

28. Complex Event Sequence
Workshop Preview - Drools
Workshop Preview - Drools
Symptom Event
CarDirectDown
Problem Event Problem Event
Webserver1 Down Webserver2 Down
Resolution Event
CarDirectUp
Resolution Event Resolution Event
Webserver1 Up Webserver2 Up
OUCE 2013 28

29. Agenda
 OpenNMS Event Management
➢Drools Platform Overview
 Drools Basic Rules
 Activation of Drools
 More Information
OUCE 2013 29

30. Drools Platform Overview
Business Logic Integration Platform
source: http://de.slideshare.net/mariofusco/introducing-drools
Expert Fusion jBPM 5 Planner Guvnor UberFire
OUCE 2013 30

31. Expert & Fusion
Expert
 Basic rule engine – core of the business logic
integration platform
 Operates on set of data (facts)
Fusion
 Can define relationships between facts over the time
 Supports: CEP/ESP, sliding windows, temporal
operators
OUCE 2013 31

32. jBPM5 & Planner
jBPM5
 Flexible and lightweight Business Process
Management (BPM) tool
 Can be integrated with almost all the other modules
 Authoring tool: jBPM5 BPMN2 Eclipse editor
Planner
 Used to optimize automated planning problems
 Combines search algorithm with the core of the
rule engine
OUCE 2013 32

33. Guvnor & UberFire
Guvnor
 Repository for Drools Knowlege Bases
 Web based Gui
 Version management
UberFire
 Uberfire is an Eclipse like workbench (web based),
built of GWT, Errai and CDI
 New Project
OUCE 2013 33

34. Agenda
 OpenNMS Event Management
 Drools Platform Overview
➢Drools Rule Basics
 Activation of Drools
 More Information
OUCE 2013 34

35. Drools Rule Basics
Business Logic Integration Platform
source: http://de.slideshare.net/mariofusco/introducing-drools
Expert Fusion jBPM 5 Planner Guvnor UberFire
OUCE 2013 35

36. Rule Engine
Rule File: NodeParentRules.drl Rule is triggered by facts - event(s):
Rule: "Webserver Down" "uei.opennms.org/webserver/down"
source: http://docs.jboss.org/drools/release/5.5.0.Final/drools-expert-
Inference Engine
Inference Engine
(ReteOO/Leaps)
(ReteOO/Leaps)
Production Working
Memory Memory
Pattern
(rules) (facts)
Matcher
docs/html_single/index.html#d0e128
Agenda
OUCE 2013 36

37. Rule File
 Text file with a .drl extension
 Package declaration must be the first element
 DRL file contains:
 multiple rules, queries & functions, imports,
globals and attributes
 Rules can be spread across multiple rule files
OUCE 2013 37

38. Rule File Anatomy
package package­name
imports
globals
functions
queries
rules
OUCE 2013 38

39. Rule Anatomy
Quotes on Rule names are optional
if the rule name has no spaces
inspried by: http://de.slideshare.net/mariofusco/introducing-drools
rule “<name>”
CONDITION: <attribute> <value>
Pattern-matching
against objects in the when
Working Memory
<LHS> salience(priority) <int>
agenda-group <string>
then no-loop <boolean>
<RHS> auto-focus
duration
<boolean>
<long>
CONSEQUENCE:
Code executed when ...
a match is found
OUCE 2013 39

40. What is a condition/pattern?
Event( uei == „uei.opennms.org/webserver/down”)
inspried by: http://de.slideshare.net/mariofusco/introducing-drools
Field Name Restriction
Object Type Field Constraint
Pattern
OUCE 2013 40

41. Rule Facts
// Java // DRL
public class Event { declare Event
private String uei; uei : String
private int severity; severity : int
inspried by: http://de.slideshare.net/mariofusco/introducing-drools
private int priority; priority : int
private Sting message; message : String
// getter and setter here end
}
// Rule
rule "Change Priority"
no­loop
when
$event : Event( severity == 7 );
then
modify( $event ) { priority = 1 };
end
OUCE 2013 41

42. Rule Consequence
Methods for Handling Facts
Methods for Handling Facts
insert()
 For inserting new facts into the session:
insert( new Event() );
modify()
 For updating existing facts in the session:
modify( $event ) { priority = 1 };
retract()
 For removing existing facts from the session:
retract( $event );
OUCE 2013 42

43. Rule Syntax
Types
Types
String:
Event( uei == ".../webserver/down")
'…' must be replaced
Regular expression: with 'uei.opennms.org'
Event( uei matches ".*nodeDown" )
Date:
Event( createTime > "13­Mar­2013" ) //
"dd­mmm­yyyy"
Boolean:
Event( isAcknowledged == true )
Enum:
Event ( type == Event.Type.CRITICAL )
OUCE 2013 43

44. Rule Syntax
Conditions / Pattern
Conditions / Pattern
And:
Event(uei == ".../webserver/down",
severity < 6)
Or:
Event(uei == ".../webserver/down" ||
severity < 6)
Not:
not Event(uei == ".../webserver/down")
Exists:
exists Event( uei matches "[A­Z][a­z]+" )
OUCE 2013 44

45. Rule Syntax
Variables / Comments
Variables / Comments
Variables
Rules can declare variables as follows:
$event : Event( $uei : uei )
Comments
# single line comment
// single line comment
/* multi line
comment */
OUCE 2013 45

46. Rule Syntax
Package / Imports
Package / Imports
Package
 Group of related rules
package org.opennms.netmgt.correlation.drools;
Imports
 Have the same purpose as standard Java imports
import org.opennms.netmgt.xml.event.Event;
import org.opennms.netmgt.model.events.EventBuilder;
OUCE 2013 46

47. Rule Syntax
Functions / Dialect
Functions / Dialect
Functions
 Can be used in conditions and consequences
function void println(Object msg) {
System.out.println(new Date() + " : " + msg);
}
Dialect
 Specifies the syntax used in any code expression
 Default value is Java
 Drools supports one more dialect called mvel
 Dialect can be set on package or rule level
OUCE 2013 47

48. Timers & Calendars
rule 'Change Severity' When the event is 'unack.', and
timer 5m30s has been 'unack.' for 5m30s then
'ack' it.
when
$evt : Event( acknowledged == false )
inspried by: http://de.slideshare.net/mariofusco/introducing-drools
then
modify( $evt ) { acknowledged = true};
end Drop events on rule 'Maintenance Mode'
weekends calendars "weekend"
when
rule 'Send AutoTask Event' $evt : Event()
timer (cron: 0/5 * * * * *) then
when retract($evt);
Event() end
then
sendEvent(); Send Event every
end five seconds
OUCE 2013 48

49. Agenda
 OpenNMS Event Management
 Drools Platform Overview
 Drools Rule Basics
➢Activation of Drools
 More Information
OUCE 2013 49

50. Activation of Drools
 Drools is part of the correlation engine
 Correlation engine is not activated by default
 Drools needs to be configured
 OpenNMS comes with:
 example Configurations
 example Rules
 OpenNMS uses Drools version: 5.1.1
OUCE 2013 50

51. Configuration
(1) Go to the opennms example directory
$> cd $OPENNMS_HOME/etc/examples
(2) Copy all example configurations and rules
$> cp correlation­engine.xml
drools­engine.xml
LocationMonitorRules.drl
NodeParentRules.drl
nodeParentRules­context.xml
$OPENNMS_HOME/etc
OUCE 2013 51

52. Configuration
(3) Edit service-configuration.xml
uncomment the service named “OpenNMS:Name=Correlator”
in $OPENNMS_HOME/etc/service-configuration.xml
(4) Restart opennms
$> sudo service opennms restart
(5) Check 'spring.log'
$> grep 'drools­correlation­engine'
$OPENNMS_HOME/logs/daemon/spring.log
2013­02­02 09:23:05,854 INFO [Main]
XmlBeanDefinitionReader: Loading XML bean definitions from
URL [jar:file:/usr/share/opennms/lib/drools­correlation­
engine­1.10.2.jar!/META­INF/opennms/correlation­engine.xml]
OUCE 2013 52

53. Event Relationship Example
Problem Event Symptom Event
nodeDown …/webserver/down
webserver events
are created
by Drools
Resolution Event Resolution Event
nodeUp …/webserver/up
OUCE 2013 53

54. Extend NodeParentRules.drl
(1) Add function sendEvent to NodeParentRules.drl
function void sendEvent(DroolsCorrelationEngine
engine,String uei, Long nodeId, String svcName,
String subSource) {
EventBuilder bldr = new
EventBuilder(uei,"Drools")
.setNodeid(nodeId.intValue())
.setService(svcName)
.addParam("source","Drools")
.addParam("subSource",subSource);
engine.sendEvent(bldr.getEvent());
}
OUCE 2013 54

55. Extend NodeParentRules.drl
(2) Add 'Webersever Down' rule to NodeParentRules.drl
rule "Webserver Down"
salience 766
when
Event( uei matches ".*nodeDown",
descr matches ".*503",
$nodeid: nodeid )
then
sendEvent(engine,
"uei.opennms.org/webserver/down",
$nodeid,"Http","webserver1",
"Critical");
println("­­­> Webserver Down Event");
end
OUCE 2013 55

56. Extend NodeParentRules.drl
(3) Add 'Webersever Up' rule to NodeParentRules.drl
rule "Webserver Up"
salience 777
when
Event( uei matches ".*nodeUp",
descr matches ".*200",
$nodeid: nodeid )
then
sendEvent(engine,
"uei.opennms.org/webserver/up",
$nodeid,"Http","webserver1",
"Normal");
println("­­­> Webserver Up Event");
end
OUCE 2013 56

57. Restart & Send Event
(4) Restart OpenNMS
$> sudo service opennms restart
(5) Send problem event
$> ./send­event.pl ­n 5 ­d “Status: 503”
uei.opennms.org/nodes/nodeDown ­x 4
OUCE 2013 57

58. Event View
OUCE 2013 58

59. Alarm View
OUCE 2013 59

60. Send Resolution Event
(6) Send 'nodeUp' Event
$> ./send­event.pl ­n 5 ­d "Status: 200"
uei.opennms.org/nodes/nodeUp ­x 3
OUCE 2013 60

61. Event View
OUCE 2013 61

62. Alarm View
OUCE 2013 62

63. Log File
$OPENNMS_HOME/logs/daemon/output.log
OUCE 2013 63

64. Agenda
 OpenNMS Event Management
 Drools Platform Overview
 Drools Rule Basics
 Activation of Drools
➢More Information
OUCE 2013 64

65. More Information
Presentation
 http://de.slideshare.net/mschneider73
OpenNMS
 http://www.opennms.org/wiki/Events#Events_and_Alarms
 http://www.opennms.org/wiki/Drools_Correlation_Engine
Drools
 http://docs.jboss.org/drools/release/5.2.0.Final/drools-expert-docs/
html/ch05.html
 http://www.jboss.org/drools/presentations
 http://mvel.codehaus.org
OUCE 2013 65

66. Comments & Questions
Thank you for your attention
Contact details:
markus.schneider73@gmail.com
www.rapideca.org
03/14/13 OUCE 2013 66