Introduce open source monitoring project Hawkular. Hawkular is Successor of the JBoss ON (RHQ)

1. Hawkular
Successor of the JBoss ON (RHQ)
JBoss User Group Korea
tedwon
2017-11-29

2. Hawkular
● Open source monitoring project
● Next generation monitoring project of JBoss ON (RHQ since 2006)
○ Hawkular started end of 2014
● Apache License v2
● Provide REST services
● Providing a generic solutions to common problems.
○ From collecting rain sensors data and send an SMS on rain,
○ Monitor docker containers
○ Application Performance Monitoring,
● Adopted to the Middleware management solution in the ManageIQ project

3. For who ?
● IoT entusiast who needs to collect metrics and possibly need to trigger alerts
● Operators who are looking for a solution to store metrics from statsD, collectd,
syslog…​
● Developers of solutions who need long term timeseries database storage
● Users of ManageIQ who are looking for Middleware management
● Users of Kubernetes/Heapster who wants to store docker container metrics in a
long term timeseries database storage

4. Hawkular Projects
● Hawkular Services & Alerts
○ flagship base project
○ based on WildFly
○ provides services
■ store metrics,
■ alert on metrics,
■ keep a graph view of an inventory
● Hawkular Metrics
○ provide TimeSeries Database (TSDB) services
○ Time Series Metrics Engine based on Cassandra for scalability
○ offers a REST API to store and retrieve metrics.
● Hawkular APM
○ Application Performance Management
○ visualize which layer of an application time is spent
○ deprecated in favor of CNCF's Jaeger

5. Red Hat Middleware Management with CloudForms
● ManageIQ project
● Middleware Manager for Red Hat Cloudforms Technology Preview
○ https://access.redhat.com/announcements/2820611
● Provide centralized management of all the JBoss middleware products from
within the CloudForms user interface.
● Red Hat Middleware Management 7.0.TechPreview
○ https://access.redhat.com/documentation/en-us/red_hat_middleware_management/7.0.techpr
eview/html-single/installing_red_hat_middleware_management_with_cloudforms/

6. Hawkular Services Quick Start

7. Hawkular Services Quick Start
● Let’s run Hawkular
○ http://www.hawkular.org/hawkular-services/docs/quickstart-guide/
● Run Cassandra
○ docker run --name hawkular-cassandra -e CASSANDRA_START_RPC=true -d
cassandra:3.0.9
● Run Hawkular Services
○ docker run --name hawkular-services --link=hawkular-cassandra -e
CASSANDRA_NODES=hawkular-cassandra -p 8080:8080 hawkular/hawkular-services
○ docker exec -it hawkular-services bash
○ http://localhost:8080/

8. Hawkular Services Quick Start
● STEP 1: ADD METRICS
● STEP 2: READ THE METRICS
● STEP 3: ADD ALERTING
● STEP 4: TEST THE ALERT TRIGGER
● Metrics REST API
○ http://www.hawkular.org/hawkular-metrics/docs/user-guide/
○ http://www.hawkular.org//docs/rest/rest-metrics.html

9. Hawkular Services Quick Start
● STEP 1: ADD METRICS
○ metrics_day_1.json => weather data collected every 30 min for the whole day of July 15th
2016.
○ curl -v -u jdoe:password -X POST http://localhost:8080/hawkular/metrics/gauges/temperature/raw -d
@metrics_day_1.json -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

10. Hawkular Services Quick Start - Read Metrics
● STEP 2: READ THE METRICS
● from when to when we want the values, and ordered by time
○ curl -v -u jdoe:password -X GET
"http://localhost:8080/hawkular/metrics/gauges/temperature/raw?start=1468578600000&end=
1468594800001&order=ASC" -H "Content-Type: application/json" -H "Hawkular-Tenant:
myTenant"

11. Hawkular Services Quick Start - Read Metrics

12. Hawkular Services Quick Start - Downsampled
● STEP 2: READ THE METRICS
● Want to know the minimum, maximum and average values for buckets that
would last 2 hours
○ curl -v -u jdoe:password -X GET
"http://localhost:8080/hawkular/metrics/gauges/temperature/stats?bucketDuration=2h&start=1
468533600000&end=1468618200001" -H "Content-Type: application/json" -H
"Hawkular-Tenant: myTenant"

13. Hawkular Services Quick Start - Downsampled

14. Hawkular Services Quick Start
● STEP 3: ADD ALERTING
○ curl -v -u jdoe:password -X POST http://localhost:8080/hawkular/alerts/import/all -d
@trigger_definition.json -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

16. Hawkular Services Quick Start
● STEP 4: TEST THE ALERT TRIGGER
○ clear console
○ curl -v -u jdoe:password -X POST
http://localhost:8080/hawkular/metrics/gauges/temperature/raw -d @metrics_day_2.json -H
"Content-Type: application/json" -H "Hawkular-Tenant: myTenant"
● In the logs of Hawkular Services, you should see 3 INFO messages showing
that the conditions were met

17. Hawkular Alerts Quick Start with UI

18. Hawkular Alerts Quick Start with UI
● You can
○ Add New Trigger
○ Ack Alert
○ Resolve Alert
● Run Hawkular Alerts UI
○ docker stop hawkular-services
○ docker stop hawkular-cassandra
○ docker ps
○ cd hawkular-alerts/examples/tutorial/lessons
○ docker rm lessons_hawkularAlerts_1
○ docker-compose up
○ http://localhost:8080/hawkular/alerts/ui
○ ./tutorial-data.sh => generate random sample data every 5secs
○ https://github.com/hawkular/hawkular-alerts/blob/master/examples/tutorial/lessons/lesson-04-ui.adoc

19. Hawkular Alert Lifecycle
https://www.slideshare.net/ponceballesteros/hawkular-alerting

20. Hawkular APM with Jaeger

21. Hawkular APM with Jaeger
● Want to trace performance issue on distributed polyglot systems e.g. msa
○ Java
○ Go
○ Python
○ Node.js
○ C++
● Hawkular developed own contemporary distributed tracing systems
○ http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html
○ for application performance management
○ for distributed & micro-service applications
● e.g., Zipkin, Google Dapper

22. Hawkular APM with Jaeger
● Hawkular deprecated Hawkular APM project
○ http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html
● Instead, Hawkular Team collaborate on the Jaeger project
○ http://www.hawkular.org/blog/2017/04/19/hawkular-apm-jaeger.html
● Uber’s Yuri Shkuro and Red Hat’s Gary Brown, both core contributors to the
Jaeger Distributed Tracing project
○ Hawkular Team contributing to OpenTracing standard
● Distributed tracing based on OpenTracing standard
● OpenTracing is the vendor neutral open standard for distributed tracing
○ Adopted by CNCF (Cloud Native Computing Foundation)

23. ● https://github.com/uber/jaeger
● http://opentracing.io/documentation/pages/quick-start
● https://eng.uber.com/distributed-tracing/
● Run Jaeger
○ docker run -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p5775:5775/udp -p6831:6831/udp
-p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411
jaegertracing/all-in-one:latest
● Jaeger UI
○ http://localhost:16686/
Let’s run Jaeger

24. Jaeger Example
● Spring Boot App /chaining call => Spring Boot /hello => WildFly Swarm /hello
○ https://github.com/pavolloffay/opentracing-java-examples
○ http://www.hawkular.org/blog/2017/06/9/opentracing-spring-boot.html
○ http://www.hawkular.org/blog/2017/07/opentracing-jaxrs.html

25. Jaeger Example

26. Jaeger Example

27. Jaeger - a Distributed Tracing System

28. References
● http://www.hawkular.org/
● http://www.hawkular.org/overview/
● https://access.redhat.com/articles/2221611
● http://www.hawkular.org/hawkular-services/docs/installation-guide/
● http://www.hawkular.org/hawkular-services/docs/quickstart-guide/
● http://www.hawkular.org/hawkular-services/docs/user-guide/inventory/
● http://www.hawkular.org/hawkular-services/docs/user-guide/inventory/inventory-v3-details.html
● http://www.hawkular.org/hawkular-apm/
● http://jaeger.readthedocs.io/en/latest/
● http://www.hawkular.org/hawkular-metrics/docs/user-guide/
● http://www.hawkular.org/hawkular-metrics/docs/user-guide/installation.html
● http://www.hawkular.org/hawkular-metrics/docs/user-guide/configuration.html
● https://www.slideshare.net/ponceballesteros/hawkular-alerting
● http://www.hawkular.org/community/docs/developer-guide/alerts-v1.html
● http://www.hawkular.org/community/docs/developer-guide/alerts-v2.html
● https://github.com/ManageIQ/manageiq-providers-hawkular/blob/master/GETTING_STARTED.md
● https://github.com/hawkular/hawkular-alerts/tree/master/examples/hello-world
● http://highscalability.com/blog/2010/4/27/paper-dapper-googles-large-scale-distributed-systems-tracing.html
● http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html
● https://eng.uber.com/distributed-tracing/
● https://www.slideshare.net/YuriShkuro/distributed-tracing-at-uber-scale-creating-a-treasure-map-for-your-monitoring-data
● http://www.hawkular.org/blog/2017/03/24/distributed-tracing-with-camel.html
● https://www.cncf.io/blog/2017/09/13/cncf-hosts-jaeger/
● https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/

29. Thank you