Open Source Monitoring for Java with JMX and Graphite - Softshake 2013

Open Source Monitoring for Java
with Graphite
Cyrille Le Clerc

©2013 CloudBees, Inc. All Rights Reserved
Thursday, October 24, 13

/me
@cyrilleleclerc

CTO

Solution Architect

Cyrille Le Clerc
DevOps, Infra as Code
Open Source
Cloud


2

You ?

DEV ? QA ? OPS ?
Monitoring Experience ?

3

Agenda


4

Agenda
• The demo application
• Monitoring performance: Why? What?
• Java webapp monitoring with JMX
• Monitoring with Graphite
• Conclusion

5

The Application


6

Your mission, should you decide to accept it, ...

e-commerce SOS Cocktails
http://www.ﬂickr.com/photos/23791504@N08/2607814349

Setup technical and business monitoring


7

SOS Cocktail


8

Monitoring
Why ? What ?


9

The Indicators to Monitor

http://www.langevin.com/blog/2013/08/29/3-ways-to-maximize-brainstorming/


10

The Indicators to Monitor
• Operating System ➔ Sysload
• JVM ➔ GC duration
• Tomcat ➔ activeSessions (active visitors)
• Application ➔ sales revenue & items

11



12

Dashboard - wiki
e-commerce web site
Graphite
Email

Seyren

Nagios


13

JMX & Metrics


14

Why JMX ?
• JMX vs. web pages vs. logs vs. ...
• Simple and secured
• Displayable with web pages and logs
• Alternatives to JMX ?

15

JMX vs. logs vs. web pages vs. ...
•

Monitoring logs

•

Parsing GB of text ﬁles ?

•

Per invocation details vs. average per minute

• Web pages
•

History ?

•

Clustered applications ?

•

Security ?


16

JMX with Spring Framework
<beans
...>

...

<context:mbean-‐export/>

...
@ManagedResource("cocktail:name=ShoppingCartController,type=...")
</beans>
class
ShoppingCartController
{

final
AtomicInteger
revenueInCentsCounter
=
new
AtomicInteger();

void
purchase(...){

...

revenueInCentsCounter.addAndGet(priceInCents);

}

@ManagedAttribute

public
int
getRevenueInCentsCounter()
{

return
revenueInCentsCounter.get();

}

17

JMX with VisualVM


18

JMX with JSP
Human readable & script friendly
Watch out for security !
$
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐
metrics.txt.jsp
Epoch

SalesRevenueInCentsCounter
SalesItemsCounter

SalesOrdersCounter
1363217376
4500
7

2
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp

curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...
{

"Epoch":1363217653,

"SalesRevenueInCentsCounter":4050,

"SalesItemsCounter":7,
...
}


19

JMX with JSP
/5ADrUWr5/jmx-sales-metrics.jsp

Human readable & script friendly
Watch out for security !
$
curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐
metrics.txt.jsp
Epoch

SalesRevenueInCentsCounter
SalesItemsCounter

SalesOrdersCounter
1363217376
4500
7

2
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp

curl
http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...
{

"Epoch":1363217653,

"SalesRevenueInCentsCounter":4050,

"SalesItemsCounter":7,
...
}


19

JMX with JSP


20

Alternatives to JMX


21

Alternatives to JMX
r();
ervo
ntege
R)
S

{
micI
UNTE
oller ",
type=CO
=
new
Ato
Contr Cents
Cart
nter
pping revenueIn
tsCou
s
Sho ame="
InCen
clas
enue
r
rev
or(n
onit
ntege

@M
micI
l
Ato

fina
nts);

...){
eInCe

ase(
pric
purch
dGet(
void

addAn

ter.
..
sCoun

.
nCent
nueI

reve

}


21

Alternatives to JMX
Metrics

class
Shopp
ingCartCont
r();
roller
{

ervo
ntege
R)
S

{
micI
UNTE

fController ",
type=CO
=
new
Ato
inal
Countes
rt
Ca

ueInCentr
revenueInCents
nter
pping reven

InCentsCou
Counter
=
M

ShoppingCa
s
Sho ame="
etrics.newC
clas
enue
rtControlle

r
rev
ounter(
or(n
e
t
r,
"revenue
teg
@Moni tomicIn
void

-‐in-‐cents")
purchase(..
;
al
A
n
.){

fi

...
nts);

...){
eInCe

ase(
revenu
pric

purch
eInCedAndGet(
void

.adntsCounter.inc(pr

} Counter
iceInCents)
...
ts
;

InCen
enue

rev

}


21

JMX
• OS: Sysload
• JVM: Garbage Collector
• Tomcat: activeSessions
• Application: Web Site Revenue

22

jmxtrans


23

jmxtrans - standalone
ecommerce
Tomcat

logistics

Graphite

Tomcat

accounting
Tomcat

...

jmxtrans - standalone

24

embedded-jmxtrans
ecommerce
Tomcat

logistics

Graphite
Librato

Tomcat

accounting
Tomcat

batch

...

embedded-jmxtrans

25

jmxtrans-agent
ecommerce
ecommerce
Tomcat
Tomcat
JVM
JVM

JVM

ecommerce
logistics
Tomcat
Tomcat
JVM
JVM
ecommerce
accounting
Tomcat
Tomcat
JVM
JVM

Graphite
Librato

...

batch
JVM

java -javaagent=jmxtrans-agent.jar=...

jmxtrans-agent

26

standalone vs. embedded vs. agent
standalone

embedded

agent

Packaging
Model

Manually connected apps
Pull

Standalone apps
Push

Standalone middleware
Push

Cursor

OPS -♦----------- DEV

OPS ----------♦-- DEV

OPS ---♦--------- DEV

Use case

Net unfriendly monitoring,
OPS

cloud, batch
DEV/QA

cloud, batch
OPS


27

embedded-jmxtrans conﬁguration


28

<dependency>

<groupId>org.jmxtrans.embedded</groupId>

<artifactId>embedded-‐jmxtrans</artifactId>

<version>1.0.8</version>
</dependency>


28

<dependency>



</dependency>

<beans
...

xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"

xsi:schemaLocation="...

http://www.jmxtrans.org/schema/embedded
...”>

<jmxtrans:jmxtrans>

<jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>

...

</jmxtrans:jmxtrans>
</beans>


28

{
<dependency>

"queries":
[


{


"objectName":
"cocktail:name=ShoppingCartController",


"resultAlias":
"sales",
</dependency>

"attributes":
["SalesRevenueInCentsCounter"]

},

],

"outputWriters":
[

{
<beans
...

"@class":
"org.jmxtrans.embedded.output.GraphiteWriter",

xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"

"settings":
{

xsi:schemaLocation="...

"host":
"${graphite.host:localhost}",

http://www.jmxtrans.org/schema/embedded
...”>

"port":
"${graphite.port:2003},"

"enabled":
"${graphite.enabled:true}"

<jmxtrans:jmxtrans>

}

<jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration>

}

...

]

</jmxtrans:jmxtrans>
}
</beans>


28

Graphite


29

Graphite
• Open Source
• Time series database and graphing tool
• Used by tech giants
• Similar

30

Graphite
• Open Source

Simplicity and Self Service

• Time series database and graphing tool
• Used by tech giants
• Similar

30

Metrics injection / Write Access
• Automatic metrics creation
• Socket protocols: “plain text” & Python Pickle serialization
echo
"softshake.happyCounter
1.2
1364338989"
|
nc
localhost
2003

• Precision and storage duration by conﬁguration
[sales_1min_for_30days_5min_for_1year]
pattern
=
^sales.
retentions
=
60s:30d,300s:365d

1.7MB

31

Metrics rendering / Read Access
• Graph Composer & URL API
• Everybody can create graphs
• Wide range of functions
•

•
•

sum, scale, derivative, timeshift, deviation, ﬁlter, ...

Various exports
png, svg, csv, json, raw text


32

The Art of Graphing


33

Monitoring on the Cloud
•
Graphite as a Service
•

http://hostedgraphite.com/

•

Addons: Tasseo

2

Zero setup

•

1

email: admin@jmxtrans.org
password: SOFTSHAKE

3

• No-OPS

34

Metrics Types
Gauge

Ever Increasing Counter

• Examples:
• Information available without transformation

• Examples:
• “per minute” aggregation required

activeRequests, dataSource.activeConnection, ...


requestsCount, revenue, ...

35

Graphs and Maths
?


Revenue per Second


36

Graphs and Maths
derivative()
?


Revenue per Second


36

Graphs and Maths
?

Per Second → Per Hour

Revenue per Hour


37

Graphs and Maths
summarize(..., “1h”)
?

Per Second → Per Hour

Revenue per Hour


37

Graphs and Maths
?

2 servers

Total Revenue per Hour


38

Graphs and Maths
sumSeries()
?

2 servers

Total Revenue per Hour


38

Graphs and Maths
?

server restart

Ignore reset to zero


39

Graphs and Maths
nonNegativeDerivative()
?

server restart

Ignore reset to zero


39

Graphs and Maths
Compare to
last week

?


40

Graphs and Maths
Compare to
last week

timeShift(..., “7d”)
?


40

JMX
• OS: Sysload
• JVM: Garbage Collector
• Tomcat: activeSessions
• Application: Web Site Revenue

41

Graphite URL API
http://localhost:8081/render/?

from=-‐7days&

title=Revenue%20per%20Hour&

vtitle=Dollars&

lineWidth=3&

xFormat=%25a%20%25H%25p&

target=

alias(

scale(

summarize(

sumSeries(

nonNegativeDerivative(

edu.servers.*.sales.revenueInCentsCounter)...)&

...


42

Sharing metrics and graphs


43

Dashboards and Wikis
dashboard.prod.md
#
Website
Traffic
<table>
<tr>

<td>

<img
src="http://graphite.prod.mycompany/graphite/render/?
width=400&height=300&title=Revenue&xFormat=%25a%20%25d
%25p&vtitle=USD&lineWidth=3&from=-‐5days&target=alias(scale(summarize(sumSeries(non
NegativeDerivative(edu.servers.*.sales.revenueInCentsCounter))%2C%221h
%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)...">

</td>

<td>

<img
src="http://graphite.prod.mycompany/graphite/render/?...">

</td>
<tr>
</table>

Wiki

https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Dashboard-PROD


44

Dashboards and Email
Send dashboards by email


45

Alerting with Metrics


46

Graphite Integration with Alerting
• Based on Graphite URL API
• RawText or JSON format
• Pattern

/render?from=-‐11minutes&until=-‐1minutes&format=raw&target=****


47

Graphite Integration with Alerting
• Based on Graphite URL API
• RawText or JSON format
• Pattern

/render?from=-‐11minutes&until=-‐1minutes&format=raw&target=****

Example
$
curl
"http://graphite.example.com/render?
from=-‐11minutes&until=-‐1minutes&format=raw&

target=keepLastValue(servers.cloudbees.jvm.os.SystemLoadAverage)"
my-‐metric,1363225680,1363226340,60|0.03,0.01,0.1,4.0,4.0,0.9,0.7,0.8,0.4,0.5,0.5


47

Graphite Alerting with Seyren

• Alerting Dashboard for Graphite
• Open Source
• Java .war + MongoDB
• Alerts: email, PagerDuty, ...

http://seyren.jmxtrans.cloudbees.net/


48

Graphite Alerting with Nagios
• Monitoring Infrastructure
• Open Source
• De facto standard
• check_graphite plugin
•

Pierre-Yves Ritschard

•

Jason Dixon / obfuscurity


49

Conclusion


50

Conclusion
•

Self-service monitoring changes everything

•

Technical and business monitoring

•

Monitoring integrated in the Continuous Delivery pipeline

•

Open Source solutions are available


51

Hosted Graphite Promo
• http://hostedgraphite.com
• 45 days trial with “SOFTSHAKE” promo code

(3) Promo Code “SOFTSHAKE”
(1) Signup

(2) Options / Enter Promo Code

help: cleclerc@cloudbees.com


52

http://jmxtrans.org/

http://demo-cocktail.jmxtrans.cloudbees.net/


53

Open Source Monitoring for Java with JMX and Graphite - Softshake 2013

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (13)

Mais de CloudBees

Mais de CloudBees (20)

Último

Último (20)

Open Source Monitoring for Java with JMX and Graphite - Softshake 2013