promgen - prometheus management tool simpleclient_java - java library to export metrics

1. promgen - prometheus
management tool
tokuhirom

2. Who are you?

3. Web Application Engineer
• Server side engineer
• Java
• Perl
• Server Admin

4. www1.example.com
node_exporter
(httpd)
jmx_exporter
(httpd)
Periodic GET request

6. Why Prometheus?

7. It’s good for lazy web programmers
Get data → Get stats
Async and lazy

8. Easy to integrate with
grafana

9. But there’s no server
list management tool

10. official method to manage
servers, is writing YAML.
With consul or something

11. YAML file contains:
exporter port number
alerting rules

12. Server admins need to
write BIG YAML FILE.

13. Editing YAML via
github pull-request

14. We want to manage it
via web app.

15. We wrote web application
to manage server list.

16. We are managing the
server list in our
deployment tool

17. promgen can integrate
with our deployment tool
(Of course, it’s pluggable)

19. Note: We haven’t use
consul or similar
tool(now).

20. Promgen
• Simple prometheus manager app
• exporter port management
• alert management

21. Screenshot

26. ProjectService
Project
Project
Host
Host
Host
Data structure
Managed by Deploy tool
Farm
Farm
Farm

27. Alerting
Alert
Manager
Hipchatpromgen-alerting
Web hook
Mail
MySQL

28. Architecture
• Ruby
• Sequel
• MySQL
• erubis
• sinatra

29. Why ruby?

30. I want to run it on
prometheus host.
We shouldn’t use much
memory

31. Part 2.
Monitoring Java
processes via Prometheus

32. There are 3 patterns

33. 1. Collect data from MBean
in target JVM
App JVM
(Runs jmx_exporter jetty)
prom
HTTP

34. • Pros
• Easy to run
• Cons
• Hard to maintain YAML file

35. - pattern: '^com.zaxxer.hikari<type=Pool ((w+)
)><>ThreadsAwaitingConnection'
name: hikaricp_pool_threads_awaiting_connection
labels:
source: "$1"
help: "HikariCP threads awaiting connection(pending
threads) $1"
type: GAUGE

36. 2. Collect data from JMX
outside from target JVM
App
JVM
jmx_exporter
JVM
prom
HTTPJMX

37. • Pros
• Easy to deploy
• Safety
• Cons
• Memory overhead
• Hard to maintain YAML file

38. 3. Collect data by
simpleclient_java
App JVM
(Run simpleclient_java jetty)
prom
HTTP

39. • Pros
• Clean
• lightweight
• no YAML required
• Cons
• Requires coding

40. simpleclient_java

41. simpleclient_dropwizard
Drop wizard
metrics
MetricRegistry
client_java

42. simpleclient_hotspot
JVM core client_java
Default

43. simpleclient_log4j/logback
log4j/logback client_java
Error/Warn rate

44. simpleclient_springboot
Spring Boot
Metrics
client_java
NEW!

45. simpleclient_jetty
Jetty
requests
status
etc.
client_java
NEW!

46. simpleclient_hikaricp
HikariCP client_java
REJECTED!

48. Questions?