Ganglia distributed monitoring

1. Ganglia Monitoring
Robert Chen
robertchen117@gmail.com

2. 
INTRODUCTION

Single host monitoring

distributed monitoring

Ganglia

3. INTRODUCTION

Why monitoring?

How monitoring?

Sys status

To resovle system issues

For system upgrade

……

Single host monitoring

Distributed monitoring

4. Single host monitoring

What need be monitored ？

system

CPU Loading ？

memory

space

network

……

hardware

temperature

……

5. 
how to get ？
Single host monitoring

/proc pseudo file-system

Kernel

Kernel parameters

Basic OS commands

ps, free, top, df, dh, ……

Log files

/var/log ……

6. 
/proc/loadavg

CPU ？

/proc/cpuinfo

uptime
Single host monitoring

7. 
memory ？

/proc/meminfo

free

hard disk

df -h

du
Single host monitoring

8. 
hardware

LM sensor

sensors-detect

sensor s

IPMI

OpenIPMI

OpenIPMI-tools
Single host monitoring

9. 
how to process ？

Shell Scripts

tools

MRTG

RRDTool
Single host monitoring

10. 
components

damons on each node of the cluster

the service collecting data from each node

GUI

most efficient tools are Ganglia,Supermon and
Hawkeye.
distributed monitoring

11. Ganglia
Ganglia is a distributed monitoring tool with web GUI

Ganglia Monitoring Daemon(gmond)

Ganglia Meta Daemon(gmetad)

Ganglia PHP Web Frontend

12.  Monitor grid or Cluster
 Multicast-based Listen/An6nounce protocol
 Depends on open standards
– XML
– XDR compact portable data transport
– RRDTool - Round Robin Database
– APR – Apache Portable Runtime
– Apache HTTPD Server
– PHP based web interface
 http://ganglia.sourceforge.net or http://www.ganglia.info

13. Failover
Poll
Poll
FailoverFailover
PollPoll
GMOND
Node
GMOND
Node
GMOND
Node
Cluster 1
GMOND
Node
GMOND
Node
GMOND
Node
Cluster 2
GMETAD
Cluster 3
GMOND
Node
GMOND
Node
GMOND
Node
GMETAD
Apache
Web
Frontend Web
Client
Ganglia Architecture

14. 
Ganglia Monitoring Daemon(gmond)
Gmond is running on all nodes, it collects CPU,memory, space and etc- these
are called “metric”
- Linux src ： ganglia/srclib/libmetrics/linux/
gmond is listening 8649, responsing to gmetad requests
- (try it) telnet localhost 8649 (XML output)
gmond can do broadcast: 239.2.11.71:8649
the XML includes all nodes information
Even just one node is dead does not impact to others

15. 
Ganglia Meta Daemon(gmetad)
Gmond does not save info, and it only collects one cluster, which is why
we need gmetad. Gmetad listens on 8651 and 8652, provide more info to
the client.
- (try it) telnet localhost 8651
- (try it) telnet localhost 8652
gmetad accesses gmond 8649 port for collecting cluster info.
gmetad saves data into RRD files
- files are here ： /var/lib/ganglia/rrds/

16. Ganglia installation
Ganglia depends on ：
zlib-1.2.2-4
libpng-1.2.1-6
freetype-2-2.1.7-2
libart_lgpl-2.3.16-1
perl

17. A. Install ganglia dependacies ：
apr-1.2.8.tar.gz ；
apr-util-1.2.8.tar.gz ；
confuse-2.6.tar.gz ；
expat-2.0.1.tar.gz
Before installing gmetad we have to install RRDTool, default
path ： /usr/local/rrdtool-<VERSION>
tar zxvf rrdtool-*.tar.gz
cd rrdtool-*
./configure --prefix=/usr
make;
make install
1 server installation

18. B. Gmetad installation
gmetad is installed --with-gmetad
tar –zxvf ganglia-3.0.3.tar.gz
cd ganglia-*
./configure --prefix=/usr/local/ganglia CFLAGS="-I/rrd/header/path"
CPPFLAGS="- I/rrd/header/path" LDFLAGS="-L/rrd/library/path" --with-
gmetad
(./configure CFLAGS="-I/usr/local/rrdtool-1.0.50/include" CPPFLAGS="-I/usr/local/rrdtool-1.0.50/include"
LDFLAGS="-L/usr/local/rrdtool-1.0.50/lib" --with-gmetad --enable-gexec)
make;
make install

19. Copy gmetad.init /etc/init.d/:
cd ganglia-3.0.3/gmetad
cp gmetad.init /etc/init.d/gmetad
copy to /etc:
cp gmetad.conf /etc/gmetad.conf
chkconfig gmetad on
chkconfig --list gmetad
GMETAD 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mkdir /var/lib/ganglia/rrds
chown nobody:nobody /var/lib/ganglia/rrds
start gmetad:
service gmetad start
Starting GANGLIA gmetad: [ OK ]
verify gmetad ： telnet localhost 8651

20. Change gmetad.conf configuration
# data_source “another source” 1.3.4.7:8655 1.3.4.8
data_source “SERVER” 10 node1 node2
data_source GMOND
data_source "taskcenter"
setuid_username "root
# default: "/var/lib/ganglia/rrds"
# rrd_rootdir "/some/other/place"
rrd_rootdir "/var/lib/ganglia/rrds"

21. 2 client:gmond installation
install ganglia:
tar zxvf ganglia-3.0.3.tar.gz
cd ganglia-3.0.3
./configure --prefix=/usr/local/ganglia
make;
make install
cd ganglia-3.0.3/gmond
gmond –t > /etc/gmond.conf
cp gmond.init /etc/init.d/gmond
chkconfig gmond on
chkconfig --list gmond
gmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
service gmond start
Starting GANGLIA gmond: [ OK ]

22. Verify gmond ： telnet localhost 8649
gmond.conf:
/etc/gmond.conf:
globals {
setuid = no
user = nobody
cleanup_threshold = 300 /*secs */
}
restart gmond.

23. 3)Install web front
cp -a ganglia-3.0.3/web /var/www/html/ganglia
For apache ， php modue mod_php should be
enabled
/var/www/html/ganglia/conf.php