monitoring system for opensource cloud computing, ZooKeeper, Hadoop, Cassandra, HBase....

1. Management/Monitoring for Cloud Computing
! ZooKeeper
! Cassandra
! Hadoop
contact@gruter.com 070-8129-2950

2. Cloumon ?
(Cloumon®)
ZooKeeper, Cassandra, Hadoop ,
.
Copyright © 2011 Gruter Ltd.. All rights reserved
2

3. Cloumon
ZooKeeper Cassandra Hadoop
• Znode • Cassandra • Hadoop
• Znode • Cassandra
• Znode • Hadoop
•
Copyright © 2011 Gruter Ltd.. All rights reserved
3

4. Why Cloumon?
•
–
–
•
– ,
–
•
–
–
Cloumon
Copyright © 2011 Gruter Ltd.. All rights reserved
4

5. Server Storage ERP
SCM
Mail
Virtualization Tech.
Automatic Management Tool
, , ,
IT .
IT
.
Copyright © 2011 Gruter Ltd.. All rights reserved
5

6. Traditional vs. Cloud Computing
Traditional Cloud(AWS)
Request Request
Firewall Security Group
Load Balancer Elastic
AutoScaling
(L4) LoadBalancer
Physical Server Physical Server
EC2 Instance EC2 Instance
(Unix/Linux) (Unix/Linux)
Cloud
Monitoring
Watch
Database Storage Various DB
Cloud Storage
(Oracle) (SAN, NAS) (NoSQL, RDBMS)
Backup
( Tape, Disk Array)
-
-
Copyright © 2011 Gruter Ltd.. All rights reserved
6

7. System Architecture for Cloud Computing
Copyright © 2011 Gruter Ltd.. All rights reserved
7

8. Software Components for Cloud Computing
Infrastructure Service Runtime Environment
Cloud Data Storage Interface Protocol
(Cassandra, Hbase) (Thrift, SOAP, REST, …)
Management API
Lightweight Cont Other Framewor
Cloud File Storage ainer(Spring) ks
(Hadoop)
Data Mapper File System API
Server Virtualization Application Server
(KVM, Xen, VMWare) (Tomcat, .NET, Thrift Server, …)
Development Environment Management/Common Service
Management Portal
IDL Compiler Common Service
(Thrift, WSDL) (Billing, Metering, SSO)
Service Repository/Framework plug-in Service Repository
Development IDE Distributed Coordinator
(Eclipse) (ZooKeeper)
Copyright © 2011 Gruter Ltd.. All rights reserved
8

9. ZooKeeper
•
– Distributed Lock Coordinator
– SPOF(Single Point Of Failure)
– : zookeeper.apache.org
•
–
•
• n replication
– Notification
• , ,
•
–
– High Availability Master Election
– Lock
Copyright © 2011 Gruter Ltd.. All rights reserved
9

10. ZooKeeper
Copyright © 2011 Gruter Ltd.. All rights reserved
10

11. ZooKeeper Usage
1) create()
ZooKeeper
create(“10.1.1.1”,
EPHEMERAL) subA_server1
getChildren(“/service/subA”,
watcher)
10.1.1.1
/service/subA
subA 10.1.1.1
client create(“10.1.1.2”,
event 10.1.1.2 EPHEMERAL)
3) getChildren cache
4) cache subA subA_server2
6) subA / watcher 10.1.1.3 10.1.1.2
cache
event
5)
(EPHEMERAL)
subA_server3
10.1.1.3
Copyright © 2011 Gruter Ltd.. All rights reserved
11

12. ZooKeeper Usage
High Availability: Master Election
1) lock master
2) master IP setData
subA_master_1
ZooKeeper (10.1.1.1)
subA_master /subA/master lock release
client (10.1.1.1)
3) getData(“/service/subA_master”)
4) cache - acquireLock()
5) watcher release event - register watcher
subA_master_2
(10.1.1.2)
6) master lock
7) IP setData
Copyright © 2011 Gruter Ltd.. All rights reserved
12

13. Cloumon ZooKeeper
•
– Znode
• Action
• / /
– CloumonManager Acton
•
– Refresh
• ZooKeeper
– ZooKeeper
–
– Historical
Notification
Listener
CloumonManager
ZooKeeper
Aciton Alarm Node Manager
Manager
(Mail, SMS) Znode
ZooKeeper
Cluster Manager Server #1
Action
Manager
Host Manager ZooKeeper
(Mail, SMS)
Server #2
Node Metrics ZooKeeper
Data Data Server #n
Copyright © 2011 Gruter Ltd.. All rights reserved
13

14. Cloumon ZooKeeper
• Latency min/avg/max: Latency
• Received: Packet
• Sent: Packet
• Outstanding: request
• Zxid: ID
• Mode zookeeper: (leader, follower)
• Node count: znode
• Client Connection ( Connection )
– remoteAddress:
– Queued: request
– Recved: packet
– Sent: packet
– Sid: session Id
– Lop: connection Operation
– Est:
– to : connection session timeout
– Lcxid: Last cxid of this connection
– Lzxid: connection Id
– Minlat: latency
– Avglat: latency
– Maxlat: latency
Copyright © 2011 Gruter Ltd.. All rights reserved
14

15. Cloumon ZooKeepper Node
•
– Cloumon instance ZooKeeper
•
– Tree View
– / /
• ACL
• Recursive
–
• Validation Check Rule
– ACL
• ACL
• ACL
– Event Notification
• Event Listener
• (Node Deleted, Data Changed, Child Changed) Listener
• Action
Copyright © 2011 Gruter Ltd.. All rights reserved
15

16. Cassandra
•
– Facebook ,
– RDBMS NoSQL
•
– Keyspace, ColumnFamily, ColumnKey, Column
•
– N Linux
– Hash(key)
– /
•
– n-replication
–
•
– : Thrift, Avro
– : Java, C/C++, PHP, Python
– 3rd : Hector
Copyright © 2011 Gruter Ltd.. All rights reserved
16

17. Cassandra
Server1
(token=A)
Storage RPC Daemon
Server5 Service (Thrift, Avro)
Server2
(token=V) (token=D)
JMX
Server4 Server3
(token=O) (token=K)
• (P2P)
•
– token token
– Server2: A → D, Server3: D → K, … Server1: V → A
• Partitioner
– Random Partitioner: Hash(Key)
– OrderPreservingPartitioner: Key
Copyright © 2011 Gruter Ltd.. All rights reserved
17

18. Cloumon Cassandra
• Cassandra Seed
– Seed
•
–
–
– Historical
• Cassandra
– , ,
Notification
Data
CloumonManager
Cassandra
Alarm Cassandra Manager
Manager Cassandra
(Mail, SMS) / Cassandra
Cluster Manager Server #1
Action
Manager
Host Manager Cassandra
(Mail, SMS)
Server #2
Metrics Cassandra
Data Data Server #n
Copyright © 2011 Gruter Ltd.. All rights reserved
18

19. Cloumon Cassandra
• 14 , 190 • Size
• db.Caches
• db.Caches
• Capacity
• db.Commitlog • Requests
• db.ColumnFailies • Hits
• db.IndexColumnFamilies
• RecentHitRate
• db.CompactionManager
• db.DynamicEndpointSnitch • MaxRowSize
• db.StorageProxy • MemtableDataSize
• db.StorageService • MemtableSwitchCount
• net.StreamingService
• net.MessagingService • MemtableThroughputInMB
• net.FailureDetector • ReadCount
• request.MutationStage • TotalDiskSpaceUsed
• request.ReadStage
• request.RequestResponseStage
• WriteCount
• request.ReplicateOnWriteStage • RecentRangeLatencyMicros
• request.ReadRepairStage • RecentReadLatencyMicros
• internal.AntiEntropyStage
• RecentWriteLatencyMicros
• internal.GossipStage
• internal.MigrationStage • Joined
• internal.MiscStage • Keyspaces
• internal.StreamStage • LeavingNodes
• internal.InternalResponseStage
• internal.FlushSorter • LiveNodes
• internal.FlushWriter • Load
• internal.HintedHandoff • RPCServerRunning
• internal.FILEUTILS-DELETE-POOL
• internal.MemtablePostFlusher
• RemovalStatus
• Token
• Status
Copyright © 2011 Gruter Ltd.. All rights reserved
19

20. Hadoop
• Cloud Computing
– /
–
• Distributed File System
– Very Large Scale Distributed File System
• 10K nodes, 100 million files, 10 PB
– Use Commodity Hardware
• self-healing: failover, recovery, backup
– Optimized for batch processing
• Distributed/Parallel Computing framework
– MapReduce ,
– HDFS
• /
–
• FIFO, Fair, Capacity
•
–
• Job:
• Task: n Task(Map or Reduce)
– /
•
• , (streaming)
Copyright © 2011 Gruter Ltd.. All rights reserved
20

21. Hadoop
Secondary
NameNode
NameNode
File Namespace
/foo/bar
NameNode
Request Meta
1
Operation
Client
Server Status
Block Management File Block
File Read/Write
DataNode #1 DataNode #2 DataNode #3 DataNode #4 DataNode #5
(rack)1 (rack)2
Copyright © 2011 Gruter Ltd.. All rights reserved
21

22. Cloumon Hadoop
• Hadoop Metrics
– Hadoop metrics context CloumonContext
•
– Hadoop CloumonManager metrics
–
–
– Historical
• Hadoop
– : , ,
Notification
Data
CloumonManager
Hadoop
Alarm Hadoop Manager
Manager
(Mail, SMS)
Hadoop
Cloumon hadoop
Cluster Manager Context Server #1
Action
Manager
Host Manager Cloumon hadoop
(Mail, SMS)
Context Server #2
Metrics Cloumon hadoop
Data Data Context Server #n
Copyright © 2011 Gruter Ltd.. All rights reserved
22

23. Cloumon Hadoop
• NameNode
– FSNamesystem
• BlocksTotal, CapacityRemain, CapacityTotal, CapacityUse
d, FilesTotal, MissingBlocks, TotalLoad
– Dfs.NameNode
• BlockCorrupted, CreateFileOps, Transactions_avg_time
– Jvm.metrics:
• gcCount, gcTimeMillis, memHeapUsed, logWarn, logError,
threadsNew
• DataNode
– Dfs.datanode
• Blocks_operation, bytes_read, bytes_written, avg_time
– Jvm.metrics:
• gcCount, gcTimeMillis, memHeapUsed, logWarn, logError,
threadsNew
Copyright © 2011 Gruter Ltd.. All rights reserved
23

24. Cloumon
ZooKeeper
Copyright © 2011 Gruter Ltd.. All rights reserved
24

25. Cloumon
ZooKeeper
Copyright © 2011 Gruter Ltd.. All rights reserved
25

26. Cloumon
Cassandra
Copyright © 2011 Gruter Ltd.. All rights reserved
26

27. Cloumon
Hadoop
Copyright © 2011 Gruter Ltd.. All rights reserved
27

28. Cloumon
Hadoop
Copyright © 2011 Gruter Ltd.. All rights reserved
28

29. Cloumon
• 1 instance
–
– metrics DB
• MySQL : 2000 ~ 3000 Transaction/sec
–
• 60
• ZooKeeper
– ZooKeeper ZooKeeper ( clien
t 1 record )
– ZooKeeper Client /60
– ZooKeeper
• Cassandra
– 190 /60 *
– Cassandra 600
– 190
• Hadoop
– 77 /60 *
– Hadoop 800
– 77
•
– Historical
– 10 12GB
Copyright © 2011 Gruter Ltd.. All rights reserved
29

30. Open Source vs. Enterprise
Open Source Enterprise
X
Server O
( )
O O
O
ZooKeeper
ACL X O
Listener X O
Cassandra X O
Hadoop O O
User Interface Bad Good
Copyright © 2011 Gruter Ltd.. All rights reserved
30

31. Cloumon release milestone
• Cloumon-0.9: 2011.03
–
– www.cloumon.org
–
– ZooKeeper , Hadoop
• Cloumon-enterprise-1.0: 2011.05
– ZooKeeper
– ZooKeeper ACL, Event Listener
– Cassandra
– Hadoop
– UI
• Cloumon-enterprise-2.0: 2011.07
– Cassandra
– Cassandra Client(Hector)
– (Dashboard)
– Cloumon Manager HA
– UI ( )
–
• Cloumon-enterprise-3.0: 2011.09
– Hadoop MapReduce
– Hbase
Copyright © 2011 Gruter Ltd.. All rights reserved
31