1. DICE Horizon 2020 Project
Grant Agreement no. 644869
http://www.dice-h2020.eu Funded by the Horizon 2020
Framework Programme of the European Union
Monitoring in Big Data
Frameworks
Gabriel Iuhasz
Institute e-Austria Timisoara
26 November 2015

2. Overview
o Introduction
o Cloud Computing and Big Data
o Monitoring Tools
o Monitoring Requirements and Solutions
o Conclusions

3. Introduction
o Big Data in Cloud computing
o Volume, Velocity, Variety and Veracity
o Cost Reduction, Rapid provisioning/time to market,
Flexibility/scalability
o DevOps and Cloud
o Development and Operations
o Communication, Collaboration, Integration,
Automation
o DevOps Monitoring
o Measurement is a key aspect of DevOps

4. Big Data in Cloud Computing
o Challenges of Big Data On Cloud
o Low Latency real-time data
o Virtualization overhead
o Multi-tenancy overhead
o Scalability
o Lack of RDBMS support
o Availability
o Data integrity/privacy

5. Hadoop Ecosystem

6. Cloudera

7. HortonWorks

8. Monitoring Architecture
o Cross layer monitoring of big data platforms
o Types of metrics are highly dependent on the type of the
application
o Have to be decided on a platform/application basis
o Centralized Monitoring
o All resource states are sent to a centralized monitoring server
o Metrics are continuously polled from monitored components
o Single point of failure
o Lacks scalability
o Decentralized Monitoring
o No single point of failure
o Central authority is diffused

9. Tools
o Hadoop Performance Monitoring UI
o Lightweight monitoring UI for Hadoop server
o Uses Hadoop metrics (using Sinks)
o SequenceIQ
o Based on ELK stack and Docker containers
o ElasticSearch can be easily scaled horizontally
o Logstash server on client side
o Ganglia
o Scalable distributed monitoring system
o Low per-node overhead
o Focused on System Metrics
o Gmond, gmetad and Web Front-end

10. Tools II
o Apache Chukwa
o Built on top of HDFS
o Easily scalable
o Potentially high overhead
o Hadoop Vaidya
o Rule Based diagnostic tool for M/R jobs
o Performes post run results analysis
o Nagios
o Plugin based architecture
o Uses a centralized server to collect metrics
o Possible to create a hierarchical deployment

11. Requirements
o Difficulties in cloud monitoring
o Scale
o Velocity or Timeliness
o Constant changes
o The need for scalability and automation
o Easy re-configurability
o Lightweight metrics collectors
o Identifying pertinent metrics

12. DICE Overview
Platform-Indep.
Model
Domain
Models
Continuous
Validation
Continuous
Monitoring
Data
Awareness
Architecture
Model
Platform-Specific
Model
Platform
Description
DICE MARTE
Deployment &
Continuous
Integration
DICE IDE
Big Data
QA
Models

13. DICE Monitoring Platform
o RESTful Web Service
o Used to deploy and configure all core/auxiliary components
o Used to query ElasticSearch
Exports metrics in: JSON, CSV, OSLC Perf. Mon 2.0 (RDF+XML)
o Used for auto-scaling of monitoring solution
o ELK Stack
o Extremely flexible/configurable
o Horizontally scalable
o Can except various input and output formats
o ETL via Logstash server (filters)
o Logstash-forwarder secure transmission (new Beats Data Shippers)
o Visualization using Kibana4
o Collectd
o Statistics collection daemon
o A lot of plugins available
o Simple configuration

14. DICE Monitoring Platform II

15. DICE Monitoring Platform Scaled

16. DICE Monitoring Platform Variant

17. Conclusions
o We have given a short overview of current
monitoring platforms Identified key requirements for
Big Data Monitoring
o Scaling, Autonomy, Timeliness
o Automation via Chef recipes
o Presented the current Architecture of the DICE
Monitoring Platform
o Currently collecting from: HDFS, YARN, Spark, Storm, Kafka
o In the near future: Cassandra possibly Trident
o Creating the full lambda architecture based anomaly
detection platform
o ElasticSearch used as serving layer

18. Thank You!
Questions?