Enviar pesquisa
Carregar
HMS: Scalable Configuration Management System for Hadoop
•
4 gostaram
•
2,392 visualizações
DataWorks Summit
Seguir
Tecnologia
Negócios
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 16
Recomendados
Hadoop Cluster Management
Hadoop Cluster Management
DataWorks Summit
PostgreSQL9.3 Switchover/Switchback
PostgreSQL9.3 Switchover/Switchback
Vibhor Kumar
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Reducing Risk When Upgrading MySQL
Reducing Risk When Upgrading MySQL
Kenny Gryp
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
PostgreSQL replication
PostgreSQL replication
Masao Fujii
MySQL's new Secure by Default Install -- All Things Open October 20th 2015
MySQL's new Secure by Default Install -- All Things Open October 20th 2015
Dave Stokes
Ora10g Rac Best Practices
Ora10g Rac Best Practices
vasanthkp
Recomendados
Hadoop Cluster Management
Hadoop Cluster Management
DataWorks Summit
PostgreSQL9.3 Switchover/Switchback
PostgreSQL9.3 Switchover/Switchback
Vibhor Kumar
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Reducing Risk When Upgrading MySQL
Reducing Risk When Upgrading MySQL
Kenny Gryp
MySQL Server Backup, Restoration, and Disaster Recovery Planning
MySQL Server Backup, Restoration, and Disaster Recovery Planning
Lenz Grimmer
PostgreSQL replication
PostgreSQL replication
Masao Fujii
MySQL's new Secure by Default Install -- All Things Open October 20th 2015
MySQL's new Secure by Default Install -- All Things Open October 20th 2015
Dave Stokes
Ora10g Rac Best Practices
Ora10g Rac Best Practices
vasanthkp
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
Rajni Baliyan
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Experts, Inc.
PostGreSQL Performance Tuning
PostGreSQL Performance Tuning
Maven Logix
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
Sameer Kumar
My two cents about Mysql backup
My two cents about Mysql backup
Andrejs Vorobjovs
GlassFish v2 Clustering
GlassFish v2 Clustering
Eduardo Pelegri-Llopart
EAP6 performance Tuning
EAP6 performance Tuning
Praveen Adupa
Built-in Replication in PostgreSQL
Built-in Replication in PostgreSQL
Masao Fujii
MySQL Backup and Recovery Essentials
MySQL Backup and Recovery Essentials
Ronald Bradford
Online MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackup
Kenny Gryp
Essential Linux Commands for DBAs
Essential Linux Commands for DBAs
Gokhan Atil
SQL Server vs Postgres
SQL Server vs Postgres
chandra sekhar pathivada , PMP,ITIL,MCTS
What's New in Postgres Plus Advanced Server 9.3
What's New in Postgres Plus Advanced Server 9.3
EDB
PostgreSQL Scaling And Failover
PostgreSQL Scaling And Failover
John Paulett
JBoss AS 7
JBoss AS 7
C2B2 Consulting
Sql server 2012 ha dr nova
Sql server 2012 ha dr nova
Joseph D'Antoni
MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD Tour
Ronald Bradford
Less13 performance
Less13 performance
Amit Bhalla
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Leighton Nelson
Sql server 2012 ha dr 24_hop_final
Sql server 2012 ha dr 24_hop_final
Joseph D'Antoni
Whirr dev-up-puppetconf2011
Whirr dev-up-puppetconf2011
Puppet
Automação do físico ao NetSecDevOps
Automação do físico ao NetSecDevOps
Raul Leite
Mais conteúdo relacionado
Mais procurados
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
Rajni Baliyan
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Experts, Inc.
PostGreSQL Performance Tuning
PostGreSQL Performance Tuning
Maven Logix
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
Sameer Kumar
My two cents about Mysql backup
My two cents about Mysql backup
Andrejs Vorobjovs
GlassFish v2 Clustering
GlassFish v2 Clustering
Eduardo Pelegri-Llopart
EAP6 performance Tuning
EAP6 performance Tuning
Praveen Adupa
Built-in Replication in PostgreSQL
Built-in Replication in PostgreSQL
Masao Fujii
MySQL Backup and Recovery Essentials
MySQL Backup and Recovery Essentials
Ronald Bradford
Online MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackup
Kenny Gryp
Essential Linux Commands for DBAs
Essential Linux Commands for DBAs
Gokhan Atil
SQL Server vs Postgres
SQL Server vs Postgres
chandra sekhar pathivada , PMP,ITIL,MCTS
What's New in Postgres Plus Advanced Server 9.3
What's New in Postgres Plus Advanced Server 9.3
EDB
PostgreSQL Scaling And Failover
PostgreSQL Scaling And Failover
John Paulett
JBoss AS 7
JBoss AS 7
C2B2 Consulting
Sql server 2012 ha dr nova
Sql server 2012 ha dr nova
Joseph D'Antoni
MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD Tour
Ronald Bradford
Less13 performance
Less13 performance
Amit Bhalla
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Leighton Nelson
Sql server 2012 ha dr 24_hop_final
Sql server 2012 ha dr 24_hop_final
Joseph D'Antoni
Mais procurados
(20)
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...
PostgreSQL Replication in 10 Minutes - SCALE
PostgreSQL Replication in 10 Minutes - SCALE
PostGreSQL Performance Tuning
PostGreSQL Performance Tuning
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
My two cents about Mysql backup
My two cents about Mysql backup
GlassFish v2 Clustering
GlassFish v2 Clustering
EAP6 performance Tuning
EAP6 performance Tuning
Built-in Replication in PostgreSQL
Built-in Replication in PostgreSQL
MySQL Backup and Recovery Essentials
MySQL Backup and Recovery Essentials
Online MySQL Backups with Percona XtraBackup
Online MySQL Backups with Percona XtraBackup
Essential Linux Commands for DBAs
Essential Linux Commands for DBAs
SQL Server vs Postgres
SQL Server vs Postgres
What's New in Postgres Plus Advanced Server 9.3
What's New in Postgres Plus Advanced Server 9.3
PostgreSQL Scaling And Failover
PostgreSQL Scaling And Failover
JBoss AS 7
JBoss AS 7
Sql server 2012 ha dr nova
Sql server 2012 ha dr nova
MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD Tour
Less13 performance
Less13 performance
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Sql server 2012 ha dr 24_hop_final
Sql server 2012 ha dr 24_hop_final
Semelhante a HMS: Scalable Configuration Management System for Hadoop
Whirr dev-up-puppetconf2011
Whirr dev-up-puppetconf2011
Puppet
Automação do físico ao NetSecDevOps
Automação do físico ao NetSecDevOps
Raul Leite
App cap2956v2-121001194956-phpapp01 (1)
App cap2956v2-121001194956-phpapp01 (1)
outstanding59
Inside the Hadoop Machine @ VMworld
Inside the Hadoop Machine @ VMworld
Richard McDougall
App Cap2956v2 121001194956 Phpapp01 (1)
App Cap2956v2 121001194956 Phpapp01 (1)
outstanding59
GeekAustin DevOps
GeekAustin DevOps
Matt Ray
Performance evaluation of cloudera impala 0.6 beta with comparison to Hive
Performance evaluation of cloudera impala 0.6 beta with comparison to Hive
Yukinori Suda
A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)
Flowdock
Automation day red hat ansible
Automation day red hat ansible
Rodrigo Missiaggia
A Groovy Kind of Java (San Francisco Java User Group)
A Groovy Kind of Java (San Francisco Java User Group)
Nati Shalom
Ansible & Salt - Vincent Boon
Ansible & Salt - Vincent Boon
MyNOG
2013 11-19-hoya-status
2013 11-19-hoya-status
Steve Loughran
Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...
Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...
Codemotion
A tour of Ansible
A tour of Ansible
DevOps Ltd.
What Big Data Folks Need to Know About DevOps
What Big Data Folks Need to Know About DevOps
Matt Ray
Cloud Foundry Open Tour China
Cloud Foundry Open Tour China
marklucovsky
Serve Meals, Not Ingredients (ChefConf 2015)
Serve Meals, Not Ingredients (ChefConf 2015)
ThirdWaveInsights
Serve Meals, Not Ingredients - ChefConf 2015
Serve Meals, Not Ingredients - ChefConf 2015
Chef
Hoya for Code Review
Hoya for Code Review
Steve Loughran
Cooking with Chef
Cooking with Chef
Ken Robertson
Semelhante a HMS: Scalable Configuration Management System for Hadoop
(20)
Whirr dev-up-puppetconf2011
Whirr dev-up-puppetconf2011
Automação do físico ao NetSecDevOps
Automação do físico ao NetSecDevOps
App cap2956v2-121001194956-phpapp01 (1)
App cap2956v2-121001194956-phpapp01 (1)
Inside the Hadoop Machine @ VMworld
Inside the Hadoop Machine @ VMworld
App Cap2956v2 121001194956 Phpapp01 (1)
App Cap2956v2 121001194956 Phpapp01 (1)
GeekAustin DevOps
GeekAustin DevOps
Performance evaluation of cloudera impala 0.6 beta with comparison to Hive
Performance evaluation of cloudera impala 0.6 beta with comparison to Hive
A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)
Automation day red hat ansible
Automation day red hat ansible
A Groovy Kind of Java (San Francisco Java User Group)
A Groovy Kind of Java (San Francisco Java User Group)
Ansible & Salt - Vincent Boon
Ansible & Salt - Vincent Boon
2013 11-19-hoya-status
2013 11-19-hoya-status
Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...
Jacopo Nardiello - Monitoring Cloud-Native applications with Prometheus - Cod...
A tour of Ansible
A tour of Ansible
What Big Data Folks Need to Know About DevOps
What Big Data Folks Need to Know About DevOps
Cloud Foundry Open Tour China
Cloud Foundry Open Tour China
Serve Meals, Not Ingredients (ChefConf 2015)
Serve Meals, Not Ingredients (ChefConf 2015)
Serve Meals, Not Ingredients - ChefConf 2015
Serve Meals, Not Ingredients - ChefConf 2015
Hoya for Code Review
Hoya for Code Review
Cooking with Chef
Cooking with Chef
Mais de DataWorks Summit
Data Science Crash Course
Data Science Crash Course
DataWorks Summit
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
DataWorks Summit
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
DataWorks Summit
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
DataWorks Summit
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
DataWorks Summit
Managing the Dewey Decimal System
Managing the Dewey Decimal System
DataWorks Summit
Practical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist Example
DataWorks Summit
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
DataWorks Summit
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
DataWorks Summit
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
DataWorks Summit
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
DataWorks Summit
Security Framework for Multitenant Architecture
Security Framework for Multitenant Architecture
DataWorks Summit
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
DataWorks Summit
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
DataWorks Summit
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
DataWorks Summit
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
DataWorks Summit
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
DataWorks Summit
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
DataWorks Summit
Computer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near You
DataWorks Summit
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
DataWorks Summit
Mais de DataWorks Summit
(20)
Data Science Crash Course
Data Science Crash Course
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Managing the Dewey Decimal System
Managing the Dewey Decimal System
Practical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist Example
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Security Framework for Multitenant Architecture
Security Framework for Multitenant Architecture
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Computer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near You
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Último
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
wesley chun
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
UK Journal
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
Khem
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Remote DBA Services
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
Khushali Kathiriya
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
DianaGray10
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
SynarionITSolutions
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
sammart93
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Roshan Dwivedi
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Juan lago vázquez
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
apidays
Último
(20)
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
HMS: Scalable Configuration Management System for Hadoop
1.
HMS: Scalable
Configuration Management System for Hadoop Kan Zhang (IBM) Eric Yang (IBM) June 5, 2012 © 2011 IBM Corporation
2.
Motivation ■
Goal: managing Hadoop stack in a data center – Multiple clusters, 10,000+ nodes, inter-cluster operations ■ Scalability – Traditional client/server architecture is cumbersome to scale – Need to keep server states in sync, load balance client requests – Fault tolerance adds further complexity ■ Real-time interaction and feedback – No poll or push, interact via asynchronous notification – Visibility into cluster state is a major pain point for sys admins ■ Cross-node ordering dependency – Example: start JobTracker after NameNode is running – Simple to specify and efficient to implement 2 © 2011 IBM Corporation
3.
HMS Approach
Controller Agent ZooKeeper Storage ■ ZooKeeper plays a central role – Fault-tolerant and scalable storage – Asynchronous messaging service 3 © 2011 IBM Corporation
4.
ZooKeeper
/ /dir1 /dir2 Watch Ephemeral Queue • A hierarchical namespace of znodes for storing data • Sequential znodes for message queuing • Watches for asynchronous notification • Ephemeral znodes for failure detection 4 © 2011 IBM Corporation
5.
Leveraging ZooKeeper ■
Storing cluster state – Each cluster node is represented by a znode in ZooKeeper – Node state is stored in its corresponding znode ■ Storing system state – Any state needed for failure recovery is persisted in ZooKeeper – Server failures are detected via ephemeral nodes ■ Distributed orchestration – Messages are exchanged asynchronously via ZooKeeper queues – Notifications are triggered by watches on queues ■ Cross-node dependency – Node states are stored in ZooKeeper and accessible to all – Watches are registered on state znodes to get notified of state change 5 © 2011 IBM Corporation
6.
HMS Overview
ZooKeeper cmd- queue cmd1 cmdStatus Status- watch queue action- action1 agent NameNode queue Controller1 worklog /hms clusters cluster1 Controller2 action- action1 agent JobTracker queue Client worklog cluster2 Controller1 live- controllers Controller2 6 © 2011 IBM Corporation
7.
Design Implications ■
All cluster state is stored in ZooKeeper – Built-in fault-tolerance and HA – Cluster and command status at your finger tips ■ Controllers and agents don’t interact directly – All communications are via ZooKeeper async notifications – Good for scalability and fault-isolation ■ Controllers and agents are stateless – Controllers can be replicated for load balancing – Controller failures are automatically detected and handled ■ Dependency specified in terms of node states – Actions come and go, but their effects are captured in node states – Node state changes will trigger dependent actions to be re-evaluated 7 © 2011 IBM Corporation
8.
Node List
{ "@url":"http://10.0.1.201:4080/v1/nodes/manifest/test", "roles":[ {"@name":"namenode","host":"host1"}, {"@name":"jobtracker","host":"host2"}, {...} ] } Roles map to hostnames 8 © 2011 IBM Corporation
9.
Package Manifest {
"@url": "http://10.0.1.201:4080/v1/software/stack/hadoop-1.0.3", "@name": "hadoop", "@version": "1.0.3", "roles": [ { "@name": "namenode", Define software stack by roles "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" } ] }, { "@name": "datanode", "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" Software download URL } ] } ] } 9 © 2011 IBM Corporation
10.
Configuration Plan {
“@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] } Run a list of scripts to configure NameNode 10 © 2011 IBM Corporation
11.
Start NameNode {
“@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] { } "@type": "scriptAction", "expectedResults": { "type": "DAEMON", "name": "hadoop-namenode", Define expected "status": "STARTED" result }, "script": "/usr/sbin/hadoop-setup-hdfs.sh", "parameters": [ "--format", "--hdfs-user=hdfs", "--mapreduce-user=mapred", "--namenode-host=${namenode}" ] }, Run a script to setup HDFS on namenode 11 © 2011 IBM Corporation
12.
Compiled Plan {
"startTime":"Thu Jun 07 13:11:29 {PDT 2012", "action":{ "status":"SUCCEEDED", "@action":"DaemonAction", "daemonName":"hadoop-namenode", "clusterName":"my-test-cluster", "actionId":1, "cmdPath":"/cmdqueue/cmd-0000000000", "actionEntries":[ "actionType":"start", "expectedResults":[ { ... }, { "name":"hadoop-namenode", { ... } "type":"DAEMON", "status":"STARTED" ], } ], "completedActions":6, "role":"namenode" }, "totalActions":6, "hostStatus":[ { "endTime":"Thu Jun 07 13:12:07 PDT 2012" "host":"bdvm021.svl.ibm.com", Host Status "status":"SUCCEEDED" } } ] }, 12 © 2011 IBM Corporation
13.
Start JobTracker {
“@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] { "@type": "daemonAction", } "actionType": "start", "dependencies": { "states": { Dependency "type": "DAEMON", of namenode "name": "hadoop-namenode", "status": "STARTED" started }, "roles": "namenode" }, "expectedResults": { "type": "DAEMON", "name": "hadoop-jobtracker", "status": "STARTED" }, "daemon": "hadoop-jobtracker" }, Start JobTracker 13 © 2011 IBM Corporation
14.
Compiled Plan 2 {
{ "action":{ "startTime":"Thu Jun 07 13:11:29 PDT 2012", "@action":"DaemonAction", "daemonName":"hadoop-jobtracker", "status":"SUCCEEDED", "actionId":5, "cmdPath":"/cmdqueue/cmd-0000000000", "clusterName":"my-test-cluster", "actionType":"start", "dependencies":[ "actionEntries":[ { "roles":[ { ... }, "namenode" ], { ... } "hosts":[ "/clusters/my-test-cluster/bdvm022.svl.ibm.com" ], ], "states":[ "completedActions":6, { "name":"hadoop-namenode", "totalActions":6, "type":"DAEMON", "status":"STARTED" "endTime":"Thu Jun 07 13:12:07 PDT 2012" } ] } } ], "role":"jobtracker" }, }, Translated Dependency 14 © 2011 IBM Corporation
15.
Node State [zk: localhost:2181(CONNECTED)
5] get /hms/clusters/my-test- cluster/bdvm021.svl.ibm.com { "states":[ { ... }, { … }, { "name":"hadoop-namenode", "type":"DAEMON", "status":"STARTED" } ] } State updates based on status reported by Agent 15 © 2011 IBM Corporation
16.
Q&A • HMS prototype
is available on GitHub https://github.com/macroadster/hms • Credits Kan Zhang (kzhang@apache.org) Eric Yang (eyang@apache.org) Jagane Sundar (jagane@apache.org) 16 © 2011 IBM Corporation