SlideShare uma empresa Scribd logo

Apache Slider

•Transferir como PPTX, PDF•
•
Shivaji Dutta
Shivaji Dutta

Apache Slider is an open source software for delivering long running jobs on Yarn

Software
1 de 29
Page1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Slider Shivaji Dutta Sr. Partner Solutions Engineer
Page2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Disclaimer This document may contain product features and technology directions that are under development or may be under development in the future. Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. This document’s description of these features and technology directions does not represent a contractual commitment from Hortonworks to deliver these features in any generally available product. Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Page3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Agenda • Apache Slider Overview • Yarn Overview • Why Slider • Slider Internals/Architecture • Slider App Packaging • Ambari and Slider • Q/A
Page4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Overview
Page5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Slider - Open Source in-incubation Project - http://slider.incubator.apache.org/index.html - Platform for - Deployment, Management & Monitoring - Long Running applications on a Hadoop/YARN Cluster - Built and Runs on Hadoop YARN Framework - It makes it EASY and SIMPLE
Page6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN
Page7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN as Cluster Operating System - Hadoop 2.0 - Resource Manager for Hadoop Cluster
Page8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN • A global ResourceManager • A resource arbitrator for the cluster • A per application ApplicationMaster • A resource negotiator for the Application • Works with the Node Manager to Launch Application Containers • A per-node slave NodeManager • Manages Resources on a Node • a per-application Container running on a NodeManager • Actual application running in the container
Page9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Flow NodeManager NodeManager NodeManager NodeManager Container 1.1 Container 2.4 ResourceManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager Container 1.2 Container 1.3 AM 1 Container 2.2 Container 2.1 Container 2.3 AM2 SchedulerClient2 Client1
Page10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN - Powerful but Complex • Powerful – Fine grained control through API • Needs Coding and Development work for creating - Yarn Application Master - Yarn Client - Yarn Container - Complex & Time Consuming to write - For Standard Applications - No Easy way of State Management - THAW - FLEX
Page11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Long Running Applications - Difference from Map-Red
Page12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Long Running Application - Needs - Management - Install - Configure - Start/Stop - Reconfigure - Activate/Reactivate - Upgrade - Rolling Upgrade - Security - Scalability - Monitoring
Page13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider
Page14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Why Slider ? • Full YARN-integration takes effort • Code for every component and action • Powerful and finer control • YARN delivers access to all the data in HDFS –and the Cluster Resources • Maturing Hadoop stack needs an Agile platform to integrate • E.g. HBASE, HIVE, MAP REDUCE, APP Servers • Integrate to Management tools like - Ambari– to monitor applications in- a cluster
Page15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider’s view of an Application Page 15 • An application is a set of components • A component is a daemon/launched exe – configuration – scripts, data files, etc. • Component may have one or more instances • Component instances are managed • Example – HBase Application (3 components) – HBase Master – HBase RegionServer – HBase REST service
Page16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider – Design (On Yarn) Page 16 YARN Node Manager Component (container)AppMaster (container) YARN Node Manager HDFS Slider Agent Application Slider AppMaster Slider Client HDFS HDFS YARN Resource Manager
Page17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Application by Slider Page 17 Slider App Package Slider CLI HDFS YARN Resource Manager “The RM” HDFS YARN Node Manager Agent Component HDFS YARN Node Manager Agent Component 1. CLI starts an instance of the AM 2. AM requests containers 3. Containers activate with an Agent 4. Agent gets application definition 5. Agent registers with AM 6. AM issues commands 7. Agent reports back, status, configuration, etc. 8. AM publishes endpoints, configurations Application Registry App Master/Agent Provider
Page18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider AppMaster/Agent/Client Page 18 AppMaster  Common YARN interactions  Common *-client interactions  Publishing needs Agent  Configure and start  Re-configure and restart  Heartbeats & failure detection  Port allocations and publishing  Custom commands if any (e.g. graceful-stop) Client  App life cycle commands (flex, status, …)
Page19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Terminology Apps on YARN • Application written to run directly on YARN • Packaging, deployment and lifecycle management are custom built for each application Slider Apps • Applications deployed and managed on YARN using Slider • Use of slider minimizes custom code for deployment + lifecycle management • Requires apps to follow Slider guidelines and packaging ("Sliderize")
Page20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider – Getting Started
Page21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Executing Slider • Install Apache Slider on to a Yarn Cluster • Create a “sliderized” Application Package • Setup the config files • Execute it from Slider client E.g. ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz - -template /work/appConf.json --resources /work/resources.json slider <ACTION> [<name>] [<OPTIONS>]
Page22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Installing Slider • 3 easy steps • Download and build apache slider project • Install Slider Client that can access the Hadoop Cluster • Deploy the slider resources • Create the hdfs folders • Done! – Ready to rock! hdfs dfs -copyFromLocal ${slider-install-dir}/slider-0.40.0/agent/slider- agent.tar.gz /user/yarn/agent
Page23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Commands Sample Slider commands • Build - Build an instance of the given name, with the specific options • Create – Build and run an instance • Destroy - Destroy a (stopped) applicaton instance • Exists - Probe the existence of the named Slider application instance • Flex - Flex the number of workers in an application instance to the new value • Freeze - freeze the application instance. The running application is stopped. Its settings are retained in HDFS. • Complete Man page http://slider.incubator.apache.org/docs/manpage.html
Page24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Application Packaging The main components • App Configuration • Configurations needed for the Application • appConfig.json • Resources • Resources required to run the application on the cluster • CPU, Memory, Priority • resources.json • Application Definition • MetaInfo.xml • Application jar file • Actual binary file
Page25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved © Hortonworks Inc. 2014 Memcached on YARN Sample Slider App Page 25
Page26 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Other Application Packages Page 26 Reference doc for Memcached Application • http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_app.html Slider github repo has other app  Accumulo  HBase  Storm  Memcached-windows
Page27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Next?
Page28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Its get Better Ambari Views for Slider • Ambari View that manages the life cycle of “Slider”ized apps
Page29 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Thank You

Recomendados

Customizing LLMs
Customizing LLMsCustomizing LLMs
Customizing LLMs
Jim Steele
 
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Po-Chuan Chen
 
Enterprise Knowledge Graph
Enterprise Knowledge GraphEnterprise Knowledge Graph
Enterprise Knowledge Graph
Lukas Masuch
 
Onnx and onnx runtime
Onnx and onnx runtimeOnnx and onnx runtime
Onnx and onnx runtime
Vishwas N
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML Lifecycle
Databricks
 
DevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflowsDevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflows
Márton Kodok
 
ONNX - The Lingua Franca of Deep Learning
ONNX - The Lingua Franca of Deep LearningONNX - The Lingua Franca of Deep Learning
ONNX - The Lingua Franca of Deep Learning
Hagay Lupesko
 
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUHow We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
Databricks
 

Recomendados

Customizing LLMs
Customizing LLMsCustomizing LLMs
Customizing LLMs
Jim Steele
 
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Po-Chuan Chen
 
Enterprise Knowledge Graph
Enterprise Knowledge GraphEnterprise Knowledge Graph
Enterprise Knowledge Graph
Lukas Masuch
 
Onnx and onnx runtime
Onnx and onnx runtimeOnnx and onnx runtime
Onnx and onnx runtime
Vishwas N
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML Lifecycle
Databricks
 
DevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflowsDevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflows
Márton Kodok
 
ONNX - The Lingua Franca of Deep Learning
ONNX - The Lingua Franca of Deep LearningONNX - The Lingua Franca of Deep Learning
ONNX - The Lingua Franca of Deep Learning
Hagay Lupesko
 
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUHow We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
How We Scaled Bert To Serve 1+ Billion Daily Requests on CPU
Databricks
 
Vector database
Vector databaseVector database
Vector database
Guy Korland
 
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Mihai Criveti
 
Tensor flow
Tensor flowTensor flow
Tensor flow
Nikhil Krishna Nair
 
How to fine-tune and develop your own large language model.pptx
How to fine-tune and develop your own large language model.pptxHow to fine-tune and develop your own large language model.pptx
How to fine-tune and develop your own large language model.pptx
Knoldus Inc.
 
Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"
Fwdays
 
From an idea to production: building a recommender for BBC Sounds
From an idea to production: building a recommender for BBC SoundsFrom an idea to production: building a recommender for BBC Sounds
From an idea to production: building a recommender for BBC Sounds
Tatiana Al-Chueyr
 
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and PrometheusRobust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
Manasi Vartak
 
How to deploy machine learning models into production
How to deploy machine learning models into productionHow to deploy machine learning models into production
How to deploy machine learning models into production
DataWorks Summit
 
Introduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AIIntroduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AI
Semantic Web Company
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language Models
Leon Dohmen
 
Introduction to Neo4j
Introduction to Neo4jIntroduction to Neo4j
Introduction to Neo4j
Neo4j
 
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
Databricks
 
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDistributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Databricks
 
GitHub Copilot.pptx
GitHub Copilot.pptxGitHub Copilot.pptx
GitHub Copilot.pptx
Luis Beltran
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMs
SylvainGugger
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
Building an Enterprise Knowledge Graph @Uber: Lessons from Reality
Building an Enterprise Knowledge Graph @Uber: Lessons from RealityBuilding an Enterprise Knowledge Graph @Uber: Lessons from Reality
Building an Enterprise Knowledge Graph @Uber: Lessons from Reality
Joshua Shinavier
 
An Introduction to Druid
An Introduction to DruidAn Introduction to Druid
An Introduction to Druid
DataWorks Summit
 
MySQL Case Study
MySQL Case StudyMySQL Case Study
MySQL Case Study
Sharath Ghosh
 
End-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNXEnd-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNX
Nick Pentreath
 
Deploying Docker applications on YARN via Slider
Deploying Docker applications on YARN via SliderDeploying Docker applications on YARN via Slider
Deploying Docker applications on YARN via Slider
Hortonworks
 
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveLLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
DataWorks Summit/Hadoop Summit
 

Mais conteúdo relacionado

Mais procurados

Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Mihai Criveti
 
Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"
Fwdays
 
How to deploy machine learning models into production
How to deploy machine learning models into productionHow to deploy machine learning models into production
How to deploy machine learning models into production
DataWorks Summit
 
Introduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AIIntroduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AI
Semantic Web Company
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language Models
Leon Dohmen
 
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDistributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Databricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
An Introduction to Druid
An Introduction to DruidAn Introduction to Druid
An Introduction to Druid
DataWorks Summit
 
End-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNXEnd-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNX
Nick Pentreath
 

Mais procurados (20)

Vector database
Vector databaseVector database
Vector database
 
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...
 
Tensor flow
Tensor flowTensor flow
Tensor flow
 
How to fine-tune and develop your own large language model.pptx
How to fine-tune and develop your own large language model.pptxHow to fine-tune and develop your own large language model.pptx
How to fine-tune and develop your own large language model.pptx
 
Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"Thomas Wolf "Transfer learning in NLP"
Thomas Wolf "Transfer learning in NLP"
 
From an idea to production: building a recommender for BBC Sounds
From an idea to production: building a recommender for BBC SoundsFrom an idea to production: building a recommender for BBC Sounds
From an idea to production: building a recommender for BBC Sounds
 
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and PrometheusRobust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus
 
How to deploy machine learning models into production
How to deploy machine learning models into productionHow to deploy machine learning models into production
How to deploy machine learning models into production
 
Introduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AIIntroduction to Knowledge Graphs and Semantic AI
Introduction to Knowledge Graphs and Semantic AI
 
And then there were ... Large Language Models
And then there were ... Large Language ModelsAnd then there were ... Large Language Models
And then there were ... Large Language Models
 
Introduction to Neo4j
Introduction to Neo4jIntroduction to Neo4j
Introduction to Neo4j
 
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
Apache Spark vs Apache Spark: An On-Prem Comparison of Databricks and Open-So...
 
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and PandasDistributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas
 
GitHub Copilot.pptx
GitHub Copilot.pptxGitHub Copilot.pptx
GitHub Copilot.pptx
 
Fine tuning large LMs
Fine tuning large LMsFine tuning large LMs
Fine tuning large LMs
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
 
Building an Enterprise Knowledge Graph @Uber: Lessons from Reality
Building an Enterprise Knowledge Graph @Uber: Lessons from RealityBuilding an Enterprise Knowledge Graph @Uber: Lessons from Reality
Building an Enterprise Knowledge Graph @Uber: Lessons from Reality
 
An Introduction to Druid
An Introduction to DruidAn Introduction to Druid
An Introduction to Druid
 
MySQL Case Study
MySQL Case StudyMySQL Case Study
MySQL Case Study
 
End-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNXEnd-to-End Deep Learning Deployment with ONNX
End-to-End Deep Learning Deployment with ONNX
 

Destaque

E01.05 the role of the pm pa1 140826
E01.05 the role of the pm pa1 140826E01.05 the role of the pm pa1 140826
E01.05 the role of the pm pa1 140826
rofr1
 

Destaque (12)

Deploying Docker applications on YARN via Slider
Deploying Docker applications on YARN via SliderDeploying Docker applications on YARN via Slider
Deploying Docker applications on YARN via Slider
 
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in HiveLLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
 
YARN and the Docker container runtime
YARN and the Docker container runtimeYARN and the Docker container runtime
YARN and the Docker container runtime
 
LLAP: long-lived execution in Hive
LLAP: long-lived execution in HiveLLAP: long-lived execution in Hive
LLAP: long-lived execution in Hive
 
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage SubsystemEvolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
 
Ethos FR-Ingredient for success!
Ethos FR-Ingredient for success!Ethos FR-Ingredient for success!
Ethos FR-Ingredient for success!
 
Natural Perfumes
Natural PerfumesNatural Perfumes
Natural Perfumes
 
E01.05 the role of the pm pa1 140826
E01.05 the role of the pm pa1 140826E01.05 the role of the pm pa1 140826
E01.05 the role of the pm pa1 140826
 
Google's Duplicate Internet Content Filter In Action
Google's Duplicate Internet Content Filter In ActionGoogle's Duplicate Internet Content Filter In Action
Google's Duplicate Internet Content Filter In Action
 
what are seo contests and how to rank better in SEO Contests.
what are seo contests and how to rank better in SEO Contests.what are seo contests and how to rank better in SEO Contests.
what are seo contests and how to rank better in SEO Contests.
 
Internet Goldrush
Internet GoldrushInternet Goldrush
Internet Goldrush
 
Traffic, Adsense, Money - The Three Card Trick of Internet Marketing
Traffic, Adsense, Money - The Three Card Trick of Internet MarketingTraffic, Adsense, Money - The Three Card Trick of Internet Marketing
Traffic, Adsense, Money - The Three Card Trick of Internet Marketing
 

Semelhante a Apache Slider

Accumulo Summit 2014: Accumulo on YARN
Accumulo Summit 2014: Accumulo on YARNAccumulo Summit 2014: Accumulo on YARN
Accumulo Summit 2014: Accumulo on YARN
Accumulo Summit
 
Developing YARN Applications - Integrating natively to YARN July 24 2014
Developing YARN Applications - Integrating natively to YARN July 24 2014Developing YARN Applications - Integrating natively to YARN July 24 2014
Developing YARN Applications - Integrating natively to YARN July 24 2014
Hortonworks
 
Bring your Service to YARN
Bring your Service to YARNBring your Service to YARN
Bring your Service to YARN
DataWorks Summit
 
Docker based Hadoop provisioning - anywhere
Docker based Hadoop provisioning - anywhere Docker based Hadoop provisioning - anywhere
Docker based Hadoop provisioning - anywhere
Janos Matyas
 
Apache Hadoop YARN: state of the union
Apache Hadoop YARN: state of the unionApache Hadoop YARN: state of the union
Apache Hadoop YARN: state of the union
DataWorks Summit
 
Apache Hadoop YARN: best practices
Apache Hadoop YARN: best practicesApache Hadoop YARN: best practices
Apache Hadoop YARN: best practices
DataWorks Summit
 
Running Non-MapReduce Big Data Applications on Apache Hadoop
Running Non-MapReduce Big Data Applications on Apache HadoopRunning Non-MapReduce Big Data Applications on Apache Hadoop
Running Non-MapReduce Big Data Applications on Apache Hadoop
hitesh1892
 

Semelhante a Apache Slider (20)

YARN Ready - Integrating to YARN using Slider Webinar
YARN Ready - Integrating to YARN using Slider WebinarYARN Ready - Integrating to YARN using Slider Webinar
YARN Ready - Integrating to YARN using Slider Webinar
 
Accumulo Summit 2014: Accumulo on YARN
Accumulo Summit 2014: Accumulo on YARNAccumulo Summit 2014: Accumulo on YARN
Accumulo Summit 2014: Accumulo on YARN
 
October 2014 HUG : Apache Slider
October 2014 HUG : Apache SliderOctober 2014 HUG : Apache Slider
October 2014 HUG : Apache Slider
 
Developing YARN Applications - Integrating natively to YARN July 24 2014
Developing YARN Applications - Integrating natively to YARN July 24 2014Developing YARN Applications - Integrating natively to YARN July 24 2014
Developing YARN Applications - Integrating natively to YARN July 24 2014
 
Authoring and Hosting Applications on YARN using Slider
Authoring and Hosting Applications on YARN using SliderAuthoring and Hosting Applications on YARN using Slider
Authoring and Hosting Applications on YARN using Slider
 
One Click Hadoop Clusters - Anywhere (Using Docker)
One Click Hadoop Clusters - Anywhere (Using Docker)One Click Hadoop Clusters - Anywhere (Using Docker)
One Click Hadoop Clusters - Anywhere (Using Docker)
 
Apache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARNApache Ambari: Managing Hadoop and YARN
Apache Ambari: Managing Hadoop and YARN
 
Bring your Service to YARN
Bring your Service to YARNBring your Service to YARN
Bring your Service to YARN
 
Pivotal cf for_devops_mkim_20141209
Pivotal cf for_devops_mkim_20141209Pivotal cf for_devops_mkim_20141209
Pivotal cf for_devops_mkim_20141209
 
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and FutureApache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
 
Docker based Hadoop provisioning - anywhere
Docker based Hadoop provisioning - anywhere Docker based Hadoop provisioning - anywhere
Docker based Hadoop provisioning - anywhere
 
Slider: Applications on YARN
Slider: Applications on YARNSlider: Applications on YARN
Slider: Applications on YARN
 
Dataworks Berlin Summit 18' - Apache hadoop YARN State Of The Union
Dataworks Berlin Summit 18' - Apache hadoop YARN State Of The UnionDataworks Berlin Summit 18' - Apache hadoop YARN State Of The Union
Dataworks Berlin Summit 18' - Apache hadoop YARN State Of The Union
 
Apache Hadoop YARN: state of the union
Apache Hadoop YARN: state of the unionApache Hadoop YARN: state of the union
Apache Hadoop YARN: state of the union
 
Apache Hadoop YARN: best practices
Apache Hadoop YARN: best practicesApache Hadoop YARN: best practices
Apache Hadoop YARN: best practices
 
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
 
Introduction to the Hortonworks YARN Ready Program
Introduction to the Hortonworks YARN Ready ProgramIntroduction to the Hortonworks YARN Ready Program
Introduction to the Hortonworks YARN Ready Program
 
Get most out of Spark on YARN
Get most out of Spark on YARNGet most out of Spark on YARN
Get most out of Spark on YARN
 
Running Non-MapReduce Big Data Applications on Apache Hadoop
Running Non-MapReduce Big Data Applications on Apache HadoopRunning Non-MapReduce Big Data Applications on Apache Hadoop
Running Non-MapReduce Big Data Applications on Apache Hadoop
 
Internet of things Crash Course Workshop
Internet of things Crash Course WorkshopInternet of things Crash Course Workshop
Internet of things Crash Course Workshop
 

Mais de Shivaji Dutta

Mais de Shivaji Dutta (8)

Life in lock down - A Data Driven Story
Life in lock down - A Data Driven StoryLife in lock down - A Data Driven Story
Life in lock down - A Data Driven Story
 
Deep learning an Introduction with Competitive Landscape
Deep learning an Introduction with Competitive LandscapeDeep learning an Introduction with Competitive Landscape
Deep learning an Introduction with Competitive Landscape
 
Aurius
AuriusAurius
Aurius
 
Deep Learning on Qubole Data Platform
Deep Learning on Qubole Data PlatformDeep Learning on Qubole Data Platform
Deep Learning on Qubole Data Platform
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystem
 
Hadoop security
Hadoop securityHadoop security
Hadoop security
 
Ambari blueprints-overview
Ambari blueprints-overviewAmbari blueprints-overview
Ambari blueprints-overview
 
Machine Learning With Spark
Machine Learning With SparkMachine Learning With Spark
Machine Learning With Spark
 

Último

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
 
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
VictoriaMetrics
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
masabamasaba
 
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 

Último (20)

BUS PASS MANGEMENT SYSTEM USING PHP.pptx
BUS PASS MANGEMENT SYSTEM USING PHP.pptxBUS PASS MANGEMENT SYSTEM USING PHP.pptx
BUS PASS MANGEMENT SYSTEM USING PHP.pptx
 
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
 
WSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security ProgramWSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security Program
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
 
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 

Apache Slider

  • 1. Page1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Slider Shivaji Dutta Sr. Partner Solutions Engineer
  • 2. Page2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Disclaimer This document may contain product features and technology directions that are under development or may be under development in the future. Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. This document’s description of these features and technology directions does not represent a contractual commitment from Hortonworks to deliver these features in any generally available product. Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
  • 3. Page3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Agenda • Apache Slider Overview • Yarn Overview • Why Slider • Slider Internals/Architecture • Slider App Packaging • Ambari and Slider • Q/A
  • 4. Page4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Overview
  • 5. Page5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Slider - Open Source in-incubation Project - http://slider.incubator.apache.org/index.html - Platform for - Deployment, Management & Monitoring - Long Running applications on a Hadoop/YARN Cluster - Built and Runs on Hadoop YARN Framework - It makes it EASY and SIMPLE
  • 6. Page6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN
  • 7. Page7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN as Cluster Operating System - Hadoop 2.0 - Resource Manager for Hadoop Cluster
  • 8. Page8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN • A global ResourceManager • A resource arbitrator for the cluster • A per application ApplicationMaster • A resource negotiator for the Application • Works with the Node Manager to Launch Application Containers • A per-node slave NodeManager • Manages Resources on a Node • a per-application Container running on a NodeManager • Actual application running in the container
  • 9. Page9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Flow NodeManager NodeManager NodeManager NodeManager Container 1.1 Container 2.4 ResourceManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager Container 1.2 Container 1.3 AM 1 Container 2.2 Container 2.1 Container 2.3 AM2 SchedulerClient2 Client1
  • 10. Page10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN - Powerful but Complex • Powerful – Fine grained control through API • Needs Coding and Development work for creating - Yarn Application Master - Yarn Client - Yarn Container - Complex & Time Consuming to write - For Standard Applications - No Easy way of State Management - THAW - FLEX
  • 11. Page11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Long Running Applications - Difference from Map-Red
  • 12. Page12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Long Running Application - Needs - Management - Install - Configure - Start/Stop - Reconfigure - Activate/Reactivate - Upgrade - Rolling Upgrade - Security - Scalability - Monitoring
  • 13. Page13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider
  • 14. Page14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Why Slider ? • Full YARN-integration takes effort • Code for every component and action • Powerful and finer control • YARN delivers access to all the data in HDFS –and the Cluster Resources • Maturing Hadoop stack needs an Agile platform to integrate • E.g. HBASE, HIVE, MAP REDUCE, APP Servers • Integrate to Management tools like - Ambari– to monitor applications in- a cluster
  • 15. Page15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider’s view of an Application Page 15 • An application is a set of components • A component is a daemon/launched exe – configuration – scripts, data files, etc. • Component may have one or more instances • Component instances are managed • Example – HBase Application (3 components) – HBase Master – HBase RegionServer – HBase REST service
  • 16. Page16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider – Design (On Yarn) Page 16 YARN Node Manager Component (container)AppMaster (container) YARN Node Manager HDFS Slider Agent Application Slider AppMaster Slider Client HDFS HDFS YARN Resource Manager
  • 17. Page17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Application by Slider Page 17 Slider App Package Slider CLI HDFS YARN Resource Manager “The RM” HDFS YARN Node Manager Agent Component HDFS YARN Node Manager Agent Component 1. CLI starts an instance of the AM 2. AM requests containers 3. Containers activate with an Agent 4. Agent gets application definition 5. Agent registers with AM 6. AM issues commands 7. Agent reports back, status, configuration, etc. 8. AM publishes endpoints, configurations Application Registry App Master/Agent Provider
  • 18. Page18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider AppMaster/Agent/Client Page 18 AppMaster  Common YARN interactions  Common *-client interactions  Publishing needs Agent  Configure and start  Re-configure and restart  Heartbeats & failure detection  Port allocations and publishing  Custom commands if any (e.g. graceful-stop) Client  App life cycle commands (flex, status, …)
  • 19. Page19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Terminology Apps on YARN • Application written to run directly on YARN • Packaging, deployment and lifecycle management are custom built for each application Slider Apps • Applications deployed and managed on YARN using Slider • Use of slider minimizes custom code for deployment + lifecycle management • Requires apps to follow Slider guidelines and packaging ("Sliderize")
  • 20. Page20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider – Getting Started
  • 21. Page21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Executing Slider • Install Apache Slider on to a Yarn Cluster • Create a “sliderized” Application Package • Setup the config files • Execute it from Slider client E.g. ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz - -template /work/appConf.json --resources /work/resources.json slider <ACTION> [<name>] [<OPTIONS>]
  • 22. Page22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Installing Slider • 3 easy steps • Download and build apache slider project • Install Slider Client that can access the Hadoop Cluster • Deploy the slider resources • Create the hdfs folders • Done! – Ready to rock! hdfs dfs -copyFromLocal ${slider-install-dir}/slider-0.40.0/agent/slider- agent.tar.gz /user/yarn/agent
  • 23. Page23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Commands Sample Slider commands • Build - Build an instance of the given name, with the specific options • Create – Build and run an instance • Destroy - Destroy a (stopped) applicaton instance • Exists - Probe the existence of the named Slider application instance • Flex - Flex the number of workers in an application instance to the new value • Freeze - freeze the application instance. The running application is stopped. Its settings are retained in HDFS. • Complete Man page http://slider.incubator.apache.org/docs/manpage.html
  • 24. Page24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider Application Packaging The main components • App Configuration • Configurations needed for the Application • appConfig.json • Resources • Resources required to run the application on the cluster • CPU, Memory, Priority • resources.json • Application Definition • MetaInfo.xml • Application jar file • Actual binary file
  • 25. Page25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved © Hortonworks Inc. 2014 Memcached on YARN Sample Slider App Page 25
  • 26. Page26 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Other Application Packages Page 26 Reference doc for Memcached Application • http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_app.html Slider github repo has other app  Accumulo  HBase  Storm  Memcached-windows
  • 27. Page27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Next?
  • 28. Page28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Its get Better Ambari Views for Slider • Ambari View that manages the life cycle of “Slider”ized apps
  • 29. Page29 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Thank You

Notas do Editor

  1. Apache Slider It is an open source project. Deployment, Management and Monitoring Distributed Application on a Apache YARN Cluster
  2. What YARN Does YARN enhances the power of a Hadoop compute cluster in the following ways: Scalability The processing power in data centers continues to grow quickly. Because YARN ResourceManager focuses exclusively on scheduling, it can manage those larger clusters much more easily. Compatibility with MapReduce Existing MapReduce applications and users can run on top of YARN without disruption to their existing processes. Improved cluster utilization. The ResourceManager is a pure scheduler that optimizes cluster utilization according to criteria such as capacity guarantees, fairness, and SLAs. Also, unlike before, there are no named map and reduce slots, which helps to better utilize cluster resources. Support for workloads other than MapReduce Additional programming models such as graph processing and iterative modeling are now possible for data processing. These added models allow enterprises to realize near real-time processing and increased ROI on their Hadoop investments. Agility With MapReduce becoming a user-land library, it can evolve independently of the underlying resource manager layer and in a much more agile manner.
  3. Servers run YARN Node Managers NM's heartbeat to Resource Manager RM schedules work over cluster RM allocates containers to apps NMs start containers NMs report container health