Openshift/Kubernetes integration with Apache YARN Shows how YARN allows Big Data/Hadoop and other workloads to run side by side on same cluster

1. Page1 © Hortonworks Inc. 2014
OpenShift scheduling docker
containers in YARN via Kubernetes

2. Page2 © Hortonworks Inc. 2014
Static website Web frontendUser DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’
s laptop
Docker: a shipping container system for codeMultiplicityofStacks
Multiplicityof
hardware
environments
Production
Cluster
Customer
Data Center
Doservicesandapps
interact
appropriately?
CanImigrate
smoothlyandquickly
…that can be manipulated using
standard operations and run
consistently on virtually any
hardware platform
An engine that enables
any payload to be
encapsulated as a
lightweight, portable,
self-sufficient
container…

3. Page3 © Hortonworks Inc. 2014
Why are Docker containers lightweight?
HDP 2.1
Hortonworks Data Platform
I/O performance comparable to Bare Metal

4. Page4 © Hortonworks Inc. 2014
Kubernetes – container orchestrator
• Service for container cluster management
• Allows deploying and managing applications running
on multiple hosts using docker
• Open sourced by Google
• Supports GCE, CoreOS, Azure, vSphere
• Used to manage Docker containers as a default
implementation
• Master – maintain state of Kubernetes Server runtime
• Scheduler, API server, registries, storage
• Minions – represent the host were containers created
• Kubelet – manage pod and container lifecycle

5. Page5 © Hortonworks Inc. 2014
OpenShift
• Red Hat’s platform as a service for
applications in the cloud that supports
both public and private cloud
• Provides high level abstraction for applications on top
of containers allowing easy scaling, service discovery,
and deployment
• Enable Docker image authors to easily deliver
reusable application components, including highly
available databases, monitoring and log aggregation
tools, service discovery platforms, and prepackaged
web applications
• Allow developers to deeply customize their runtime
environments while preserving operational support at
scale for those applications

6. Page6 © Hortonworks Inc. 2014
6
Kubernetes/YARN Integration
YARN Node Manager
YARN Node Manager
Kubernetes
AppMaster

7. Page7 © Hortonworks Inc. 2014
Demo: OpenShift scheduling docker container in YARN
Running in docker

8. Page8 © Hortonworks Inc. 2014
Demo: 1. Launch Docker container on YARN via OpenShift
#openshift kube create pods –c demo.json
#cat demo.json

9. Page9 © Hortonworks Inc. 2014
Demo: 2. Enter into container and confirm web server started
#openshift kube create pods –c demo.json

10. Page10 © Hortonworks Inc. 2014
Demo: 3. Webapp on Openshift interacting with Hadoop

11. Page11 © Hortonworks Inc. 2014
Demo: 4. Launch other workloads on YARN
YARN allows Big Data and other workloads to run side by side on same cluster

12. Page12 © Hortonworks Inc. 2014
Questions?