presentation at Scale 16x covering op premise deployment of container orchestrators

1. Running Containerized in
an on-prem Datacenter

2. 2
presenter
• Steve Wong
former: Strategic Open Source Partner Engineer
{code} team, Dell Technologies
• next: to be announced soon
• @cantbewong

3. 3
Agenda
• What is a Container Orchestrator
• Why would you want an Orchestrator?
• Why on prem?
• Types of Orchestrators Compared
• Deployment Toolchain choices
• Networking
• Storage

4. 4
What is an orchestrator?
“Kubernetes is an open-source platform for automating deployment,
scaling, and operations of application containers across clusters of hosts,
providing container-centric infrastructure”

5. 5
What is an orchestrator?
Two viewpoints
From a resource perspective:
It’s an infrastructure abstraction layer:
– Aggregates systems into a single resource pool.
From an application perspective:
It simplifies management of distributed technologies
– A scheduler (= dispatcher).
– Dispatches workloads which consume the pooled resources.

6. 6
Typical Orchestrator Goals
• Modularity – interfaces and APIs documented,
open, replaceable
• Enable apps to be oblivious to hosting details
• Enable users/devs to write once, run in any cloud
or on-prem
• Enable operators to utilize distributed computing
resources without becoming an expert in
distributed computing

7. 7
Why would you want a container orchestrator?
Docker, Microservices and
container-based
development, with CI/CD
Seek Google-like
datacenter operations, off
the shelf, instead of home-
grown
Machine Learning, AI,
Streaming Data Analytics,
Fast Data, Big Data
Need app and service
portability across public clouds
+ on-prem... with consistent
staff skill set, and no cloud
provider lock-ins
Need self service IT user
experience, without using
Amazon, or using clouds
supplemented with on-
prem and edge capacity
Automated and Integrated
security, networking,
storage management,
logging, health monitoring

8. 8
App Store Experience: self hosted Data Services
ANALYTICS
STREAM INGEST
NOSQL
SEARCH
CACHE
Elastic MapReduce
Kinesis
DynamoDB
CloudSearch
ElastiCache
AWS-specific services
RELATIONAL DB RDS

9. 9
Why on prem?
You want a cloud like user experience
but
• You can’t afford latency (control
and notification or legacy
interaction) or reliability issues
• You can’t connect at all.
• Legal data governance issues
• On prem is more cost effective
• Cloud provider(s) are your
competitor
• Need for Data ingress
preprocessing for local control and
notification loops, data reduction

10. 10
Orchestrator class
analogy
A PaaS platform takes more
responsibility – often making
decisions for you
A container orchestrator is less
opinionated

11. 11
PaaS vs CaaS vs IaaS
Simple test:
If you can install a database on it, it
is not PaaS.
If the database is prepackaged in a
Docker container, it’s CaaS

12. 12
A continuum of choice

13. 13
common ancestors

14. 14
What’s your ideal datacenter?
deploy from open source / commercial distro
bare metal / Iaas / Caas / Paas / SaaS

15. 15

16. 16
Your choice can be AND not OR

17. 17
Public Cloud vs On Premise – What’s different?

18. 18
On prem differences - Network
Vs. Public Clouds (AWS, GCE, etc.)
• No standardized mandated network ingress, egress
service /with API
Vs Existing legacy network infrastructure
• Network needs are different

19. 19
Some distributions provide an ingress
management
Mesosphere DC/OS
OpenShift Routes
Pivotal PKS / Cloud Foundry
And standard Kubernetes is
commonly used in conjunction
with NGINX
forward a “wildcard” domain

20. 20
Network – What if I drop microservices on
existing legacy network infrastructure?
Common goals
• Fast
• Secure
• Low latency
• Never fail
Gaps - You take care of:
• service discovery
• App level flow control
• Security (beyond basic
connectivity)
• Network partitioning protection
• Policy
• Health service checks, metrics
And how’s the latency on that API?

21. 21
Storage… What does the 12 factors say?
Why?
Easy to replace, upgrade,
automate scale-up and scale-down

22. 22
everything is stateless? – how’s that going to work

23. 23
What if it isn’t “the other guys problem”?
Suppose you are that guy maintaining the
“backing store”
There are valid reasons to do this yourself
• You want to pick your own tool and version
• You want to customize
• You want to stay portable across clouds
• You want to avoid database monoliths

24. 24
“12 factor”
stateful backing services are not just SQL database

25. 25
Monolithic = bad

26. 26
Container advantages make sense for stateful
too
Container attributes:
• Consistent environment
– same anywhere
• Dependency
management -
packaging
Orchestration can add:
• Health monitoring
• Automated rollouts and
rollbacks
• Declarative configuration
• App/package store deploy
experience

27. 27
On prem differences – stateful container apps
Common
• File systems
• Block
volumes
Gaps - You take care of:
• Posix mountable storage
• Security (beyond basic
connectivity)
• Policy?
• Health service checks, metrics

28. 28
Background – container orchestrators
Popular container orchestrators have independently
evolved storage interfaces

29. 29
Background – storage providers
Selected open source and commercial vendors have
solutions – sometimes usable across orchestrator
platforms

30. 30
Variations of storage interface:
Is this good for the community?
Users
Container Orchestrators
Storage Providers

31. 31
CSI: Goals
The Container Storage Interface (CSI) is modeled on
the successful OCI and CNCF sponsored CNI
interoperability initiatives in the container and
network space respectively.
Its goal is to provide a vendor neutral, curated
specification that allows standardized storage
plugins to be published and utilized across multiple
container orchestrators, including Apache Mesos,
Cloud Foundry, DC/OS, Kubernetes.

32. 32
CSI: Overview
• Control plane interface
– CSI “steps aside” after wiring volume to container– not a
bottleneck in the data IO plane
– Flexible deployment
• Focus on volume lifecycle
– Create
– Publish/Unpublish (to nodes, to containers)
– Destroy
• Service-oriented
– Long running
– gRPC; CO is a client of plugin services

33. 33
Community: Who is involved with CSI

34. 34
CSI Roadmap: Beyond intro release
Considering these -
• Snapshot support
• Volume resizing
• Quota
• Windows OS/container support
• User ID & credential passthrough to storage
provider
This is deemed out of scope - up to orchestrator
platform to implement, differentiate
• Storage class (aka profiles)

35. 35
Choosing on prem storage - considerations
Existing support in orchestrators
Future or current CSI support
Interop – with Caas, Paas, legacy
Model
• Appliance
• Software defined on commodity hardware
– Deployment
› Independent
› Optional converged
› Fully Converged (container packaged)
– Runs in kernel vs user space

36. 36
On-prem consideration: Install technology
DC/OS (Apache Mesos distribution)
• Advanced installer – ssh access to Linux nodes
• Other variants available
Kubernetes
• Deploy from github
– kubeadm command line to Linux nodes
– minikube – not a production solution
• Certified Distributions
– Docker Swarm
– Mesosphere DC/OS
– RedHat OpenShift
– Pivotal / VMware PKS
– Typhoon (https://github.com/poseidon/typhoon)
– More (the CNCF maintains the official list, these differ
in more than install technology)

37. 37
Install technology differences
Ansible vs BOSH vs shell script, others
Variations in abstraction level
• Ansible, Chef, Puppet originally desgned to manage
software on existing servers
• Teraform, BOSH designed to manage clusters
Variations in distribution source
• download at deploy time
• use binary repository (pre-downloaded)
• build from source (pre-downloaded)
Servers are a “pet” vs “cattle”

38. 38
presenter
Steve Wong
twitter @cantbewong
this deck: