Breaking down your monolith into microservices — or starting fresh with a microservice architecture on a new project — is only half the battle. The next step is making your containerized microservice application fault tolerant and highly available, but how do you get there? How do you go from your local development machine to a production-grade cluster? See how container orchestration platforms like Kubernetes and Swarm can be essential assets for your production workloads, and learn how to replace bottlenecks and single points of failure with highly available, scalable microservice deployments.

1. Laura Frank
Director of Engineering, Codeship
Scalable and Available Services
with $CONTAINER_TOOL

3. R A I N I N G O N YO U R PA R A D E
Highly-available applications
existed before containers

4. We love to think we’re solving new
problems in new ways

5. We shouldn’t confuse new
tools with new problems

7. Container tooling
has changed the way we design,
build, run, and ship applications.

8. is a new solution for a
longstanding problem.
Container tooling

9. Containers aren’t the point
We reason about services

11. Before the late 1980s

12. 1990s-ish

13. 3:00am when you’re on call

14. How can we guarantee
availability in an environment
that will definitely fail?

15. DISTRIBUTED APPLICATIONS ENGINEERING, 1998
“Redundancy and recovery are
the two main approaches to
solve this problem.”

16. An Imprecise Guideline
ignoring many system constraints
redundancyrequired
(numberofreplicas)
time to recover from failure (generic time units)

17. Container tools have some pretty sweet ways
to deal with both redundancy and recovery.

18. Recovery
Control Theory FTW

19. Your orchestration platform is
continuously trying to reconcile
actual state with declared state.

20. Desired
State
-
ClusterOrch
actions to converge state
Actual State
at time T

21. An Observability Problem
If a system can’t be observed, it
can’t be controlled.

22. An Observability Problem
Failure Process State User Input

23. Desired
State
-
ClusterMe!
Actual State
at time T

24. An Observability Problem
Offloading the responsibility of
observability to an orchestrator improves
the level of controllability in your system

25. Atomic Scheduling Units
Scheduler
Orchestrator
taskN
task0
task1
Service Spec
desired state
Service Object
actual state

26. Kubernetes Master
Desired State
Scheduler Controllers
API Server
task0
task1
etcd

27. Kubernetes Master
Desired State
etcd
converged!
Scheduler Controllers
API Server

28. Using an orchestration tool, your
system never fails…
it just doesn’t converge

29. Redundancy
Replicating and scheduling for high availability

30. HA
application
problems
scheduling
problems
task scheduling problems

31. binpack

32. binpack

33. spread

34. spread (optimized for HA apps)

35. Most modern orchestration systems use
an optimized scheduling algorithm for
dispatching services across a set of nodes.
G R E AT N E W S

36. It is not your tool’s responsibility to know about your
system and business constraints
• topology* (some schedulers are topology aware)
• specifics like OS, kernel, instance family
• PII and other compliance
YO U S T I L L H AV E TO D O W O R K

37. These tools work on the service
level, not the infrastructure level
R E M I N D E R

38. Scheduling Constraints
Restrict services to specific nodes, such as
specific architectures, security levels, or types,
first apply a label to the nodes
docker service create
--constraint 'node.labels.type==web' my-app
in Docker

39. nodeSelector has been around since 1.0, but there
are alternatives which are more expressive
nodeAffinity has been around since 1.2 (still in beta).
nodeAntiAffinity does the opposite — you can repel
things from one another.
in Kubernetes
Scheduling Constraints

40. requiredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: some-node-label-key
operator: Exists
in Kubernetes
Scheduling Constraints

41. requiredDuringSchedulingIgnoredDuringExecution
in Kubernetes
Scheduling Constraints
requiredDuringSchedulingRequiredDuringExecution
This allows labels to change while the pod is
running and won’t result in eviction

42. Implements a spread strategy over nodes that belong to a
certain category.
This is a “soft” preference
--placement-pref ‘spread=node.labels.key’
in Docker
Placement Preferences

43. preferredDuringSchedulingIgnoredDuringExecution
in Kubernetes
Placement Preferences

44. Topology-aware Scheduling
us-east-1
us-east-2
us-east-1
us-west-1

45. Topology-aware Scheduling
us-east-1
us-east-2
us-east-1
us-west-1

46. Topology-aware Scheduling
Kubernetes has a topology-aware scheduler! Read the docs.
In Docker, apply labels to your nodes, and use a placement
preference like:
--placement-pref ‘spread=node.labels.region’

47. An Imprecise Guideline
ignoring most constraints
redundancyrequired
(numberofreplicas)
time to recover from failure (hypothetical time units)

48. The Future of Orchestration
Warning: opinions

49. A Framework for Evaluation
Genesis Custom Built Product Commodity
Visible (Lots of Management) Invisible (No Management)

50. Genesis Custom Built Product Commodity
Wardley Maps (simplified)
Time
InvisibleVisible

51. Genesis Custom Built Product Commodity
InvisibleVisible
Electricity
18th Century
Electricity
19th Century
Electricity
now

52. Genesis Custom Built Product Commodity
Electricity
Compute
InvisibleVisible

53. Genesis Custom Built Product Commodity
Container Runtime
2000s Container Runtime
2014-2015
Container Runtime
now
InvisibleVisible

54. Genesis Custom Built Product Commodity
Container
Orchestrator
Container Runtime
InvisibleVisible

55. Genesis Custom Built Product Commodity
Container
Orchestrator
Container Runtime
InvisibleVisible
?
?
?

56. Orchestration is
becoming commoditized.
Orchestrators will not be
able to differentiate easily.

57. C O M M O D I T I Z AT I O N
If you have a hand-rolled solution for
running apps with containers, it’s safe
to migrate to an orchestration platform.

58. I N N OVAT I O N
Solutions to old problems get
commoditized, but it leaves room
for genesis elsewhere

59. Genesis Custom Built Product Commodity
Container
Orchestrator
Container Runtime
InvisibleVisible
?
?
?
Istio & service mesh tools
Whatever Heptio is building
Storage!

60. Closing Thoughts

61. How can we guarantee
availability in an environment
that will definitely fail?

62. DISTRIBUTED APPLICATIONS ENGINEERING, 1998
“Redundancy and recovery are
the two main approaches to
solve this problem.”
Google became a company in 1998!

63. Laura Frank
Director of Engineering, Codeship
@rhein_wein
Thanks!