My presentation from QConSF 2016 outlining some design strategies for building robust applications in the docker and container ecosystem. Video coming soon, I hope.

1. Architecting for Failure
in a Containerized World
Tom Faulhaber
Infolace

5. How can container tech help us
build robust systems?

6. Key takeaway: an architectural
toolkit for building robust
systems with containers

7. The Rules
Decomposition Orchestration and
Synchronization
Managing Stateful Apps

8. Simplicity

9. Simple means:
“Do one thing!”

10. The opposite of
simple is complex

11. Complexity exists
within
components

12. Complexity exists
between
components

13. Example: a counter
Counter
Service
1 2 3 4 50 …
Counter
Service
1 2 3 4 50 …x Counter
Service
1 2 3 4 50
1 2 3 4 50 1 2 3 4 50

14. Example: a counter
Counter
Service
1 2 3 4 50 …
Counter
Service
1 2 3 4 50 …
Load
Balancer
1 2 3 4 50 1 2 3 4 50

15. State + composition =
complexity

16. Part 1:
Decomposition

17. Rule:
Decompose vertically

18. App Server
Service
#1
Service
#2
Service
#3

19. App Server

20. Rule:
Separation of concerns

21. Example: Logging
App
Core
Code
Logging
Driver
Config
Logging Server

22. Example: Logging
Logger
App
Core
Code
Logging
Driver
Config
Logging Server
StdOut

23. Aspect-oriented programming

24. Rule:
Constrain state

25. Relational DB
Session Store

26. Rule:
Battle-tested tools

27. Redis
MySQL

28. Rule:
High code churn
→Easy restart

29. Rule:
No start-up order!

30. time
a
b
c
d

31. time
x
a
b
c
d

32. time
x
a
b
c
d
x
x
x

33. time
x
a
b
c
d
x
x
x

34. time
a
b
c
d

35. time
a
b
c
d

36. time
a
b
c
d

37. Rule:
Consider higher-order failure

38. The Rules
Decomposition
Decompose vertically
Separation of concerns
Constrain state
Battle-tested tools
High code churn, easy
restart
No start-up order!
Consider higher-order
failure
Orchestration and
Synchronization
Managing Stateful Apps

39. Part 2:
Orchestration and
Synchronization

40. Rule:
Use Framework Restarts

41. • Mesos: Marathon always restarts
• Kubernetes: RestartPolicy=Always
• Docker: Swarm always restarts

42. Rule:
Create your own framework

43. Mesos
Agent
Framework
Executor
Mesos
Master
Framework
Driver
Mesos
Agent
Framework
Executor
Mesos
Agent
Framework
Executor

44. Rule:
Use
Synchronized State

45. Synchronized State
Tools:
- zookeeper
- etcd
- consul
Patterns:
- leader election
- shared counters
- peer awareness
- work partitioning

46. Rule:
Minimize
Synchronized State

47. Even battle-tested state management is a headache.
(Source: http://blog.cloudera.com/blog/2014/03/zookeeper-resilience-at-pinterest/)

48. The Rules
Decomposition
Decompose vertically
Separation of concerns
Constrain state
Battle-tested tools
High code churn, easy
restart
No start-up order!
Consider higher-order
failure
Orchestration and
Synchronization
Use framework restarts
Create your own framework
Use synchronized state
Minimize synchronized state
Managing Stateful Apps

49. Part 3:
Managing Stateful Apps

50. Rule (repeat!):
Always use battle-tested tools!
(State is the weak point)

51. Rule:
Choose the DB architecture

52. Option 1: External DB
Execution cluster
Database cluster

53. Option 1: External DB
Pros
• Somebody else’s problem!
• Can use a DB designed for
clustering directly
• Can use DB as a service
Cons
• Not really somebody else’s
problem!
• Higher latency/no reference
locality
• Can’t leverage orchestration,
etc.

54. Option 2: Run on Raw HW
HDFS
Mesos
Marathon
App
HDFS
Mesos
Marathon
App
HDFS
Mesos
Marathon
App

55. Option 2: Run on Raw HW
Pros
• Use existing recipes
• Have local data
• Manage a single cluster
Cons
• Orchestration doesn’t help with
failure
• Increased management
complexity

56. Option 3: In-memory DB
Mesos
Marathon
App
MemSQL
Mesos
Marathon
App
MemSQL
Mesos
Marathon
App
MemSQL

57. Option 3: In-memory DB
Pros
• No need for volume tracking
• Fast
• Have local data
• Manage a single cluster
Cons
• Bets all machines won’t go
down
• Bets on orchestration
framework

58. Option 4: Use Orchestration
Mesos
Marathon
App
Cassandra
Mesos
Marathon
App
Cassandra
Mesos
Marathon
App
Cassandra

59. Option 4: Use Orchestration
Pros
• Orchestration manages
volumes
• One model for all programs
• Have local data
• Single cluster
Cons
• Currently the least mature
• Not well supported by vendors

60. Option 5: Roll Your Own
Mesos
Marathon
App
ImageMgr
Mesos
Master
Framework
Mesos
Marathon
App
ImageMgr
Mesos
Marathon
App
ImageMgr

61. Option 5: Roll Your Own
Pros
• Very precise control
• You decide whether to use
containers
• Have local data
• Can be system aware
Cons
• You’re on your own!
• Wedded to a single
orchestration platform
• Not battle tested

62. Rule:
Have replication

63. The Rules
Decomposition
Decompose vertically
Separation of concerns
Constrain state
Battle-tested tools
High code churn, easy
restart
No start-up order!
Consider higher-order
failure
Orchestration and
Synchronization
Use framework restarts
Create your own framework
Use synchronized state
Minimize synchronized state
Managing Stateful Apps
Battle-tested tools
Choose the DB architecture
Have replication

64. Fin

65. References
• Rich Hickey:
“Are We There Yet?” (https://www.infoq.com/presentations/Are-We-
There-Yet-Rich-Hickey)
“Simple Made Easy” (https://www.infoq.com/presentations/Simple-
Made-Easy-QCon-London-2012)
• David Greenberg, Building Applications on Mesos, O’Reilly, 2016
• Joe Johnston, et al., Docker in Production: Lessons from the
Trenches, Bleeding Edge Press, 2015

66. The Rules
Decomposition
Decompose vertically
Separation of concerns
Constrain state
Battle-tested tools
High code churn, easy
restart
No start-up order!
Consider higher-order
failure
Orchestration and
Synchronization
Use framework restarts
Create your own framework
Use synchronized state
Minimize synchronized state
Managing Stateful Apps
Battle-tested tools
Choose the DB architecture
Have replication