“The Elements of Style” is one of the most important and foundational guidelines on how to write well. It has effectively summarized, in a list of seminal guidelines, how to harness the power of the English language to write high quality prose of almost any kind. In computing, we have similar guides for various technologies. Python offers “The Zen Of Python”, Ruby has “The Rails Doctrine”, and so on... One of the powers these documents wield is that they help serve as a “north star” that guides an entire community toward the same goals. I believe we need a similar guide for Kubernetes. It would describe how app developers and operators should think about and use the features in Kubernetes to build and deploy reliable, stable apps. Armed with such a guide, we could all hope to better understand the “essence” of Kubernetes in pursuit of building better cloud native apps. We don’t have anything like this today, but many in the Kubernetes community have strong, detailed opinions for what should go in this guide. Much of it is tribal knowledge or scattered in blog posts. In this talk, I’ll try to bring many of these opinions together and lay out an “Elements of Kubernetes” guide for app developers and operators alike. I’ll do so by relating each “element” to stories and details I’ve seen in the community that reveal what makes a good Kubernetes and cloud native app. This talk was given at KubeCon / CloudNativeCon 2017 on December 7th, 2017 in Austin, TX

1. The Elements of Kubernetes
Aaron Schlesinger
Microsoft Azure Containers
Lead, SIG-Service-Catalog

2. ★ Kubernetes is growing fast
★ Usage, development, projects
★ We’re in the wild west
Why we’re here

3. Building an app for Kubernetes
Dev/Test Containerize CI/CD
Staging Pre-prod Prod
ResilienceMonitoring Tracing Logging Observability

4. Lots to figure out
No one size fits all

5. We need a “north star” for people
building cloud-native apps

6. ★ Help app operators/developers decide what to do
★ Stand up to rapid technology changes
★ Guide SIGs
Best practices, not rules
North Star

7. What we have now
★ Opinions
★ Evidence
★ Fragmentation

8. I’ve seen the good & the bad

9. I’m here to propose ideas

10. Observability is golden

11. ★ Kubernetes schedules containers
★ Kubernetes observes containers
★ You observe containers
Observability is golden

12. ★ Resource limits
★ Readiness & liveness checks
★ HPA
Kubernetes observing your app

13. You observing your app
Adopt a rich ecosystem of cloud native tools:
★ Logging
★ Service mesh
★ Tracing

14. When all else fails, crash

15. Crash-only software
★ You’ll have bugs, network outages, disk issues, etc...
★ Kubernetes is your retry loop

16. What that might look like
★ Your app connects to a DB on startup
★ App fails to connect, crashes
★ Tracing sees the conn. failure
★ Kubernetes restarts it
★ Monitoring, log aggregation pick up the restart
★ Alerting notifies if too many restarts
★ ...

17. Unordered is better than
ordered

18. ★ Kubernetes & your app are distributed systems
★ Ordering is very hard in distributed systems
★ Try not to rely on ordering
Unordered is better than ordered

19. Use Kubernetes primitives.
★ Sidecar for locks, leader election
★ Resource versions in Kubernetes resources
★ Init containers (can be messy)
But sometimes you need it

20. Loose coupling is better than
tight coupling

21. Loose coupling is better than tight
★ Kubernetes is always watching
★ Your app should tolerate dynamism

22. What that might look like
★ Pod => Pod messaging via Services
★ Crash if you can’t connect (crash-only)
★ Look for Kube resources via labels, not names

23. … But tight coupling isn’t
always wrong

24. ★ Pods have >1 containers on purpose
★ Run tightly coupled containers in a pod
Tight coupling isn’t always wrong

25. ★ Envoy
★ Fluentd logging driver
★ Metrics
What that might look like

26. Record your configuration

27. Record your configuration
★ Kubernetes APIs are declarative
★ Keep the latest working configuration in your repository
★ Let Kubernetes reconcile

28. What might that look like

29. Ask for the least

30. Ask for the least
If you’re:
★ Configuring RBAC permissions
★ Configuring containers in a pod
★ Asking for CPU shares or memory
★ Asking for disk space with a PersistentVolumeClaim
Ask for the fewest possible of them. Leave more for Kubernetes

31. What that might look like
★ Read-only permissions for your monitoring system
★ One CPU share for each web frontend
★ Minimal disk for your log aggregator
★ Tiny memory for your local proxy

32. Build on the shoulders of
giants

33. Build on the shoulders of giants
★ Kubernetes provides a big API
○ … to abstract functionality that’s hard to get right
★ Maybe the community doesn’t do what you need
○ … but try to find the next best thing and build atop

34. What that might look like
Helm for managing your app lifecycle
★ Or the Helm API (which is gRPC)
Traefik for Ingress
★ Or Traefik => service mesh => your app
Fluentd for Logging
★ Or app => local translator => fluentd

35. Parting thoughts: where
should we go from here?

36. Who should define our guidelines?
★ I’ve started the conversation here
★ We all have a wealth of experience
★ We need to share it

38. We need your thoughts
https://github.com/arschles/kube-best-practices

39. Thank you
aaron.schlesinger@microsoft.com
@arschles
github.com/arschles

40. Community Extras

41. Pay it forward

42. Pay it forward
★ If you “build on the shoulders of giants,” open source it
★ You’ll be moving cloud native forward
★ We’ll progress as community & concept because of your work

43. Disagree constructively

44. ★ Cloud native is young
★ If you disagree with a choice, others do too
★ Make your voice heard, and offer solutions
Disagree constructively