Many presentations on Microservices offer a high-level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some considerable DevOps work and start learning about deployment pipelines, metrics, and logging. Don’t panic. In this presentation we’ll discuss what we learned over the past four years by highlighting our mistakes. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll see how team communication is more important than one might realize. Most importantly, we’ll show how - while an individual service is simple - the infrastructure demands are now much more complicated: your organization will need to introduce and become increasingly dependent on various technologies, procedures, and tools - ranging from the ELK stack to Grafana to Kubernetes. Lastly, you’ll come away with the understanding that your resident SREs will become the most valued members of your team.

1. Surviving In A Microservices
Environment
Steve Pember
DevOps Days, Boston 2018
@svpember

2. @svpember
Microservice Blog Posts & Presentations
• Microservices: Yay!
• Microservices: Boo!
• Smashing The Monolith (and here’s how we did it)
• Microservices + Technology X
• Microservices + Methodology Y

3. Microservices have many lessons to offer

4. What even are Microservices?

6. @svpember
Why Choose Microservices?
• Reduce Coupling!
• Right Tool for the Job
• Continuous Delivery
• Smaller codebases are easier to reason about
• Easy to replace old services
• Efficient Scaling
• Can move quickly (once you’re up and running)

9. Help?

10. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication
• Miscellaneous Advice

11. @svpember
Microservice Topics
• Infrastructure

12. Writing a Microservice is EASY.
Operating a Microservices
platform is HARD

13. @svpember
Infrastructure
• How do we manage the logs?

14. Centralized Logs are your
#1 Priority

15. @svpember

16. @svpember

17. @svpember

18. @svpember

19. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?

20. You’d best be monitoring your
platform.

22. –Johnny Appleseed
“Type a quote here.”

24. Your Infrastructure Needs to Scale
Along With Your Application

25. Your Infrastructure Needs to Scale
Along With Your Application

26. @svpember

27. –Johnny Appleseed
“Type a quote here.”

28. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?

29. @svpember

30. @svpember

31. @svpember

32. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?
• How do we deploy our code?

34. @svpember

35. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?
• How do we deploy our code?
• Do we have any coding conventions?

36. Check Style? Coverage?

37. @svpember

38. @svpember

39. @svpember

40. @svpember

41. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?
• How do we deploy our code?
• Do we have any coding conventions?
• Can I generate a Service template?

42. wget https://github.com/klaviyo/base-template/archive/
release.zip
(This is not a real url, but you get the idea)

43. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?
• How do we deploy our code?
• Do we have any coding conventions?
• Can I generate a Service template?
• How do we share code?

44. Whoops! Gotcha!

46. @svpember
Sharing Code
• It’s OK to reimplement functionality within each service
• Setup your own internal Artifactory!
• DO share infrastructure libraries (e.g. communications)
• NEVER share business logic
• NEVER share Domain objects, Models

47. @svpember
Infrastructure
• How do we manage the logs?
• How about metrics / telemetry?
• How/where are our builds done?
• How do we deploy our code?
• Do we have any coding conventions?
• Can I generate a Service template?
• How do we share code?
• How do we manage our (multiple) environments?

48. @svpember

49. @svpember

50. Test & Develop in Isolation

51. @svpember

52. @svpember

53. @svpember

54. @svpember

55. @svpember

56. @svpember
Fail

57. @svpember
Fail

58. @svpember
Success

59. @svpember
Success

60. @svpember
Microservice Topics
• Infrastructure
• Architecture

61. @svpember
Architecture
• Do we have an overall design vision?

62. Does Anyone Know What We’re
Doing?!

65. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?

66. @svpember

67. @svpember

68. Be Careful When Choosing
New Tech

69. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?

70. Integration Tests are the Best Tests

71. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?

72. @svpember
Test Environment
• Run periodically (e.g. nightly)
• Each service generates fixture data
• Service data reset after EACH test

73. Consider NOT testing platform
Instead, monitor error rates

74. @svpember

75. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?

76. HTTP vs Async Events

77. Ensure you have Circuit Breaker or a
Dead Letter Mechanism

78. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?
• Do we follow an overall architectural style?

79. Make Sure Everyone Is Aware

80. Applies to Both Intra- and Inter-
Service

81. Everyone
needs to
read =>

82. @svpember
Architecture
• Do we have an overall design vision?
• What technologies do we use?
• How much freedom do we have in choosing new technologies?
• How do we test an individual service?
• How do we test the platform as a whole?
• How do our services communicate?
• Do we follow an overall architectural style?
• When do we create new services?

83. @svpember
New Service?
• Initially: design platform around Bounded Contexts, create services from inner
Contexts
• Don’t create services ‘just because’
• Don’t create services that are basically CRUD-wrappers
• Do make an effort to identify boundaries
• Ensure a service has/will have proper context boundaries before creating
• If two services need to communicate synchronously or frequently, good
candidates for MERGING

84. # of services < # of developers

85. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication

87. Conway’s Law is REAL

88. @svpember
Microservice Topics
• Infrastructure
• Architecture
• Team Communication
• Miscellaneous Advice

89. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services

90. Any idea
what these
do?

91. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see

92. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready, but don’t be afraid of bugs

93. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready, but don’t be afraid of bugs
• If a service A has another service B as a dependency => release B first

94. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready, but don’t be afraid of bugs
• If a service A has another service B as a dependency => release B first
• How to bootstrap a new service?

95. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready, but don’t be afraid of bugs
• If a service A has another service B as a dependency => release B first
• How to bootstrap a new service?
• API or Message Versioning is just.. blech.. a Big Thing

96. @svpember
Miscellaneous Advice
• Don’t get cute with the naming of services
• New Feature -> walk backwards from what the user will see
• Release when a feature is ready, but don’t be afraid of bugs
• If a service A has another service B as a dependency => release B first
• How to bootstrap a new service?
• API or Message Versioning is just.. blech.. a Big Thing
• Don’t release Friday afternoons

97. In Summary

98. Most of the Challenge is In
infrastructure

99. <3
Microservices

100. Please share your survival tips!
@svpember

101. Thank you!
@svpember

102. @svpember
Images
• Bad Odds (Jon Snow): https://www.reddit.com/r/gameofthrones/comments/4owm95/s6e9_megathread_gifwallpaperscreenshot_requests/
• Say Anything / John Cusak: http://www.glamour.com/story/happy-birthday-john-cusack-and
• Wilderness Survival Guide: https://www.kobo.com/us/en/ebook/the-wilderness-survival-guide-1
• XKCD: Code Review: https://xkcd.com/1513/
• Monolithic vs Microservices: @alvaro_sanchez
• Picard FacePalm: https://www.flickr.com/photos/30418788@N03/8381426895
• Council of Ricks: https://www.reddit.com/r/rickandmorty/comments/3exy00/it_looks_like_there_might_some_sort_of_council_of/
• Drago (break you): https://giphy.com/gifs/GWD5nSpiHxs3K
• Wrecking Ball: https://www.flickr.com/photos/rhysasplundh/5202454842/in/photostream/
• Rocky Stairs: http://pixgood.com/rocky-stairs.html
• Technical Difficulties: https://www.youtube.com/watch?v=EC15BmzsdhM
• Grafana example: https://i.imgur.com/KOqcD6L.png
• Zipkin example: https://blog.buoyant.io/2016/05/17/distributed-tracing-for-polyglot-microservices/
• Assembly Line: http://www.solidsmack.com/culture/humans-need-apply-new-short-film-explores-future-robots-manufacturing-automation/
• Site Reliability Engineer Shirt: https://www.sunfrog.com/102539321-160396457.html
• Watson, “Wait”: https://giphy.com/gifs/B8qMV3f7PH7nq
•

103. @svpember
Links
• Eric Evans: DDD and Microservices: https://www.youtube.com/watch?
v=yPvef9R3k-M
•