During our SOA transition at Hootsuite, we have noticed that visibility into our service relationships, dependencies and status is paramount to keeping our team, our build pipeline and application running smoothly. I’d like to share with you an API we baked into our SOA architecture that enables us to explore our applications service dependency graph in real time.

1. Service Explosion
HOW HOOTSUITE MANAGES ITS
GROWING MICROSERVICE LANDSCAPE
Specialist Software Developer - Mobile Web and APIs
Adam Arsenault
@Adam_Arsenault

3. ● Road to SOA
● Service Graph
● Voltron
● Demo
What We’ll Talk about
App
S2
S3
S4
S1

4. ● Founded in 2008
● PHP Monolith
● SOA started in 2013
● Hyper growth
● Continuous Integration
Road to SOA

5. “Organizations which design
systems ... are constrained to
produce designs which are copies
of the communication structures
of these organizations”
- Melvin Conway

6. Engagement MobilePublisher Analytics
Platform Tools Labs
Stable Teams

7. KABOOM!!!!

8. Ex. 1 - Integration Failures
● Dev merges changes
● Changes go to staging
● Integration tests fail
● Release pipeline frozen
KABOOM!!
!!

9. Ex. 2 - Production Downtime
● Service goes down in production
● On call / teams affected get
notifications
● Sift through flood of notifications to
figure out what’s broken
KABOOM!!
!!

10. VISIBILITY

11. The Service Graph

12. The Service Graph
App

13. The Service Graph
App
S1 S2

14. The Service Graph
App
S2 S3 S4S1

15. The Service Graph
App
S2
S3
S4
S1

16. The Service Graph
App
S2
S3
S4
S1
S5
S6
S7

17. The Service Graph
App
S2
S3
S4
S1
S5
S6
S7

18. The Service Graph
App
S2
S3
S4
S1
S5
S6
S7

20. The API

21. Dependency
Something that a service needs to
function properly.
Types:
1. Internal
2. Traversable
App
Cachedb S1

22. /status/about
Returns metadata information about
the service or app such as version,
description, maintainers, links to
documentation, and gets the status
of each individual dependency.

23. /status/:dependency
A configured status endpoint at
'/status/:dependency
Examples:
● '/status/service-core'
● '/status/db'
OK

24. /status/aggregate
Returns the overall status by
checking all registered status checks
and giving a simple response.
Examples:
● OK
● CRIT - error message
OK

25. /status/traverse
Enables service graph traversal and
execution of an "action" at the last
level of traversal.
App
S2
S3
S4
S1

26. Usage

27. Monitor
OK / CRIT

28. Debug
S1
db S2

29. Explore and Learn
App
S2
S3
S4
S1
S5
S6
S7

30. Document

31. Status of single machine Overall status of application
and services
Alerts / notifications Troubleshoot by drilling down
Monitoring Strategy

33. Technologies

34. Architecture
Browser 1 Browser N...
Play App
Status Poller
Actor
WS
Actor 1
WS
Actor N
...

35. DEMO

36. “When there is a production issue, I see lots of
people go to Voltron to perform diagnostics on
what might be wrong”
Geordie Henderson - VP Software Development

37. “Voltron is often the first to tell us when
snowflake is down”
Brandon Okert - Junior Software Developer Publisher

38. “When a critical service goes down, everything
starts alerting and reporting problems, but
Voltron gets through the noise by letting you
drill down”
Michael Reid - Senior Software Developer Platform

39. “We suspected the connection between
dashboard and Billing Service was broken, but
Voltron told us the communication channel
was okay.”
Martin Jung - Software Developer Mobile Web and APIs

40. Lessons Learned

41. • Productivity
• Happiness
Visibility Empowers

42. • Automate all the things
• Identify problems early
and fix
• 10x factor
SOA Tools Early

43. • Standardize
• Add to a service framework
• Share common status checks
Make Checking Status Easy

44. • Synchronized views
• Performance
Websockets for Real time

45. Future Work

46. Analytics

47. Real Time Graph View

48. Open Source

49. Thank you!
Questions?
Specialist Software Developer - Mobile Web and APIs
@Adam_Arsenault
Adam Arsenault