SpringOne Platform 2016 Speaker: Michael Klishin; Software Engineer, Pivotal. Designing a system in terms of [micro] services is hype du jour but it's not without trade-offs. Debugging a distributed system can be challenging. In this talk we will cover how one can troubleshoot a distributed service-oriented system, focusing on Java, Spring, and RabbitMQ. Key points of the talk: -Multi-service system stability depends on how applications are designed as much as middleware -Multi-service systems are harder to debug -Several known tools and techniques exist to help: the audience will learn about them -Some problems are well understood and have known solutions: we will cover a few -Some tools in this spaces are underdeveloped or lacking "Structured platforms" help with troubleshooting and visibility, and affect how applications can/should be developed

1. Troubleshooting
RabbitMQ
and services that use it

2. Who am I?
• Staff Engineer, RabbitMQ @ Pivotal

3. Who am I?
• Staff Engineer, RabbitMQ @ Pivotal
• @michaelklishin, github.com/michaelklishin

5. The monolith problem

21. Troubleshooting publishers

22. Troubleshooting publishers
• I/O exceptions (shutdown handlers)

23. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms

25. When in doubt, borrow
ideas from TCP

26. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers

27. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers
• Invalid payload (e.g. fails to deserialize or
decrypt)

28. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers
• Invalid payload (e.g. fails to deserialize or
decrypt)
• Identifying publisher instances

29. Troubleshooting publishers
• identifying blocked (throttled) publishers

30. Client-provided
connection names in
RabbitMQ 3.6.3+

31. Troubleshooting publishers
• identifying blocked (throttled) publishers
• retries

32. Troubleshooting publishers
• spring-amqp can cover all of the above

35. Troubleshooting consumers

36. Troubleshooting consumers
• I/O exceptions

37. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS

38. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming

40. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming

41. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming
• Redelivery metrics

42. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming
• Redelivery metrics
• Identifying consumer instances

43. Troubleshooting consumers
• Consumer utilization (reported by HTTP API)

44. Troubleshooting consumers
• spring-amqp can help with some of the above

46. — W. Edwards Deming
“In God we trust, all others must bring data…”

47. — W. Edwards Deming
“In God we trust, all others must bring data…”

48. — What do you do for
a living?

49. — What do you do for a living?
— Tell people to read the logs.

50. Sources of data useful for
debugging

51. Sources of data useful for
debugging
• Metrics

52. Sources of data useful for
debugging
• Metrics
• Your logs

53. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs

54. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs
• Tracing data

55. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs
• Tracing data
• Wireshark (tcpdump, libpcap)

58. Collecting data from
RabbitMQ

59. Collecting data from
RabbitMQ
• Logs

60. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status

61. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment

62. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)

63. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)
• HTTP API (lots of metrics)

64. http://{hostname}:15672/api

65. curl -u guest:guest http://127.0.0.1:15672/api/overview | python -m json.tool
curl -u guest:guest http://127.0.0.1:15672/api/nodes/{node} | python -m json.tool
curl -u guest:guest http://127.0.0.1:15672/api/queues | python -m json.tool

66. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)
• HTTP API (lots of metrics)
• Message tracing ("firehose")

67. Collecting data from
RabbitMQ
• HTTP API (lots of metrics)
• Message tracing ("firehose")
• Infrastructure metrics

69. Common theme?

70. Common theme?
• Collect logs system-wide

71. Common theme?
• Collect logs system-wide
• Collect metrics system-wide

72. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide

73. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)

74. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)
• Analyze

76. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)
• Analyze
• Sounds like something a structured platform can help
with!

77. Distributed system
debugging is a problem
far from being solved.

79. Thank you

80. Thank you
• @michaelklishin

81. Thank you
• @michaelklishin
• github.com/michaelklishin

82. Thank you
• @michaelklishin
• github.com/michaelklishin
• mklishin@pivotal.io