Mike Guthrie's presentation on distributed monitoring solutions for Nagios. The presentation was given during the Nagios World Conference North America held Sept 27-29th, 2011 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. Distributed Monitoring with Nagios: Past, Present, Future Mike Guthrie [email_address]

4. Large installations that exceed the CPU and memory resources that a single machine can offer.

8. In 1 second: About 16 scripts or binary processes are being launched, with about 16 sets of results coming in and being processed by Nagios and written to disk.

9. When the check schedule exceeds CPU limitations, you get “check latency”

12. A 30mn task on 1 server = 5 hours on 10 servers.

13. Consider how to effectively view information across multiple machines

14. As data quantity increases, discerning useful information from it becomes more important

15. Viewing 10,000 hosts and 50,000 services on a page is too much raw data to be effective information

16. The Classic Distributed Model Central Server (Passive Only) Active Checks Distributed servers running active checks, forwarding results to a central server Active Checks Active Checks Active Checks Active Checks Active Checks Active Checks Active Checks Forward Results After Every Check

17. The Classic Distributed Model

19. OSCP runs after every check

21. Child servers can also do local monitoring without forwarding results

22. Distributed servers can also receive passive checks and forward them along, creating a multi-level tree structure

24. All built into the Nagios Core package

25. Extremely flexible for checks, performance graphing, notifications, etc.

27. Which server issued the check?

28. Where to process/view performance data?

33. Unconfigured Objects

35. Works like a load balancer

38. The Cluster Model – DNX

40. Worker node executes the check, and send results directly to results queue

41. Checks are not associated with any particular worker node

42. Bypasses the nagios.cmd pipe to eliminate a potential bottleneck

43. If a worker goes down, all checks continue

45. Checks redistributed if a worker is down

47. If the master goes down, all checks cease

48. The Cluster Model – Mod Gearman

51. If the master goes down, all checks cease

52. Effectively viewing more than 10k+ services on a single machine

54. Central viewer unifies all servers

55. Central viewer polls data from each server and displays tactical data in the UI

57. MNTOS

58. check_MK Multisite

59. The Central Dashboard Model

61. Monitoring and object configurations compartmentalized to each server

62. Good for geographically distributed servers where local management is required

63. Unified login for all XI servers (basic auth still required for Core machines)

65. User-level control of server views

66. High level overview

67. Very little CPU usage

69. Free 60 day trial, requires a license

70. The Central Dashboard Model: Nagios Fusion

71. The Central Dashboard Model: MNTOS

72. The Central Dashboard Model: Multisite

74. Offload performance data processing to a different machine

75. Mount disk i intensive files to a RAM disk

76. A Nagios Core installs can run between 10 - 20k checks depending on what is being checked and how it is configured

78. Aggregated reporting and statistics

79. Business process views and monitoring

80. What do you, as admins, need to see in this area of software development?

82. Any of these models can be mixed and combined

84. An environment that implemented 250k services being overseen by a single server took almost an entire year of planning and implementation to do it right

85. Environments can scale even larger with the right logistics planning in place

87. Dave Williams: Author of DNX