1.
1
MONITORING AT
SERVICE PROVIDER
SCALE
June 23, 2014

2.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
INTRODUCTIONS
CHRIS JACKSON
Rackspace CTO (DevOps Practice)
chris.jackson@rackspace.co.uk
@chriswiggy
+44 (0) 7712 854462

3.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
3
Image courtesy of http://www.toastmasters.krakow.pl/

4.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
4
THE DEVOPS RACK APPLIANCE…
ALL OF THE THINGS IN 3-CLICKS!
Image courtesy of Axiomtek

5.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
5
HOW DO WE APPLY DEVOPS
TO SCALE SERVICE?

6.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
THE DEVOPS PRACTICE AT RACKSPACE
6
THE FIRST WAY – CREATE A DELIVERY SYSTEM
THE SECOND WAY – AMPLIFY FEEDBACK LOOPS
THE THIRD WAY – LEARN & EXPERIMENT CONTINUOUSLY
This is PRODUCT & SUPPORT delivering great service powered by automation
!
Our DevOps practice is a way to accelerate feedback by removing complexity
!
The goal is to never hold our customers up, we must always be faster…
!

7.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALING TRANSFORMATION
7
SCALING TRANSFORMATIONS
Image courtesy of Wordpress

8.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
8
SO WHAT ABOUT MONITORING?

9.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
9
WHY #MONITORINGSUCKS
FOR RACKSPACE
We generate over 40,000 events per second on just our Cloud Monitoring platform.
If a customer is Managed, we troubleshoot ANY alarm that is raised.

10.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
10
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of Spot On Lists

11.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of Cepolina

12.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
12
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of New Line Cinema

13.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
13
WHAT ARE WE DOING
TO FIX THIS?

14.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALE MONITORING TUNING
14
WHEN TUNING IS MANUAL:
EFFORT=tN
Where t = time to configure a single node and N = number of nodes
0 100 200 300 400 500
Effort
# Nodes

15.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALE MONITORING TUNING
15
WHEN TUNING IS INTELLIGENT:
EFFORT=t1-N
Where t = time to configure a single node and N = number of nodesEffort
# Nodes
Shrink this!

16.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
TOOLS FOR INTELLIGENT CONFIGURATION
16
Configuration Management
If you know what you want to measure!
If your constraint is just the number of nodes!
Gets complicated when you’re thresholding all of the things…!
Configuration Discovery
We call it Waldo, you know it as Sartori in Openstack!
Uses ohai-solo to find information about your configuration!
Compare to best practice and make changes accordingly…!
Image courtesy of Walker Books Ltd

17.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
MONITORING DATA ANALYTICS
17
EVENT GENERATION EVENT PROCESSING
EVENT HANDLING
EVENT ANALYTICS
TUNING INSIGHT
CREATE FEEDBACK LOOPS INSIDE MONITORING
REMOVE WORK BY TUNING ALARMS

18.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
INTELLIGENT ALERTS
18
ANOMALY MONITORING PATTERN RECOGNITION

19.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
WHAT ARE WE USING?
19
statsd

20.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
TOOLCHAIN INTEGRATION
20
EVENT GENERATION EVENT PROCESSING
EVENT HANDLING
EVENT ANALYTICS
TUNING INSIGHT
statsd

21.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HELP US TO DO THE HARD WORK
21
YOU
US
Things We Want to Know™!
How do you use monitoring?!
How intelligent is the tuning?!
What use of anomaly or threshold?!
Approach for creating tool chains?!
What do you dislike doing?!
We’ll build it and open
source it!!
Image courtesy of Dave Hogan/Getty Images

22.
Twitter - @chriswiggy
http://www.rackspace.com/devops
http://devops.rackspace.com
22
THANK YOU

23.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
MORE THAN A CLOUD COMPANY
WE PROVIDE
SUPPORTFOR EVERY STEP OF YOUR JOURNEY
JUST STARTING?
RACKSPACE ADVISORY
SERVICES
NEED SERVICES?
RACKSPACE HYBRID
CLOUD
NEED SUPPORT?
RACKSPACE DEVOPS
AUTOMATION SERVICE