It's all about telemetry

It’s all about telemetry
Monitoring what matters in a useful way.

Tuesday, June 26, 12

Theo Schlossnagle @postwait

I write software
I write books
I give talks
I participate in the industry
I speak frankly about industry issues


Data, data, everywhere.

A billion pageviews / month.
100k database queries / second.
1MM memcache queries / second.
500k MQ messages / second.
10MM I/O operations / second.


Big Data

Most new big data problems are

solvable


Big Data

Most new big data problems are
created by our solutions, and thus
solvable
despite their ROI


That’s a whole lot of data

Think in terms of logs (too many do)
About 26 trillion log lines / month
@ 40 bytes compressed: 1PB / month
Just because it is possible
does not mean it will return on investment
(and does not mean it won’t)


It’s all “useful”; which data?

Think in terms of cost/beneﬁt.
Sure the data is useful, but it costs money to store
Does it cost you more to have it or not to have it?
Maybe the right approach is to keep that level of detail
for a few days?


Double-edged sword.

Eroding granularity over time
keeps storage under control


Double-edged sword.

K E
TA
Eroding granularity over time

S
keeps storage under control

M I

1 year
at a glance


1 week
looks normalish


1 day
conﬁdence of normalcy increases


1 week
that looks different


1 day
yup, that’s not at all like that other week


Other methods

What do you store?
How do you store it?
Why is it useful?
Winning the cost benefit game by
reducing costs more significantly than
reducing benefits


0 0.5 1 1.5 2 2.5 3

1

eﬁt
Ben
o st
C 0.75

0.5

0.25

monitoring activity ➠

Positive Value
Be in the green.


0 1 2 3 4 5 6 7 8 9 10
10

7.5

5

o st
C
2.5

Beneﬁt

There’s a bigger picture
It’s not as easy as you think.


0 0.5 1 1.5 2 2.5 3

1

eﬁt
Ben
o st
C 0.75

0.5

0.25


Value is difference, not area
Green can be misleading


0.5 1 1.5 2 2.5 3
0.5

0.25

-0.25

-0.5

-0.75

Value = Beneﬁt - Cost -1

Green means we have positive return


0.5 1 1.5 2 2.5 3
0.5

0.25

-0.25

-0.5

-0.75

It’s not about return -1

Well, it’s not only about return


0.5 1 1.5 2 2.5 3
0.5

0.25

-0.25

-0.5

-0.75

It’s about maximizing return -1

This is a bit like black magic


Technique 1: text

Store changes


Technique 2: numeric
Store rollups
(i.e. statistical aggregates over ﬁxed windows)
over 1 minute store
min/max/avg/stddev/covariance/50%/95%/99%
lots of information
heavy lossy compression of high-frequency data
loses population distribution information


Database replication
Lag (green) and rate of lag change (purple)


Storage Usage
We can see growth.
More useful, we can use this to project.

With simple numeric data


Unknowns can be predicted


In sane ways with conﬁdence


Full Disclosure

You see awesome examples of predictive analytics
Like the real-world one on the previous slide
In practice, almost all data streams predict one thing:
they have no fucking clue.


Technique 3: histograms

Store histograms
over 1 minute store
counts of datapoints seen in various buckets
retains complete population distribution
loss of precision


Histograms 101
This.
This is a histogram.
It shows the frequency of
values within a population.
Height represents frequency


Histograms 101
This.
Now, height and color
represents frequency


Histograms 101
This.
Now, only color


Histograms ➠ time series
This.
Now, only color


Histograms ➠ time series
This.
Now, only color

at a single time interval


API Service Times
We can see a full population shift
of several milliseconds

Combining techniques

In our system (as a reference point)
Arbitrary numbers of numeric data points
on a single stream
occupy 32 bytes of space for statistical aggregates and
occupy about 2k of space for a histogram
These means we can store these transforms on
numeric data in perpetuity


Combining techniques
Text is a bit harder
You need to be careful
Some data sources can be constantly changing
Producing gobs of change data
You’re doing it wrong
Find these and ﬁx them


Correlating Events
Change Management vs. Performance


What to monitor?

Most people don’t monitor the things that matter most


Monitor the Business

Financials:
Revenues. Costs. Margins. AR. Account delinquency.
Marketing:
Web analytics. Campaigns. Costs. Returns.
Convergence.


Monitor the Support

Customer Service:
Problems. Time investment. Customer satisfaction.
Resolution time.


Monitor the Engineering

Engineering:
Deployments. Test coverage.
Bug reports. Bug ﬁxes. Effort spent.
Operations:
Faults. Pages. Escalations. Provisioning time.
Equipment defect rates. 3rd party failure rates.


Monitor the Service
Systems:
Networks. Systems. Storage.
Databases:
Performance. Error rates. Backups.
Middleware:
Herein lies the magic and room for awesomeness


Monitor the Middleware

Your systems are complex
Monitor their interactions
Messaging, APIs, etc.


Monitor all the things.

But, perhaps most importantly...


Monitor all the things.

But, perhaps most importantly...

USE UNIFIED TOOLING


What we use...

reconnoiter
SNMP, nad, resmon, statsd, HTTP traps, jdbc, etc.
statsd (clients)
javascript beacons


Middleware mix
API service times, trafﬁc, user signup rates.


Thank you!


It's all about telemetry

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

More from Theo Schlossnagle

More from Theo Schlossnagle (17)

Recently uploaded

Recently uploaded (20)

It's all about telemetry