DealerTrack is the nation’s first and largest credit application network for the automotive industry, connecting 17,000 dealers and over 1000 lenders. Senior Director of Technology Architecture, and one of the founding members, Chris DeMeo detailed DealerTrack’s complex environment, spanning multiple geos and data centers and with the diverse architecture that comes with multiple acquisitions over the course of several years. As is common with our customers, they brought Splunk in for a unified view of their data. “Our developers had full visibility into the anatomy of a problem to rectify quickly and avoid similar errors in the future. We’ve had so many ‘a-ha’ moments with Splunk, it’s become second nature to expect them,” Chris said. DealerTrack has moved from reactive to proactive, and they have a much better understanding of their transaction volume, transaction mix, and watch dashboards for bunching or other challenges during load tests.

1. New
York
City
October
27,
2011

2. Chris
DeMeo
Sr.
Director
Technology
New
York,
October
27,
2011
Architecture
Copyright
©
2011,
Splunk
Inc.

3. About
Me
! One
of
the
founders
of
DealerTrack
! With
the
company
since
incepJon
! Technology
Architecture
Lead
-­‐
Infrastructure
New
York,
October
27,
2011
3
Copyright
©
2011,
Splunk
Inc.

4. About
DealerTrack
! Leading
provider
of
on-­‐demand
soOware
and
services
for
the
automoJve
retail
industry
! Industry’s
First
and
Largest
Credit
ApplicaJon
Network
! ConnecJng
approximately
17,000
dealers
with
more
than
1,000
lenders
! Dealer
Management
SoluJons,
Inventory
SoluJons,
Sales
SoluJons
and
Processing
SoluJons
New
York,
October
27,
2011
4
Copyright
©
2011,
Splunk
Inc.

5. The
Challenge
• Web
logs,
proprietary
logs,
database
logs,
operaJng
system
logs
–
No
way
to
look
through
them
easily,
it
took
hours
• Over
Jme,
acquired
several
companies
resulJng
in
more
dependencies
and
the
need
to
consolidate
logs
across
geo's
and
datacenters
• In
POC
there
was
doubt
about
the
value
of
the
tool.
How
many
hours
would
it
take
to
get
up
and
running?
How
quickly
before
we
see
value?
"We
needed
a
tool
that
could
correlate
all
logs."
New
York,
October
27,
2011
5
Copyright
©
2011,
Splunk
Inc.

6. The
First
of
Many
Aha
Moments
• Oracle
RAC
database
cluster
failover
-­‐
root
cause
analysis
• E-­‐contracJng
transacJon
-­‐
correlaJon
of
customer
impacJng
bug
using
3
correlated
log
types
• Database
connecJons
–
custom
input
allowed
trending
in
new
ways
• With
Splunk
ability
to
quickly
get
to
root
cause
(assuming
you
logged
something!)
• Different
from
anything
we
used
before
• Ability
to
show
a
developer
the
anatomy
of
a
problem
–
transacJon
tracing
New
York,
October
27,
2011
6
Copyright
©
2011,
Splunk
Inc.

7. The
Second
of
Many
Aha
Moments
• Capacity
Planning
–
now
has
the
ability
to
easily
validate
transacJon
mixes
and
volumes
• Capacity
Planning
–
idenJfied
“bunching”
of
transacJons
during
load
tests
• Increased
product
usage
and
customer
retenJon
–
through
idenJficaJon
of
their
legacy
product
usage
• Outage
Impact
–
more
accurate
outage
impact
with
access
to
addiJonal
logs
New
York,
October
27,
2011
7
Copyright
©
2011,
Splunk
Inc.

8. What
Also
Helped…..
• AOer
3
months
execuJves
felt
Splunk
paid
for
itself
• Ease
of
Use
• Development
can
build
their
own
dashboards
• Business
folks
saw
value
and
asked
for
informaJon
• We
had
Splunk
up
and
running
in
less
than
2
months
own
dashboards
New
York,
October
27,
2011
8
Copyright
©
2011,
Splunk
Inc.

9. Vale
Add
to
Lending
Partners
• Real-­‐Jme
visibility
into
credit
applicaJon
and
decision
volumes
across
all
lenders
• Report
to
lenders
where
they
stand
against
the
compeJJon—drives
more
compeJJve
response
New
York,
October
27,
2011
9
Copyright
©
2011,
Splunk
Inc.

10. Historical
Data
Search
! Log
view
uJlity
for
our
own
proprietary
logs…
! Our
own
tool
doesn’t
search
over
more
than
7
days
of
data
! Now
we
can
search
historical
data
for
trending
analysis
New
York,
October
27,
2011
10
Copyright
©
2011,
Splunk
Inc.

11. Current
Environment
and
Set
up
! Deployed
across
4
applicaJon
environments
! Non-­‐Cloud
based
-­‐
Run
our
own
equipment
! Each
environment
has
1
server
(will
make
into
HA
configuraJon)
! Everything
runs
on
a
single
server
! Virtual
machines
in
non-­‐producJon
environments
! Local
storage
New
York,
October
27,
2011
11
Copyright
©
2011,
Splunk
Inc.

12. Coming
Soon
! Syslog
inputs
! HA
configuraJon
with
mulJple
search
heads
and
SAN
storage
! Looking
to
install
Web
AnalyJcs
app
for
business
users
! IntegraJon
with
other
monitoring
tools
! User
adopJon-­‐-­‐work
on
user
educaJon-­‐-­‐how
they
are
using
it
! Want
to
give
Security
group
ability
to
search
across
years
of
data
! Rollout
to
subsidiaries
so
each
has
their
own
indexer
"Finding
data
in
our
logs
that
we
never
even
knew
was
there
and
that
people
are
ge8ng
value
from."
New
York,
October
27,
2011
12
Copyright
©
2011,
Splunk
Inc.

13. Lessons
Learned
! Before
we
didn't
have
the
data,
now
we
want
to
save
everything
! Professional
Services
and
EducaJon
–
They
armed
us
and
made
us
more
than
dangerous!
! Developed
internal
user
doc
with
best
pracJces
and
use
cases
! Knowledge
of
your
logs
will
help
to
realize
value
quickly
New
York,
October
27,
2011
13
Copyright
©
2011,
Splunk
Inc.

14. Thank
you
Chris
DeMeo
Sr.
Director
Technology
Architect
New
York,
October
27,
2011
Copyright
©
2011,
Splunk
Inc.