1. Applica'on
Performance
Monitoring:
Improve
user
experience,
reduce
user
abandonment
Chris&an
Ledwidge
Director
Solu&ons
Architecture
Monitoring
Quest
So9ware
The most comprehensive Oracle applications & technology content under one roof

2. Going
to
tell
you
a
li<le
story
before
we
begin
• Established
well
established
travel
company
wants
to
get
into
the
online
travel
booking
engine
business
– They
have
an
established
brand
in
the
package
holiday
market
with
their
own
fleet
of
aircra9
to
service
– Margins
are
under
the
gun
and
they
need
to
find
new
areas
for
revenue
genera&on
• Its
seems
an
obvious
choice
they
know
the
travel
business
a9er
all
• Have
a
good
name
that
should
help
drive
users
to
their
new
website
• They
ramp
up
a
big
team
of
marke&ng,
IT,
and
business
owners
to
kick
this
project
off
• Some
solid
ini&al
success
and
word
of
mouth
spreads
and
site
hit
counts
move
in
the
direc&on
they
had
hoped
for
The most comprehensive Oracle applications & technology content under one roof

3. Story
con&nued
• BUT!
• Booking
numbers
are
missing
forecasts
by
a
solid
margin
• Calls
to
their
call
center
are
going
thru
the
roof
– Business:
these
call
are
ea&ng
up
our
margin
aren't
they?
• Nobody
in
IT
seems
to
be
able
to
tell
the
business
why
they
are
not
conver&ng
the
solid
site
hit
rate
into
completed
transac&ons
• Lots
of
finger
poin&ng
and
conspiracy
theories
are
geSng
thrown
around
– It’s
the
network!
– No
it’s
the
DB!
– Its
that
new
Java
applica&on
that
THEY
wrote!
– Why
did
we
ever
go
Virtual?
• Only
thing
the
business
cares
about
it
why
are
they
not
conver&ng
all
of
these
poten&al
customers
into
paid
clients?
The most comprehensive Oracle applications & technology content under one roof

4. Part
One
User
Experience
Management
(UEM)
Indicators
• What
are
they
and
what
do
they
do
• Five
key
indicators
and
how
they
can
be
used
• What
sources
can
be
used
to
obtain
indicators
• Important
points
to
remember
The most comprehensive Oracle applications & technology content under one roof

5. UEM
Indicators:
Bridge
the
Gap
between
Business
and
IT
Useable? UEM Indicators
Stable?
Right Content? • User counts
Right Features? • Changes in user behavior
Fast Enough? Business • Geographical variations
• Problem impact
• Service level agreements
Web User Web App Back-end Systems
Available?
Fast Enough?
Short MTTR?
Proactive Problem Solving?
IT
The most comprehensive Oracle applications & technology content under one roof

6. UEM
Indicator:
User
Counts
Twenty-five users have registered in the last 15
Use Case minutes, but normally at this time, I see 200 or more
people registering.
Business IT
Did
our
promo&onal
campaign
stop
Are
there
any
outages
impac&ng
producing
results
or
is
the
link
broken?
registra&ons
that
we
need
to
communicate
to
the
business?
Is
the
reduc&on
in
registra&ons
isolated
to
How
many
users
are
being
impacted
and
a
specific
region
or
is
it
across
the
board?
from
which
region(s)?
The most comprehensive Oracle applications & technology content under one roof

7. UEM
Indicator:
Changes
in
User
Behavior
Typically 25% of users complete this transaction, but
Use Case today we see fewer than 15%; why the changes in
user behavior?
Business IT
Did
the
new
site
rollout
hide
the
submit
Are
problems
with
the
applica&on
causing
bu<on
to
where
some
users
can
find
it
problems
with
this
transac&on
to
where
anymore?
users
can’t
complete
it?
Where
along
the
path
are
we
losing
the
How
can
we
help
the
business
restore
users
and
when
did
it
change?
service
and
measure
the
business
impact
of
the
change?
The most comprehensive Oracle applications & technology content under one roof

8. UEM
Indicator:
Geographical
Varia&ons
We see unexpected geographical variations in user
Use Case volumes from our EMEA region.
Business IT
How
many
users
are
being
impacted
and
Is
our
European
proxy
server
having
for
how
long?
performance
issues?
Is
this
a
problem
we
are
causing
or
is
it
one
Is
there
a
specific
service
provider
that
we
of
our
applica&on
delivery
partners?
or
an
ISP
that
our
users
are
using
experiencing
issues?
The most comprehensive Oracle applications & technology content under one roof

9. UEM
Indicator:
Problem
Impact
A user just called in and reported a problem that our
Use Case monitors didn’t pick up; what is the problem impact?
Business IT
When
did
the
problem
start
and
what
Is
there
a
specific
web
server
area
of
the
site
is
being
impacted?
throwing
these
excep&ons?
Is
the
error
being
displayed
to
end
users
or
Is
the
right
web
administrator
is
it
just
an
image
failing
that
they
don’t
involved
to
get
this
fixed
or
really
see?
diagnosed?
The most comprehensive Oracle applications & technology content under one roof

10. UEM
Indicator:
Service
Level
Agreements
Use Case SLA goals are not being met?
Business IT
Do
the
performance
agreements
that
are
Are
the
IT
systems
mee&ng
the
agreed-­‐
in
place
guarantee
a
sa&sfactory
user
upon
delivery
of
services
to
end
users?
experience?
When
the
SLA
viola&ons
started
occurring
Were
our
Web,
App,
or
Database
systems
were
there
devia&ons
in
transac&on
at
fault
or
was
it
an
external
interface?
conversions?
The most comprehensive Oracle applications & technology content under one roof

11. User
Experience
Management
Data
Sources
• User Counts
• Changes in User Behavior
• Geographical Variations
• Problem Impact
• Service Level Agreements
Data Sources
• Web Log Files User Experience
• Synthetic Transaction Robots Management
• Client-Side Tagging
• Web Traffic Analysis Appliances
Indicators
• Content Capture Appliances
• User Surveys
The most comprehensive Oracle applications & technology content under one roof

12. Web Log Files
Data Source
Usefulness Limitations
• Error tracking • Difficult to identify
• User click stream analysis individual user sessions.
• Demographic analysis • Data set is partial due to
caching.
• Performance
The most comprehensive Oracle applications & technology content under one roof

13. Synthetic Transaction Robots
Data Source
Usefulness Limitations
• Measures transaction • Measures only a small
performance and percentage of locations.
availability via scripted • Exercises a small
transactions executed percentage of the
from remote locations at technology stack in larger
set intervals. environments.
• Sometimes referred to as
a “clean room”
measurement.
• Good for service level
reporting.
The most comprehensive Oracle applications & technology content under one roof

14. Client-Side Tagging
Data Source
Usefulness Limitations
• Good for measuring true • Inserting and managing
the scripts adds
client performance
complexity and overhead
• Gathering browser-based to the application
events needed to • Data security concerns
understand Web 2.0 user
interactions
The most comprehensive Oracle applications & technology content under one roof

15. Web Traffic Analysis Appliances
Data Source
Usefulness Limitations
• 100% sampling of • Asynchronous
applications make it
transaction performance,
difficult to see cause and
volume, and errors. effect.
• Analysis has many • Analyzed data set is
perspectives including by partial due to external
caching.
server, by geography, by
ISP, and even by user.
The most comprehensive Oracle applications & technology content under one roof

16. Content Capture Appliances
Data Source
Usefulness Limitations
• Stores the full user • Storage capacity
sessions, including the concerns
data returned from the • Privacy and security
web server. concerns
• Session replay and other
perspectives on the
session that are
attainable only when full
payloads are accessible
The most comprehensive Oracle applications & technology content under one roof

17. User Surveys
Data Source
Usefulness Limitations
• True voice-of-the- • Typically only small
customer samples are gathered,
• Can validate metric- and small samples may
based hypotheses represent extremes.
• Answer questions like:
o Why are customers
happy?
o Why are customers
ready to leave?
The most comprehensive Oracle applications & technology content under one roof

18. Part
Two
• Measuring
user
experience
in
web
1.0
and
2.0
environments
• Top
5
performance
measurement
challenges
• Applying
approaches
to
address
the
challenges
• Key
elements
to
remember
The most comprehensive Oracle applications & technology content under one roof

19. User
Experience
Management
Performance
Measurements
Internet
Web App Back-end Systems
Request
Response
Client
Time
Network
Time
Processing
Time
•
Web
1.0
–
Simple
requests
and
responses
•
Web
2.0
–
Asynchronous
requests
and
responses
•
CDNs
and
content
hos&ng
–
third-­‐party
effects
The most comprehensive Oracle applications & technology content under one roof

20. Web
1.0
–
Simple
Page
Request
No
Third-­‐party
Connection 1 Connection 2 Connection 3
1. Request
ini&ated
Client Server Client Server Client Server
GET main.html 2. Server
gets
first
1 packet
2 GET style.css
3. Server
sends
last
GET frame1.html
packet
4. Final
response
GET image1.gif
received
by
client
GET news.asp
3
4
The most comprehensive Oracle applications & technology content under one roof

21. Web
2.0
–
Asynchronous
Requests
Connection 1
1. Request
ini&ated
Client Server
GET item.js 2. Server
gets
first
1 packet
2
3. Server
sends
last
packet
3
4. Final
response
4 received
by
client
The most comprehensive Oracle applications & technology content under one roof

22. CDNs
and
Remote
Content
–
Third-­‐Party
Effects
Connection 1 Connection 2 Connection 3 Connection 4
Client Server Client Server Client Akamai Client Akamai
GET main.html
GET style.css
1
GET frame1.html
GET perf.js
GET news.asp
2
3
4
The most comprehensive Oracle applications & technology content under one roof

23. User
Experience
Management
Measuring
Performance
to
Solve
IT
Challenges
“The user tries to explain her
“We initially can’t tell if the problem, but it really doesn’t “We don’t know how many
issue is on the client, in the help me resolve it.” and which users are having
network, or buried in the this experience.”
backend infrastructure.”
“We learn about most “We lack historical data to
problems only when a user proactively detect unwanted
calls in” performance trends and their
IT root cause.”
Internet
Web User
Web App Back-end Systems
The most comprehensive Oracle applications & technology content under one roof

24. Challenge
One
“We
learn
about
most
problems
when
users
call”
Management
Approach
Minimal
Coverage
BeCer
Coverage
Best
Coverage
• Log
parser-­‐-­‐
alerts
on
• Log
parser-­‐-­‐
alerts
on
• Synthe&c
package-­‐-­‐to
errors
or
performance
errors
or
performance
add
transac&on
perspec&ve
• Synthe&c
package-­‐-­‐to
• Selec&ve
page
add
transac&on
instrumenta&on-­‐-­‐
perspec&ve
captures
third
party
content
provider
issues
• Passive
sniffer-­‐-­‐low
overhead
capture
all
solu&on
The most comprehensive Oracle applications & technology content under one roof

25. Challenge
Two
“User
explana'on
is
not
useful
for
resolu'on”
Management
Approach
Minimal
Coverage
BeCer
Coverage
Best
Coverage
• Call
the
user(s)
back
and
• Enlist
support
teams
to
• Selec&ve
page
inquire
about
issue
recreate
issue
instrumenta&on-­‐-­‐
captures
desktop
events
• Scrape
web
logs
for
• Use
online
desktop
• Passive
sniffer-­‐-­‐captures
traces
of
informa&on
sharing
to
record
full
content
and
about
the
issue
customer
scenario
recreates
scenario
The most comprehensive Oracle applications & technology content under one roof

26. Challenge
Three
“Can’t
isolate
issue
to
client,
network,
or
backend”
Management
Approach
Minimal
Coverage
BeCer
Coverage
Best
Coverage
• Look
for
anomalies
in
• Look
for
anomalies
in
• Selec&ve
page
web
logs-­‐-­‐dismiss
web
logs-­‐-­‐dismiss
instrumenta&on-­‐-­‐
problem
if
no
anomalies
problem
if
no
anomalies
captures
third-­‐party
content
performance
• Use
synthe&c
solu&on
to
• Passive
sniffer-­‐-­‐
isolate
issue-­‐-­‐if
it
shows
differen&ates
client
from
up
network
&me
for
requests
The most comprehensive Oracle applications & technology content under one roof

27. Challenge
Four
“Don’t
know
how
many
users
experiencing
issue”
Management
Approach
Minimal
Coverage
BeCer
Coverage
Best
Coverage
• Count
anomalies
in
web
• Count
anomalies
in
web
• Selec&ve
page
logs-­‐-­‐es&ma&ng
impact
logs-­‐-­‐es&ma&ng
impact
instrumenta&on
when
from
counts
from
counts
third-­‐party
content
being
served
• Use
synthe&c
solu&on
to
• Passive
sniffer-­‐-­‐assess
judge
impact
according
impact
of
all
first-­‐party
to
bad
loca&ons
requests
The most comprehensive Oracle applications & technology content under one roof

28. Challenge
Five
“No
historic
info
to
detect
performance
anomalies”
Management
Approach
Minimal
Coverage
BeCer
Coverage
Best
Coverage
• Customer
sa&sfac&on
• Web
log
analysis
tools
• Selec&ve
page
surveys
with
long
term
reten&on
instrumenta&on
when
third-­‐party
content
being
served
• Web
log
analysis
tools
• Synthe&c
solu&on
with
• Passive
sniffer-­‐-­‐for
first
with
long
term
reten&on
base
lining
capabili&es
party
base-­‐lining
and
anomaly
detec&on
The most comprehensive Oracle applications & technology content under one roof

29. Important
Points
to
Remember
• User Experience Management encourages a common
understanding between IT and the business
• Common indicators enable collaboration
• Sources of User Experience Management information are
many—use those that best fit your needs
The most comprehensive Oracle applications & technology content under one roof

30. Part
Three
• Why
Manage
Transac&on
Conversions
• Web
Analy&cs
vs.
Transac&on
Conversion
Analysis
• Measuring
Transac&on
Conversions
• Top
5
Challenges
to
Managing
Conversions
• Points
to
Remember
The most comprehensive Oracle applications & technology content under one roof

31. Why
Manage
Transac&on
Conversions
• Why
do
we
manage
transac&ons
the
way
we
do?
• Example
Transac&on
Types
ü Shopping
Carts
ü Ad
campaigns
ü Customer
registra&on
ü Login
The most comprehensive Oracle applications & technology content under one roof

32. Why
Transac&on
Analysis—We
already
have
Web
AnalyCcs?
Web
Analy'cs
Transac'on
Analysis
• Primary
goal:
How
effec&ve
your
• Primary
goal:
What
kind
of
marke&ng
is…
experience
your
users
have…
• Answers
“How
effec&ve
is
my
• Answers
“What
influenced
user
marke&ng?”
transac&on
conversions?”
• Observe
how
users
use
your
• Observe
how
user
transac&on
website:
conversions
are
affected
by:
– What
transac&ons
they
use
and
how
– Content
o9en
– Applica&on
func&onality
– Click
stream
analysis
– Applica&on
useability
– Applica&on
stability
Complementary solutions
The most comprehensive Oracle applications & technology content under one roof

33. User
Experience
Management
TransacCon
Conversion
Analysis
The most comprehensive Oracle applications & technology content under one roof

34. Measuring
Transac&on
Conversions
Session
Page 1 Page 2 Page 3 Page 4
Hit Hit Async Hit Hit Async Hit Hit Async Hit Hit Async
Transaction
Start Step 2 Step 3 End
Capturing
Measurements
• Unique
Request
ID
(Off
the
wire)
• JavaScript
Instrumenta&on
– URL,
Page
Title,
Content,
Variable,
Loca&on,
– Captures
page
or
request
level
etc..,
informa&on
per
session
– Persists
current
transac&on
state
per
session
in
– Reports
transac&on
progress
back
to
memory
central
monitoring
server
– Result
sent
to
a
third
or
first
party
The most comprehensive Oracle applications & technology content under one roof

35. User
Experience
Management
Increasing
Conversion
Rates
through
Conversion
Analysis
“We know where people
“I need to understand bad “We can’t identify how
are abandoning the
conversion trends immediately many and which users
website but we don’t
to reduce lost business.” are associated with
understand why.”
conversion issues.”
“How much money have “When people abandon the
we lost because of this site, they often call customer
problem?” service – which drives up our
Business cost of service.”
The most comprehensive Oracle applications & technology content under one roof

36. “Revenue
loss
-­‐
a
Key
Indicator
example”
1.
Recognized
add-­‐to-­‐cart
page
My
Transac'on
Record
2.
Page
analyzed
My
Transac'on
Record
Transac&on
Name
Buy
Funnel
•
Parse
$$
amount
Transac&on
Name
Buy
Funnel
3.
Updates
Transac&on
Record
Username
Mike
Smith
Username
Mike
Smith
•
Add
to
current
Cart
Amount
Browser
IE
8
4.
Transac&on
Completes
Successfully
Browser
IE
8
Cart
Amount
$625
5.
Increment
Sale
Amount
Cart
Amount
….
•
Set
Lost
Sale
Amount
to
0
Transac&on
State
Ac&ve
Transac&on
State
Complete
Success
Sale
Amount
N/A
Sale
Amount
$625
Lost
Sale
Amount
N/A
The most comprehensive Oracle applications 0
Lost
Sale
Amount
& technology content under one roof

37. “Understanding
Bad
Conversion
Trends”
My
Transac'on
Record
Transac&on
Name
Buy
Funnel
Username
Mike
Smith
Browser
IE
8
Cart
Amount
$625
Transac&on
State
Ac&ve
Sale
Amount
N/A
Lost
Sale
Amount
N/A
• Quan&ta&ve
measures
available
in
Analy&cs
and
UEM
• Allows
us
to
form
a
hypothesis
about
conversion
trends
• Qualita&ve
valida&on
is
the
next
step
to
validate
• Session
replay
is
that
valida&on
The most comprehensive Oracle applications & technology content under one roof

38. “The
5
W’s
of
Transac&on
Abandonment”
• Where
are
users
dropping?
My
Transac'on
Record
My
Transac'on
Record
• Which
users
are
dropping?
Transac&on
Name
My
Transac'on
unnel
Transac&on
Name
Buy
F
Record
Username
My
Transac'on
unnel
Buy
F
Record
Mike
Smith
• Why
are
users
dropping?
Transac&on
Name
Username
Username
My
Transac'on
unnel
Buy
F
Record
Mike
Smith
Browser
Transac&on
Name
8
My
Transac'on
unnel
IE
Buy
F
Record
Mike
Smith
Browser
Transac&on
Name
8
IE
Buy
Funnel
• When
did
users
start
dropping?
Cart
Amount
Username
Cart
Amount
Username
Transac&on
State
Browser
$625
Browser
Transac&on
Name
8
IE
$625
Mike
Smith
Ac&ve
IE
8
Buy
Funnel
Mike
Smith
Cart
Amount
Username
$625
Mike
Smith
• What
were
users
doing
when
they
dropped?
Transac&on
State
Browser
Sale
Amount
Amount
Cart
Transac&on
State
Browser
Sale
Amount
Amount
Cart
Ac&ve
IE
8
N/A
N/A
$625
Ac&ve
IE
8
$625
Lost
Sale
Amount
Transac&on
State
N/A
Ac&ve
Sale
Amount
Amount
Cart
N/A
$625
Lost
Sale
Amount
Transac&on
State
N/A
Ac&ve
Sale
Amount
N/A
Lost
Sale
Amount
Transac&on
State
N/A
Ac&ve
Sale
Amount
N/A
Lost
Sale
Amount
N/A
Sale
Amount
N/A
Lost
Sale
Amount
N/A
Lost
Sale
Amount
N/A
The most comprehensive Oracle applications & technology content under one roof

39. “People calling customer service drive up costs”
• Relying
on
customers
to
call
in
when
problems
with
transac&ons
is
costly
– Lost
revenue
– Damage
to
brand—reduces
repeat
purchases
– Call
center
personnel
are
expensive
• Transac&on
Conversion
Analysis
provides
insight
to:
– Spot
unwanted
conversion
trends
before
they
impact
the
business
– Increases
efficiency
of
call-­‐center
personnel
—look
at
a
user
session
to
recreate
the
problem
The most comprehensive Oracle applications & technology content under one roof

40. Points
to
Remember
ü Transaction analysis is insurance for achieving
and maintaining desired business outcomes
ü Weigh the costs of not doing versus doing
conversion analysis
ü Is analytics enough or do you need something
more to make you analytics actionable?
The most comprehensive Oracle applications & technology content under one roof

41. Thank
you
for
your
&me
Please
drop
by
the
Quest
booth
in
the
vendor
hall
if
you
would
like
to
have
a
look
at
our
approach
to
Applica&on
Performance
Management
The most comprehensive Oracle applications & technology content under one roof