1. Craig Sullivan
The Multi-variate advantage Group CX Manager
How testing saved my life Belron®
@OptimiseOrDie

2. Naked self promotion
@OptimiseOrDie
• Conversion Rate Optimisation • Contact deflection and
• A/B and Multi-variate testing online self service
• Cross channel optimisation • Site search analytics
• User Centered Design • Site, Page and Campaign
optimisation
• Usability testing
• Test design and execution
• Customer Research & Insight
• A long usable site portfolio
• Web Analytics
• ROI on UX improvements
• Browser and email
compatibility • Over 18 years of boring
meetings
Group CX Manager for Belron® (Safelite)
I get to do it with crowds

3. Outline
• What is optimisation?
• Belron® Testing Culture
• What is MVT
• Our testing program
• Advantages of testing
• The results
• Cross channel optimisation
• Case study : Chat ROI
• Resources

4. What’s optimisation?
@OptimiseOrDie

5. A set of techniques, implemented in
order to influence customer
behaviour towards these outcomes:
• Increased revenue or profit
• Increasing NPS (Customer Sat) scores
• Lower cost for business or contact centres
• Increasing productivity or labour flexibility
• Simply delighting customers
We may fail with 90% of what we try.
The 10% of winners let us shift
behaviour hugely and measurably.

6. ®
Belron Testing Culture
@OptimiseOrDie

7. If you go to the men's washrooms at the
Schiphol airport in Amsterdam, you may
notice there's a fly in the urinals. It’s
screen printed .
So what do you think most men do? That's
right, they aim at the fly when they
urinate.
They don't even think about it, and they
don't need to read a user's manual; it's
just an instinctive reaction that means
85% less spillage!
The interesting feature of these urinals is
that they're deliberately designed to take
advantage of this inherent human male
tendency.
This is my job.

8. Small team, big challenge
• Team of 3 - 2 Optimisers, 1 x part time designer
• Higher ROI or lower costs, without spending more
• Stuff that doesn’t hinder SEO, PPC, Performance
• Measurable ROI - not ego or opinion
• Fast return and low cost techniques
• Over 26M page tests, 17 international tests live now
• ROI is > 50 x optimisation costs
• Total cost = ~800k for two years. ROI > $40M USD

9. Primer on testing
=

10. A/B test
Control Version A Version B

11. Multi variate Pizza
Ingredients Experiment Recipes
=

12. Month 1 - results
+2% +3% +1% +2%
+9% +4% -2% +1%
+5% +13% -7% -9%

13. So what about ingredients?
+20% -2% +5%
+3% +2% +6%
+6% +10% -18%

14. Weed out some losers…
+2% +3%
+9% +4%
+5% +13%

15. The final…
+10% +13%

16. Our testing program
@OptimiseOrDie

17. Insight - Inputs Forms
Remote Search
analytics Depth
testing analytics
interviews
Eye tracking Market
research
Customer A/B and
contact MVT testing
Social
analytics
Insight Unstructured
data
Web
Clicktale
analytics
Usability Competitor
testing Ops and CS Customer evals
teams Surveys services

18. Killer idea generation sources
• Analytics data
• Customer insight – contact, customer feedback, research
• Usability testing
• Diary studies
• Clicktale recordings
• Business owners – market data, customer mix, local knowledge
• Customer contact centre immersion (listening)
• Books, Websites, Tips
• Other optimisers
• Test data – variables plotted on axis
• Making lots of mistakes – Fail faster

19. Typical test lifecycle
Other test
Gather and results – good Brainstorm &
Instrument
review inputs variables & Hypotheses
learnings
Signoff (Legal, Design variables
Instrument Test Plan
Marketing, & creative
analytics Document
Brand) elements
Next Wave
Wave design Results of wave Washup
(repeat)
Business led Do verification
Repeat from
reporting (with = wave + 90/10% Declare winner
start
£x, without = £y) split

20. What’s our testingand SpokeThe traffic picture
Hub mix – model – homepage traffic

21. What’s our testing mix?
• A/B testing – Homepage, Landing pages, Funnels
• MVT testing – high traffic landing or home pages
– Mini (less than 8 variables)
– Midi (8-32)
– Maxi (>32 variables)
And 5 types of specialised testing:
• Funnel testing – bedding in new funnels
• Landing page optimisation
• Cross channel testing
• Iterative feature and product testing
• Micro testing
• Telephony MVT

22. Level 1 Level 2 Level 3 Level 4 Level 5
Starter Level Early maturity Serious testing Core business value You rock, awesomely
Dedicated team Ninja Team
Local Heroes Small team Cross silo team
Culture Volume Testing in the
Chaotic Good Low hanging fruit Systematic tests
opportunities DNA
_____________________________________________________________________________________________ _
Process Ad Hoc Outline process Well developed Streamlined Company wide
_______________________________________________________________________________________________
+Funnel +Cross channel +Spread tool use
Guessing + Multi variate optimisation testing Dynamic adaptive
Testing A/B testing Session replay Call tracking Integrated CRO targeting
focus Basic tools No segments Some segments and analytics Machine learning
Micro testing Segmentation Realtime
________________________________________________________________________________________________
Bounce rates + Funnel analysis + Funnel fixes + offline Multichannel
Analytics Low converting
integration funnels
Big volume Forms analytics
focus landing pages & High loss pages Channel switches
Single channel Cross channel
picture synergy
________________________________________________________________________________________________
+Regular usability + All channel view
Analytics + Customer sat
testing/research +User Centered of customer
Insight Surveys
Prototyping
Design scores tied to UX
Driving offline
Contact Centre Rapid iterative
methods Low budget Session replay
Layered feedback
testing and
using online
usability
Mini product tests design All promotion
Onsite feedback driven by testing
________________________________________________________________________________________________
Continual
Mission Get buyin Prove ROI Scale the testing Mine value improvement

23. International testing examples
@OptimiseOrDie

25. UK Wave 2 – 21/9 to 26/10

30. What is the ROI then?

31. Another example

32. Testing advantages
@OptimiseOrDie

33. Your ‘design’ team?
Web/Online
Pesky Meddling Kids?
Marketing
Who
designs this
stuff
anyway?
Brand
IT
THE
CEO CUSTOMER Legal

34. BENEFITS OF TESTING:
• Reduced ego and opinion (execs + others)
• Measurable ROI
• Faster time to market
• Increased conversion in the pocket early
• Iterative and rapid product lifecycles
• Faster, smaller, focused, measured releases
• Continual testing = continual improvement
• SLED vs. Agile = faster return, better focus
• Less reliance on anecdotal evidence
• Business owners part of testing culture
• Introduces a testing culture
• Large cross silo team now gets involved
• Reduces timidity – be brave and curious!
• Helps us hit the sweet spot...

35. Cross Channel Optimisation
@OptimiseOrDie

36. Insight - Inputs Ringback
Phone Email
Retargeting Chat
Mobile
TV handsets
Display Apps
Affiliate
Exploding Tablets
Channels
Desktops
Direct
Social &
SEO Micro
Partners
PPC
DM

37. TV advertising and Web
TV - Off TV - On

38. People images that suck…

39. Photo Guidelines
• Groups, vans, stores, corporate guff = • Female images work best in almost
NO
every country – they slay the guys
• Single person – not groups or even
person + customer • Natural, authentic and not ‘model’
• Open, Friendly, Natural, Smiling,
or ‘stock’ image types work best for
Engaged = YES people
• Uniform – has huge effect, even • We get better images now, but we
wearing branded ball cap improves keep working on the guidelines!
conversion • We still got a genital framing
• Look straight at viewer or slightly picture the other week
towards CTA
• Photoshoots and tests are now
• Plain backgrounds only
being used to drive advertising
• No folded arms, arms on hips, pointing, strategy
scowling, cheesy smiles, hands behind
back, hanging useless by the side, • Pre TV test of potential ‘actors’ on
clasped by groin, fiddling with hands website. X factor meets split
• Arms with prop instead – hold a testing.
clipboard, rag, tool – I’m working for
*you*

40. Cross channel mini studies
• Mini studies:
– Abandon testing
– Emergency phone switchoff & Overspill systems
– Contact deflection systems and self servicing
– Scripting Experiments
– APIs and other systems integration
– Chat testing
– Phone tracking

41. Abandon testing
Reduce unwanted phone hangups. Test plan
– Remove corporate guffspeak – Two countries, big phone systems,
– Don’t mention the website, esp. – Test variants of messages, music,
if this is the web number (D’oh!) timings, amounts quoted
– Don’t give a generic or corporate – Messages make a big impact –
drone apology message experiment freely with volumes
– Quote real KPIs and timings (ING – Opportunity is large due to high
direct, average queue time) call volumes we have
– Make routing emotionally driven – What people hate personally vs.
– not confusing categories that what those same people have on
map poorly onto customer needs their company lines, lol
– Experiment with queue timing – – On a mission here!
find sweet spot, don’t overquote

42. Switchoff Overspill
If you have spikes, have a throttle: Overspill systems:
– Busiest TV & Radio campaign so far, Q2 – Get a UX designer to make you a fast
2011 for Autoglass® entry system
– Designed a phone ‘switchoff’ system, – Make them simple, easy to learn and
permanently on tap
leverage your web platform
– Routed majority of contact to a web
form – Give API access to overspill
– Prevented a backlog situation from companies
developing – Sometimes heavy IT integration isn’t
– Bought time and still exceeded needed
customer expectations (<4h excellent) – “Nobody needs a freaking training
– Used twice to dampen inbound call course to use facebook” - Make sure
volumes your internal apps and call centre
overspill systems are built for instant
learnability
– This could save millions in process
efficiency. Same as web, really.

43. Contact deflection & self service
• People like immediate gratification! • What about your measurement?
• The iceberg of contact opportunity • Basics – Ratings, Did you find this
• Make people like you for not talking answer helpful?
to them on the phone –double win • Intermediate – Tracking behaviour
• If you stick it in help, will it go away? and contact channels
• No elephant graveyard, please (FAQ) • Add tracking of self service events or
• Instrument for analytics and ROI behaviour
• Set it up to measure and improve • Set up a game or reward structure for
staff
• Gamification of content? Adoption?
• Continually improve and get proof
• LOVEFiLM Help Centre:
• Add phone tracking - make avoidable
– Ratings, Deflection tracking, self contact visible
service tracking
• Remove all barriers to self servicing
– V1 :Please view in your account for high cost/volume contacts
secion, V2 : Please visit your
account, V3 : Please check your • Try Amazon help – where is phone?
next billing date, V4 - Final • Help experts? Sticky Content are
version – to show the real info : brilliant (www.stickycontent.co.uk)
“Your next billing date is 1 Feb”

44. Script Optimisation
• How can we optimise phone conversion?
– Dedicated test teams
– Script testing often limited range/shift in companies now
– Now – we can use A/B or MV testing, with API integration
– Try out 3,4,5 different versions of key areas or entire script
– ‘Wrapping’ sends XML message event, triggers analytics tag
– Analyse in the same way as an A/B or multi-variate web test
– Fast optimisation of handling scripts, using a mini lab
– Should yield 3-5% higher performing scripts, minimum
– Shows what variables drive script performance globally
– These emotional and psychological triggers can be mined
– Aim for Q2 2012 for a proof of concept

45. Platform APIs
• Use of web platform APIs now for:
– Contact centre messages
– Facebook app
– Mobile site and apps
– Scheduling
– Analytics tag firing
• Just making APIs, automagically makes:
– People use them
– Improvements happen
– New ideas happen, that you didn’t expect
– Incompatible stuff get glued together
– Complex IT integration projects go away

46. Chat Testing Ringback
Contact
Form
Contact
Us
You must measure behaviour
across contact channels
during tests like these
A - Chat B - Chat C - Chat
Split
A B C

47. A - Chat Free B – With Chat
Damage 0844 371 0047 Damage 0844 371 0051
Appt 0844 371 0048 Appt 0844 371 0052
Details 1 0844 371 0049 Details 1 0844 371 0053
Details 2 0844 371 0050 Details 2 0844 371 0054

48. Chat testing
• 1 month trial – properly A/B tested • Customers happy, yes, but not in a
• Measured all the contact channels for good business way
movements – what happened?
• Like putting your customer service
• Over a week, chat saves 17 calls – desk into the street
yay!
• You must measure influences in
• But swapped that for 329 chats @
contact volumes when trialling
8.25 minutes. Ouch.
Chat
• Increased ringback requests? Email
form about the same. • Be very careful – prove your ROI
on any pilot.
• No statistically significant impact on
conversion • Only 7-10% use and out of those,
• Assumption was big impact, huge less than 50% are influencable etc.
impact! etc.
• Anecdotally, a wonderful tool. • The productivity Myth of chat
Agreed. • Your situation may differ – please
TEST!

49. Call tracking
How it works:
• Rent a bank of numbers
• Add a dynamic ‘tag’ to your page
• Phone number appears automatically
• Customer calls this number
• Captures the keywords, landing page, converting page, traffic source
• Geo mapping captures customer location
• When the call is hung up, optionally record the outcome code (sale, loss)
• Have a special number for Facebook, SEO, Adwords traffic etc.
• Lets you analyse from web -> phone call -> wait time -> wrapping
• Builds a link between web and call, finally

50. Call tracking
Why you should use it!
• Most people under-report on web driven phone contact or sales
• You can’t work out your advertising effectiveness, if you don’t include
phone
• If you don’t measure phone, your A/B tests may be wrong
• You can capture all your web AND phone, in one easy interface
• Allows you a multi-channel view of all your contact sources
• Lets you record offline campaigns as well with specific numbers (print, TV
etc.)
• You can experiment with routing, messages, on hold music – much more
easily
• If you can’t afford lots of phone numbers, this is an easy way to get
tracking

51. Company Website Coverage
Who?
Mongoose Metrics* www.mongoosemetrics.com UK, USA, Canada
Ifbyphone* www.ifbyphone.com USA
TheCallR* www.thecallr.com USA, Canada, UK, IT, FR, BE, ES, NL
Call tracking metrics www.calltrackingmetrics.com USA
Hosted Numbers www.hostednumbers.com USA
Callcap www.callcap.com USA
UK, SE, FI, NO, DK, LT, PL, IE, CZ, SI,
Freespee* www.freespee.com
AT, NL, DE
Adinsight* www.adinsight.co.uk UK
Infinity tracking* www.infinity-tracking.com UK
Optilead* www.optilead.co.uk UK
Switchboard free www.switchboardfree.co.uk UK
Freshegg www.freshegg.co.uk UK
Avanser www.avanser.com.au AUS
Jet Interactive* www.jetinteractive.com.au AUS
* I read up on these or talked to them. These are my picks .

52. Call tracking
What data can you get?
• Data to keyword, group or traffic source level – for Phone
contact / conversions
• Splits by location, traffic source, time, date
• Data on the call – how long to answer, to abandon, call
duration
• Data post call – was there a sale? (call wrapping)
• Linkage between marketing activity and phone ROI on web
channels (very important)
• Which parts of your site drive contact
• Tagging supports GA, Omniture, Others
• Complete call analytics (including recording, if wanted)

53. Tracking tips
• Make contact points trackable (phone, email, form, chat etc.)
For example:
• Collapsible phone section on Contact Us form – tracks intent
• Phone ‘reveal’ icon, button or widget – tracks intent
• Tap to call wrapper on Phone
• Use a smart email form instead of email address
• Use ‘ringme’ ringback functionality
• If email being generated, tweak subject line
• Integrate call centre events for maximum coverage
• If you can’t buy phone tracking, make your own (mail me!)
• Use phone tracking for major traffic groups, say:
• PPC groups, Organic, Print, PPC, email, TV, Brand keywords
• See your phone traffic in GA, Omniture and most packages

54. Follow the money – ROI example
• 12.4% from Web increase
• 12.5% increase from Multi Variate Testing
• 3.5% increase in downstream conversion rate
• 4% increase in online conversion from mobile optimised site
• Reduced call time
• Faster booking time
• 5.5% increase in NPS (Customer sat) score
• Customer delight – feedback. Less questions and worries
• Total increase in conversion (measured) = 32%
• Actual increase in conversion = 41%
• NO additional marketing spend needed
• Web Channel : ROI positive within 2 weeks

55. More reading. Slides and resources on slideshare.net
Email : sullivac@gmail.com
Twitter : @OptimiseOrDie
: linkd.in/pvrg14
Slideshare : slidesha.re/nlCDm6

56. CRO and Testing resources
• 101 Landing page tips : slidesha.re/8OnBRh
• 544 Optimisation tips : bit.ly/8mkWOB
• 108 Optimisation tips : bit.ly/3Z6GrP
• 32 CRO tips : bit.ly/4BZjcW
• 57 CRO books : bit.ly/dDjDRJ
• CRO article list : bit.ly/nEUgui
• Smashing Mag article : bit.ly/8X2fLk