An overview of the quantitative and qualitative data provided by live chat, and how to measure the sales, marketing, and customer support ROI of a chat widget.

1. olark.com
Measuring Chat Success
Presented by Karl Pawlewicz, Voice of Olark
February 7, 2017
@karlpawlewicz | @Olark

2. olark.com
Agenda
What does it mean to be a Data Driven Business?
What's in the Olark data stack?
How are we organized to make data accessible?
What kind of data does chat (or a similar customer comms channel) provide?
Real world examples of data driven decisions
Where can you go online or IRL for more insight on data best practices?
Q&A

3. olark.com
What does 'data driven' mean?
Leadership/CEO
"Data has been an essential part of business since the existence of business. The
main difference is now it's easier to get both good and bad data. To be data driven
is not to ignore hunches and intuition, but to figure out the best method to validate
your ideas, and to act on it.
"Data drive decision making reduces uncertainty, but does not remove it. Without
data, we are just guessing, and hoping for the best."
"Assess our performance at setting and achieving objectives.
"Help product managers make decisions to determine where to focus their efforts,
and understand the results of our efforts in the past, and help frame the revenue
impact."
"Deciding where to invest in growth, I use information from inside and outside of
the organization to make informed decisions on the potential impact of various
options."

4. olark.com
What does 'data driven' mean?
Marketing
"Using data to make better team decisions, I can see which campaigns are working
(or not), and easily calculate ROI - not just for dollars spent, but also for the time
and resources we're investing into each activity.
Being data-driven, especially in marketing, makes it easy to think strategically, plan
better, help our own team understand how their contributions affect our goals, and
show the company as a whole how we're impacting the bottom line."

5. olark.com
What does 'data driven' mean?
Product/Research
"The way I see it, data is most useful when it helps you identify "obvious" and
inarguable truths i.e. "Wow what a huge difference in performance, we should
definitely do that thing" or "Wow that's clearly a bad idea, it got dominated by this
other thing."
Data helps provide definite answers for smaller, simpler problems. But sometimes,
for bigger and more difficult decisions that seem to be 50/50 even after you've seen
all the data, those are the ones where great intuition come in and (after a certain
point) no amount of additional data would help.
Steve Jobs is a common example of someone with great product insights. No
amount of data would have been able to definitively tell him either way that a new
device like the iPad would be a huge success. However, I'm sure data allowed Apple
to become pretty confident in taking that risk."

6. olark.com
What does 'data driven' mean?
Engineering
"- project requirements include identifying / collecting / analyzing key metrics on
which to base decisions
- project deliverables include some report on those metrics
More generally, projects are seen as experiments, and therefore are not complete
without the data and the analysis."

7. olark.com
What's in the Olark data stack?
Customer data:
Looker
Segment
Amplitude
Google Analytics
Marketing data:
Hubspot
Unbounce
SEMRush
Moz
Google Analytics
Third Party APIs
Monkey Learn
Builtwith
Alexa
FirstOfficer

8. olark.com
How are we organized?
Team size:
Data department of one consults with all departments to understand common and
team-specific needs.
Important aspects of this system:
One central warehouse so that all team members are using the same data. It's kept up
to date and is known to be high-integrity.
Data accessibility in incredibly important, but having standard definitions that everyone
on the team knows and can trust is as important. If multiple team leads are building
reports, it helps to have the data team define and implement the metric definitions
so that everyone is computing the same thing, and not people reporting metrics
with the same name differently.

9. olark.com
Data from chat?
Quantitative:
Chat volume
How's my team doing
Revenue
Where are people chatting on my site
What percentage of people chat
What countries are visitors chatting from
Qualitative:
Product feedback
What are people saying about my company (sentiment)
What are people saying about my competitors
Which chats lead to sales

10. olark.com
Data from chat?
Quantitative - Volume

11. olark.com
Data from chat?
Quantitative - How's my team doing

12. olark.com
Data from chat?
Quantitative - Revenue

13. olark.com
Data from chat?
Quantitative - Where are people chatting on my site

14. olark.com
Data from chat?
Quantitative - What percent of people chat

15. olark.com
Data from chat?
Quantitative - Where country are visitors chatting from

16. olark.com
Data from chat?
Qualitative -
Product feedback

17. olark.com
Data from chat?
Qualitative - How are people
comparing me to my competitors?

18. olark.com
Data Driven Decisions IRL
What do you want chat to do?
Improve customer happiness?
How to measure: Chat reports, NPS,
Increase conversions?
How to measure: Number of conversations, Number of sign-ups
Increase sales?
How to measure: Revenue reports, CRM results
Product feedback?
How to measure: Chat transcripts
Marketing insight?
How to measure: Chat transcripts

19. olark.com
Data Driven Decisions IRL
OnePageCRM -
Alan O'Rourke at OnePageCRM:
Manually compared new customers in their CRM with Olark logs to see who talked to customer
success before signing up.
New customers who chatted to customer success before signing up were almost twice the size of our
average customer.
His two person live chat team brought in an additional $24,000 in new business in over 12 months (not
including renewals).
240 hours on chat, earning $100 every hour - an average that out performed most of their other
marketing channels.
Based on this data, Alan made the decision to hire another person to staff chat on OnePageCRM's
marketing site in a timezone they weren't already covering.
SOURCE: https://blog.olark.com/how-to-measure-live-chat-success-with-analytics

20. olark.com
Upper90Soccer.com -
Ben Jata of Upper90
"Should I add more
people on chat?"
Conversion Rate increased
11% when chatting
Full-time chat specialist
generated $40K in revenue
in roughly 6 months
Jan
16
Feb
16
Mar
16
Apr
16
May
16
Jun
16
Jul
16
Aug
16
Sept
16
Oct
16
Nov
16
Dec
16
OLARK
Total chats 5 4 10 7 7 154 328 407 391 416 431 428
Great chats 2 1 4 0 1 24 45 54 82 48 46 10
Chats to review 0 0 0 1 0 3 5 0 6 4 3 1
Average chats per day 1.67 1.33 2.5 1.4 1.75 8.56 13.12 15.65 17.77 16.64 17.96 22.53
Median initial response time
(seconds) 16 13 30 18 24 28 25 24 19 19 18 23
Offline messages 23 54 46 36 38 55 59 55 64 55 73 64
Ratings (5 stars max)
Overall chat 4.5 5 4.75 3 5 4.85 4.63 4.78 4.75 4.8 4.86 4.55
Responsiveness 5 5 5 4 5 4.83 4.67 4.84 4.83 4.74 4.89 4.91
Knowledge 5 5 4.75 5 5 4.64 4.57 4.74 4.67 4.67 4.71 4.55
Friendliness 3 5 5 3 4 4.79 4.7 4.8 4.89 4.7 4.84 4.91

21. olark.com
Data Driven Decisions IRL
AcquireConvert.com -
Giles Adam Thomas of AcquireConvert:
Use the same language your customers use to increase conversions
Copy transcripts to a spreadsheet
Evaluate for commonly used words and or phrases
Insert those phrases into your landing page and CTA copy
"We saw a 176.33% conversion rate increase through this copy change alone."
SOURCE: https://blog.olark.com/how-to-use-transcripts-to-increase-conversion-rates-and-profits

22. olark.com
Data Driven Decisions IRL
Disruptive Advertising -
Jacob Baadsgaard of Disruptive Advertising:
Tested four 'unexpected' greeters on the homepage for their business, a PPC ad agency:
“What’s your best marriage advice?”
“If your AdWords account was an animal, what would it be?”
“Who’s your #1 competitor?”
“What are you struggling with right now?”
Winning test was _____________
Also tested on on a client who sells promo products for lanyards:
“How many lanyards are you looking for?”
“Hi! I’m Brad! Let me know if you need any help!”
“What event do you need the lanyards for?”
The Greeter that won by a landslide was “How many lanyards are you looking for?”
Using the Olark Greeter in clever ways on clients' landing pages increased conversion rates by as much
as 37 percent!
SOURCE: https://blog.olark.com/how-clever-greeters-increase-conversion-rates

23. olark.com
Want to learn more?
1. Pandas, scipy, scikit-learn (python libraries) are all well documented
2. reddit.com/r/datasets/ is a fun place to find datasets to teach yourself things
3. UMcoursera course starting soon: www.coursera.org/learn/python-data-analysis
4. Books:
a. https://www.amazon.com/Python-Machine-Learning-Sebastian-
Raschka/dp/1783555130/
b. https://www.amazon.com/Data-Science-Business-Data-Analytic-
Thinking/dp/1449361323/

24. olark.com
One last thought on data people...
1) Domain Expert
You need someone with domain expertise on you data team. The particular domain obviously depends your data teams role. Here, it's mostly company
health and customer behavior. If your data team is focused more on your product (e.g. fitness monitor company has data team to do something interesting
with your heartrate time series) then people you hire should have some exposure to that or a similar domain and exhibit the ability to learn about it
independently.
2) Researcher
Design experiments, execute them, make sure the results are statistically valid. Compromise between business goals and timelines and confidence (not
something you have to do in academic science). Doubt everything.
3) Computer Scientist
Making computers do stuff. Know how to optimize queries, use efficient computation patterns. Maintain production code, design and or use complex
algorithms. If you're team does a lot of ML, this expertise can run very deep (not the case here). A person should be able to read about how an algorithm
works, understand how to use it, it's limitations, red flags, etc. It's a bad sign if someone just williy-nilly applies techniques without checking that the premise
of the technique holds, and assumes all results are valid.
4) System Administrator
I manage our data warehouse. Many tasks are similar to that of a database administrator. I need to make sure our data is secure, the databases are
optimized for the types of queries we're doing, they're up, the data is correct, design the schema, etc. Security is pretty easy, since it's not customer facing
and there are relatively few users. This may not be the case at larger companies. Eng ops also helps and hand holds this a bit, same with keeping the
database up and running. Verifying and cleaning the data is awful and time consuming. "data munging" is the thing that every data person spends more time
doing than they would like.
SOURCE: https://www.aapor.org/AAPOR_Main/media/Task-Force-
Reports/BigDataTaskForceReport_FINAL_2_12_15_b.pdf

25. olark.com
Thank you! Questions?
Karl@Olark.com
Olark Data Scientist: Bens@olark.com
@karlpawlewicz | @Olark