Chatbots and Digital Assistant Analysis

1. David Wright
David.Wright@kantar.com
Digital Assistants –
a brand’s best friend?

2. What are Digital Assistants?
Personal Assistants Customer Service Assistants
Chatbots

3. Messenger “Chatbot” Global Reach
Messengers are where that conversation is at.
WeChat, China - 700 million users
Facebook Messenger – 1 billion users
KIK – 300 million users
Line, Japan – 215 million users

4. “A brand’s best friend?” - The experiments
Usability Test
How do people interact with a DA?
Behavioural Analysis
SIRI + Facebook Messenger interaction
patterns
Facebook Brand
Page Analysis
What do direct brand to consumer
conversations look like now?
Messenger ‘Chatbot’
Prototype and Analysis
1 2
3 4

5. 1
Location based questions
“If I ask for a baked chicken recipe, or how to remove a stain, why not link
straight to a 'how to' video on YouTube from a reputable brand?”
“I need to adjust the way I talk, so Google understands me".
Weather and Time Questions Information Searches
“I like it when Siri replies to me (via voice), not when it gives me a list of tiny
links"
“I like it when there is a photo in the result”
Usability Test

6. Usability test summary
People set the conversational bar
very high
The knowledge base is controlled by Apple or Google.
The brand is not included in the conversation.
These assistants are not a brand’s best friend.
For brands though…
“This was the first time I used SIRI in two
years. Wow it’s got really good. When I first
used it didn’t understand me. I didn’t use it
again."
1

7. Behavioural data analysis
Word frequency for mobile searches that contain “How”
Are device makers now
ceding the customer
relationship to search
engines?
TV
FIX
MONEY
WORK
REMOVE
CAR
ANDROID
PLAY
PHONE
FREE
HAIR
IPHONE
COOK
RESET
APP
XBOX
CARD
WATER
CLEAN
UNLOCK
GALAXY
CALORIES
BABY
DOG
HOME
INSTALL
ONLINE
BECOME
GOOGLE
FACEBOOK
OPEN
WORTH
BOX
GROW
MUSIC
OIL
CONNECT
MOVIES
PAY
WINDOWS
“How do I connect YouTube from
Phone to TV?”
“How do I edit my contacts on my
Samsung Galaxy?”
2

8. Behavioural data analysis
What to people do on SIRI?
70K SIRI SEARCHES 4K IMAGES TAGGED
2
Movies 18%
Music 1%
Nature 6%
Other 9%
Sports Faces 48%
Sports Logos 18%
Wolfram 8%
Maps 8%
Bing Search 63%

9. Behavioural data analysis
“Do Messenger Patterns indicate how people might want to interact?”
52%
18%
30-60 seconds
5%60-90 seconds
30-60 seconds
0-30 seconds
Individual
messenger
sessions are brief
and succinct.
2

10. Behavioural test summary
Brands need to be part of the digital
conversation in their domain space
Are succinct as more than half of individual messenger
interactions are less than 30 seconds.
Are often a Q & A hunt for information.
Images can bring this information to life.
Digital conversations:
2

11. Facebook brand page analysis
“What do brand consumer digital interactions look like now?”
u = user (the person who first posted to the page and started a conversation chain)
b = the brand
0 = other users have decided to comment within the conversation chain.
3

12. Facebook Brand Page Analysis
26% of customer posts
were not answered
22% were responded to
and the conversation
was complete
3

13. A large and complex
long tail of
interactions.
For the 13K page
posts there were 942
different chain types.
For 10% of multi-chain
conversations the
brand was not present.
Facebook brand page analysis
3

14. Facebook brand page analysis
Tesco had the
smallest % of
unanswered posts
and the highest
chain variation.
3
0
10
20
30
40
50
60
0
100
200
300
400
500
600
700
800
900
1000
Boots Lidle Aldi UK Tesco Marks and
Spencer
% Unanswered
Chain Types

15. Facebook page test summary
Public Facebook posts are
likely the tip of the iceberg.
Open conversation ‘chains’ can be complex.
Facebook brand page conversations are public.
In KLM’s case 7 times more messages across the private channel.
Brands are likely to prefer direct/private digital communication.
For brands though…
3

16. Meet “Kit” the chatbot
Structured Survey
Open “AI” chat
4
Tested with 150
Kantar Employees

17. Messenger “Chatbot” prototype and analysis
Finally we can participate!
“
4

18. The good
A simple “structured” survey with elements including
“buttons” and “quick replies” worked!
Say “Hi” to Kit!
4

19. The bad - aka attempts at open conversations
0
10000
20000
30000
40000
50000
60000
70000
80000
1 3 5 7 9 1113151719212325272931333537394143454749515355575961636567
Total Duration of Chat (seconds)
Majority of interactions 1-15 Minutes.
Conversation ID
Seconds
Conversations that last 16 hours
Conversations that
span 1-2 hours
Although individual interactions are
short and sweet a user pause and pick
up the conversation at any time.
It was easy to get lost in open chat and
forget the purpose of the interaction.
52% of posts to the bot in “open
conversation mode” were one word
only.
4

20. The bad - aka too smarty pants too quickly
”What do you do to treat it?”
“I have a broken ankle”.
“I am wearing a boot.”
“Has it been a tough few days?”
“Yes.”
“Ok thanks. Please remember to say Hi again!”
“Do you not feel sorry for me?””
4

21. 4
AI tools – initially subtle and in the background

22. A brand’s best friend? Key conclusions
Chatbots combined with vast Messenger networks offer an unprecedented
opportunity for brands to connect with customers at scale.
 However chatbot creation is not straight forward.
Customers expect brief and accurate interactions.
 However direct brand to consumer conversations can be complicated and
nuanced.
Early adopters and influencers will try to stump an AI enabled chatbot which may
create a PR risk.
DAs/chatbots have the potential to become a brand’s best friend.
In the short term, due to the complexities of an AI managing open conversation, it is likely the brand
bot experiences will start simply and rely heavily on buttons and other structured UI elements.