David Raine and Travers Korobanik from Walmart.ca share their experience using UserTesting to improve their website optimization program.

1. Customer Insights at Scale
Uncovering the “Why”
David Raine and Travers Korobanik

2. UserTesting’s on demand customer
insights platform enables companies
to create powerful products and
exceptional customer experiences.
#UTwebinar

3. We partner with thousands of customers
#UTwebinar

4. Across all platforms, at all stages of development
Watch customers interact with:
DESKTOP SMARTPHONE TABLET
WEBSITES APPS &
PROGRAMS
PROTOTYPES &
WIREFRAMES
SURVEYS PHYSICAL
PRODUCTS
Anywhere people interact with your brand:
AT HOME AT A STORE ON THE GO
On any platform or device:
#UTwebinar

5. 8
David Raine
Manager, Site Optimization
Travers Korobanik
Associate Manager, Experimentation and
Research

6. Customer insights at scale:
Uncovering the ‘why’
How UserTesting and regular customer listening sessions transformed
our testing program
May 2017

7. Introductions
David Raine
Manager, Site Optimization
Conversion Optimization, SEO, Customer Journey
First Job:
Dominion Save-a-Centre, Cashier
You May Have Seen My Work at:
Credit Canada
Telus
Koodo Mobile
David
Raine
Walmart Canada – 20177

8. Introductions
Travers Korobanik
Associate Manager,
Experimentation & Research
A/B Testing, User Testing
First Job: Mascotto’s Marina, Gas attendant
Favourite Quote: “Better to have it and not need
than need it and not have it”
You May Have Seen My Work at:
Sportsnet.ca
Toyotacertified.com
Travers
Koroba
nik
Walmart Canada – 20178

9. Introductions
Karen Connor-Froude
Sr. Graphic Designer, Studio
Christina Ramsamujh
Sr. Deployment Specialist, Deployment Team
Nick van Vlymen
Web Developer, Site Health
Tina Lin
Deployment Specialist, Deployment Team & QA
Walmart Canada – 20179
Extended Scrum team

10. Serving customers in Canada
since 1994
Be lean
Walmart Canada – 201710
About Walmart Canada

11. Walmart Canada – 201711
About Walmart Canada
Serving customers in Canada
since 1994
Now with over 400 stores
Nationwide

12. Be lean
Walmart Canada – 201712
About Walmart Canada
Serving customers in Canada
since 1994
Now with over 400 stores
Nationwide
And employing 95,000 associates

13. Over 16 different departments
Be lean
Walmart Canada – 201713
About Walmart.ca

14. Be lean
Walmart Canada – 201714
About Walmart Canada
Over 16 different departments
….Even fresh groceries!

15. Be lean
Walmart Canada – 201715
About Walmart Canada
Over 16 different departments
….Even fresh groceries!
Visited by over 600,000
Canadians daily

16. Be lean
Walmart Canada – 201716
About Walmart Canada
Over 16 different departments
….Even fresh groceries!
Visited by over 600,000
Canadians daily
Ps….we’re hiring!
Careers.walmart.ca

17. Walmart Canada – 201717
Let’s go back in time….

18. Walmart Canada – 201718
We were selling a lot of these.

19. Walmart Canada – 201719
And these.

20. Walmart Canada – 201720
And we were running a whole lot of tests.

21. Walmart Canada – 201721
And we were running a whole lot of tests.

22. Walmart Canada – 201722
And we were running a whole lot of tests.

23. Walmart Canada – 201723
And we were running a whole lot of tests.

24. Walmart Canada – 201724
And we were running a whole lot of tests.

25. Walmart Canada – 201725
And we were running a whole lot of tests.

26. Walmart Canada – 201726
And we were running a whole lot of tests.

27. Walmart Canada – 201727
We were finding leaky conversion funnels.

28. Walmart Canada – 201728
We were finding leaky conversion funnels.
We were doubling our test output.

29. Walmart Canada – 201729
We were finding leaky conversion funnels.
We were doubling our test output.
We were zen with our testing platform.

30. Walmart Canada – 201730
Things were great!

31. Walmart Canada – 201731
Until they weren’t.

32. Walmart Canada – 201732
We were launching tons of tests, but we were
scattershot.

33. Walmart Canada – 201733
We were pushing incremental site enhancements,
but unsure where to go next.

34. Walmart Canada – 201734
Our site templates were becoming a Frankenstein of
individually winning tests.

35. Walmart Canada – 201735
We were burning out our team.

36. Walmart Canada – 201736
Truly listening to the customer, changed that.
+

37. Walmart Canada – 201737
It wasn’t easy. User research was seen as inferior to
‘real data’.

38. Walmart Canada – 201738
But inspiration can come from unexpected places.

39. Walmart Canada – 201739
Source: https://hbr.org/2016/11/the-brands-that-make-customers-feel-respected

40. Walmart Canada – 201740
Rob Walton
Walmart Shareholders Meeting
2014
I brought something with me today. I
recently came across one of Dad's
"Chairman's letters." He was always
communicating, always sharing ideas. This
one is from 1985, almost 30 years ago.
“Customers are not dependent on us. We
are dependent on them. Customers are
people who bring us their wants. It is our
job to fill those wants. Customers are not
cold statistics. They are flesh and blood
human beings with emotions and
feelings like our own.”

41. Walmart Canada – 201741
Customer research is data.

42. Walmart Canada – 201742
Customer research adds context to transactional
data.

43. Walmart Canada – 201743
But the real proof point?

44. Walmart Canada – 201744
Launching more successful tests.

45. Walmart Canada – 201745
Here’s how we did it.

46. 01
How
How to structure a
process to
conceptualize, plan and
launch A/B tests, even
with a part time team!
Walmart Canada – 201746
Key Takeaways
Why
How to leverage
usertesting.com to
generate better test
ideas, and share
insights when longer
tests would be time
prohibitive.
Help
How usertesting.com
has helped Walmart.ca
overcome other
common optimization
team challenges.
02 03

47. 01
Every Day Low Cost:
Everything we do, every day
should be contributing to our
customers saving money…so
they can live better.
Be lean
Walmart Canada – 201747
How To Do More With Less

48. Think Lean:
There are more A/B test ideas
than time to support, UserTesting
allows us to gain insights on test
ideas before we commit our
(precious) time to them.
Be lean
Walmart Canada – 201748
Ask Why, Why, Why!?
02

49. Improve, improve, improve
A test story is never really
complete. Once implemented
UserTesting allows us to monitor
the feature and how our
customers engage with it in the
‘wild’, this lets us build on the
idea or go back to the drawing
board.
Be lean
Walmart Canada – 201749
Fail Forward Faster…With Help
03

50. 01
Shared resources mean dedicated
time with each team
member…dedicated but not
sacred.
Be lean
Walmart Canada – 201750
How We Test: Resources

51. 01
Aligning with corporate
initiatives, marketing campaigns,
seasonal events or other
operational goals can provide a
drive amongst the team that
otherwise may have felt like
friction.
Be lean
Walmart Canada – 201751
How We Test: Planning

52. 01
Test within the team, what one
person likes, several others may
not.
Be lean
Walmart Canada – 201752
How We Test: Conceptualizing

53. 01
3-Week Sprints, with overlapping
planning & prioritization
meetings.
Be lean
Walmart Canada – 201753
How We Test: Process
Plan
Build
QA
WEEK 1 WEEK 2 WEEK 3

54. 01
Though we are a small team,
everyone (in the entire division)
has the same tools, access and
ability to review, monitor and
report on their own test stories.
Be lean
Walmart Canada – 201754
How We Test: Launch, Monitor & Report

55. 01
Walmart Canada – 201755
How We Test: With Help

56. Every person on every team has
an opinion, hypothesis or
assumption about how we should
do something.
Walmart Canada – 201756
Why We Test: (Too?) Many Options02

57. Decisions based on numbers
alone will only get us so far.
Walmart Canada – 201757
Why We Test: Answering The ‘Why’
“It is so frustrating having a white website, then
a white background and then the girl’s wearing
a white top..”
02

58. “A picture paints 1000 words…but
a UserTesting video can be worth
its weight in gold (that makes
sense, right?)”
- Travers Korobanik
Walmart Canada – 201758
Why We Test: Sharing Insights
“Why are the prices hidden - I don't
like/it's kind of dumb."
02

59. Walmart Canada – 201759
Why We Test: It’s In Our DNA
“You can't just keep doing what works one time,
everything around you is changing. To succeed,
stay out in front of change.”
02

60. Walmart Canada – 201760
Testing Challenges: Tech Stack0
3
Be lean
What we see may not be what
our customers are seeing…

61. Walmart Canada – 201761
Testing Challenges: Bandwidth0
3
Be lean
6 people serving over 600,000
customers a day.

62. Walmart Canada – 201762
Testing Challenges: Buy-in0
3
Be lean
Everyone is encouraged to ask
‘why, why, why’…even between
teams internally.

63. Walmart Canada – 201763
Testing Challenges: Futureproofing0
3
OMNI focus, larger business
initiatives, more customer-centric
focus…how can we keep up?
Be lean
?

64. 01
How
- Be Lean, be agile
- Utilize all data points
- Alignment across the
team
Walmart Canada – 201764
Key Takeaways
Why
- Listen to customers
- Share insights
- Believe in it
Help
- Tech
- Bandwidth
- Buy-in
- Partnership
02 03

65. Questions?