You’ve done the button color A/B test, you’ve optimized your landing pages for better conversion. What next? At B2B organizations large and small, there is still tremendous potential for experimentation to drive innovation and growth. Learn how Brion’s growth team enables rapid iteration across a variety of different domains, teams, and organizations within Cisco. With an organization of 70,000 employees and many distributed divisions, enabling experimentation can be a complex initiative. Learn the framework for upleveling from random testing to explicit strategy to position your org for triple digit growth.

1. An Experimentation Framework
Brion Hickey
Growth, Cisco
How to Position for Triple Digit Growth
Hudson Arnold
Senior Strategy Consultant
Optimizely

2. Operationalizing Experimentation
for Enterprises
Mass adoption of experimentation and the resulting alignment gap

3. “The connected economy continues its path of creative destruction, so much so that it is
changing the nature of competition itself. Nearly all companies surveyed by Gartner (89%)
believe that customer experience will be their primary basis for competition ... For
example, consumer product companies that have relied on developing new features,
improved customer service and product innovation to drive growth now see a future
where competitive advantage will be based on the customer experience.
“In many industries, hypercompetition has eroded traditional product and service
advantages, making customer experience the new competitive battlefield. With an
overabundance of alternatives and ubiquitous access to pricing and product information,
consumers have little reason to remain loyal to a particular brand.”
Historic Changes Put Customer Experience
At The Forefront of Competition
Jake Sorofman, Research Director, Gartner

4. Digital Leaders Have Already Fully
Operationalized Experimentation
Jake Sorofman, Research Director, Gartner

5. “Most large organizations embrace the idea of
invention, but are not willing to suffer the string
of failed experiments necessary to get there.
Outsized returns often come from betting against
conventional wisdom, and conventional wisdom is
usually right.
Given a ten percent chance of a 100 times payoff, you
should take that bet every time. But you’re still going to
be wrong nine times out of ten.”
—Jeff Bezos
2015 Shareholder Letter

6. “Every product change
Netflix considers goes
through a rigorous A/B
testing process before
becoming the default
user experience.”

8. The only constant is change, and the rate of change is increasing.
The only way to keep up is to be constantly experimenting and
innovating. Hyper-growth and experimentation are very closely
linked.
The philosophy is you have to build your company to be a
big experimental engine and it has to start right at the
beginning.
It’s not easy to just “retrofit” that engine in later—it’s a cultural shift.
You have to be in the mindset of constantly testing crazy ideas, new
business models, new products, and new processes.
—Jeff Holden
Chief Product Officer

9. Digital Leaders Set the Standard for
Experimentation ...
… Magnifying the ‘Alignment Gap’

10. Mind the Alignment Gap
VS
Leading Experimentation Organizations Lagging Experimentation Organizations
● Teams work on independent optimization roadmaps
● Central groups leverage dedicated technology to collect
data, conduct experiments and surface insights
● Strategy and metrics for digital performance directly
linked to company success and widely circulated
● Collective expertise creates virtuous cycle of learning and
best practices
● Experimentation treated as series of tactics to ‘improve
conversion rate’
● Skill sets and tools divided across functional silos or (if
available at all), while individuals or small teams work on
experiments with limited support
● Goals and processes poorly formalized, lack executive
sponsorship
● Consumer intentions and digital behavior poorly understood

11. “Most [Customer Intelligence] professionals that Forrester interviewed
estimated that 30% or fewer digital customer interactions are used in
some form of online testing, while only 61% have had an active online testing
program for two or more years.†”
Optimize Customer Experiences With Online Testing And Continuous Optimization, January 26, 2016
James McCormick
Principal Analyst
Consumer Insights
This ‘Alignment Gap’ is an
Opportunity to Leapfrog Competition

12. An Experimental Framework To
Position for Triple Digit Growth
As a strategic focus to ensure impact and ROI
from experimentation for your company
Brion Hickey
Growth, Cisco

13. About Me
Over 120 companies worldwide
Then Now

14. I have cat

15. Growth at Cisco
Support a diverse organization
Empower with best practices
Deliver ongoing ROI
70,000+ employees across a
variety of organizations.
How to get started or accelerate
experimentation depending on
the needs of the organization.
Help Cisco employees focus
on the right metrics to drive
success.

16. 1. Frequency of A/B Tests Launched
2. Reach of A/B Testing Capability
3. Complexity of A/B Tests

17. 1. Frequency of A/B Tests Launched

18. 1. Frequency over Ideas
Challenge Status Quo
● How many test has your team launch
last week, month year?
● Be honest with that number and
measure: 0, 5, 10, 30?
● What would it take to double the number
of experiments?
● Identify the bottlenecks and fix?
Designers, front dev, QA?

19. 1. Frequency over Ideas
Reduce Process
No committees
No brainstorming
No meetings
Just go!

20. 1. Frequency over Ideas
Reduce Process
“Don’t take it
personally.”
“Relax! It’s just a test.”
“Conflicting ideas? Let’s
test them :)”

21. 1. Frequency over Ideas
Why?

22. Recipe A Recipe B
66% increase in leads

23. 1. Frequency over Ideas
More experiments = more knowledge
● Share
● Create your own Summit - We
did :D
● Document findings
● Create easily accessible
knowledge base
Example: Understand what your users are
responsive to relative to your category.
Expose and exploit

24. 2. Reach of A/B Testing Capability

25. 2. Reach of A/B Testing Capability
Landing
page
Conversion
● This is our typical thought process.
● What do we mean by reach?
● Is anything missing?

26. 2. Reach of A/B Testing Capability
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7
● Is every aspect of your funnel measure?
● Are you experimenting at each step?
● What else do we mean by reach?

27. 2. Reach of A/B Testing Capability
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7
Landing
page
(Mobile)
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7

28. 2. Reach of A/B Testing Capability
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7
Landing
page
(Mobile)
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7
Landing
page
(Localized)
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7

29. 2. Experiment Reach Translates to Compound Gains
More than just landing
pages and leads
Map out the entire customer
journey
Reach positions you for greater
likelihood of compound gains
Illustrate friction points
Focus on “micro yeses”
Homepage Customers Page
Pricing Page Contact Sales Form

30. 2. Reach of A/B Testing Capability
Why?

31. Reach Translates to Compound Gains
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7
● No reason to boil the ocean
● Focus on base hits
● Wins compound

32. Product Page Example
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5
50% more visitors
down funnel

33. Product Page Example
Landing
page
ConversionStep 1 Step 2 Step 3 Step 4 Step 5
50% more visitors
down funnel
20% more
visitors down
funnel

34. 3. Complexity of A/B Tests

35. 3. Complexity is a Benefit in Optimization
Simple A/B testing yields
outstanding results
Increase in complexity provides
depth of understanding
Examples:
Multivariate testing
Behavioral targeting and personalization

36. Complexity Example
Landing page ConversionStep 1 Step 2 Step 3 Step 4 Step 5
50% more visitors
down funnel
20% more
visitors down
funnel6% more visitors
down funnel

37. Impact for Cisco
● Ability to move quickly and
iterate, make decisions with
data
● Generate a demonstrable
impact on company’s growth
KPIs
● Enable teams to experiment
at their pace while promoting
best practices

38. Takeaways
Optimize for Speed of Experiments
Who are you Impacting?
Strive for Complexity
Take a hard look at what’s
preventing you from moving faster.
Understand the reach of your
experiments and increase it.
Aim for more complex
experiments to increase
understanding.

39. Thank you!

40. Questions?

41. Appendix

42. Leading vs. Lagging
• Website & Apps are marketing and sales
channels, only recently focus of top
leadership, if at all
• Enfranchised teams work on independent
optimization roadmaps
TEAMS
EXPERIMENTATION ORGANIZATIONS
• Experimentation treated as series of tactics
to ‘improve conversion rate’
• Experimentation is deeply embedded into
company culture
EXPERIMENTATION
• Skill sets and tools divided across
functional silos or not available at all
• Central groups manage data collection and
analysis, empowers individuals with resources
and process
TOOL SETS
SKILL LEVELS • Lone individuals or small teams working
through design, production, and analysis
despite limited experience
• Dedicated, customized technology
conducts experiments and surfaces insights
PERFORMANCE • Goals and processes poorly formalized,
lack executive sponsorship
• Strategy and metrics for digital performance
directly linked to company success and
widely circulated
PRACTICES • Consumer intentions and digital behavior
poorly understood
• Collective expertise creates virtuous cycle of
learning and best practices

43. “When it comes to innovation, most managers must operate in a world where they lack
sufficient data to inform their decisions. Consequently, they often rely on their experience or
intuition. But ideas that are truly innovative—that is, those that can reshape industries—
typically go against the grain of executive experience and conventional wisdom.”
Why don’t more companies conduct rigorous tests of their risky overhauls and expensive
proposals? Because most organizations are reluctant to fund proper business experiments
and have considerable difficulty executing them. Although the process of experimentation
seems straightforward, it is surprisingly hard in practice.”
Experimentation is the Key to Unlocking Innovation
But it is Historically Underfunded by Businesses
—Stefan Thomke and Jim Manzi