TO INFORM,
ENGAGE AND
EMPOWER
THE WORLD

CONSUMER
EXPERIENCE TEAM
PRODUCT ENGINEERING
CUSTOMER CAPABILITY TEAMS

CONSUMER
SCIENCE TEAM
IMPROVING OUR DECISIONS
Objective
Facts
What
Confirms
Your
Beliefs
What you see

OUR
EXPERIMENTATIO
N JOURNEY

X6
EARLY DAYS
•2014 Redesign Beta
- Simple redirect test
•Headline testing POC
- Optimizely
- Visual Revenue
•Optimizing template monetization
•Homepage zone arrangements
•Ad Hoc experiments
WE DIDN’T KNOW MUCH

X7
AWARENESS TURNING POINT
1.30%
-29.26%
0.80%
-47.30%
2.99%
-0.97% 0.26%
1.02%
Var.1: Replace “TOP STORIES” - Desktop Var.2: Replace “News and Buzz”- Desktop Var.3: Replace “More from CNN” - Desktop Mobile Web – Replace More from CNN
-20.38%
-1%
+34%
Overall Footer:
-61.43%
- 86%
+24%
+46%

X8
CHALLENGES WE FACED
•News cycle
- Audience mix changes with big news events
•Content matters and changes quickly
•Optimal layout depends on content
•Advertising business model centers on attention
- Proxy metrics to habit are hard to identify
•Every single interaction represents a micro-transaction
•No paid subscription means no big payback conversions
EASIER SAID THAN DONE

EDITORIAL TESTING
Leverage existing toolkit
Test programming
approaches
FOCUS
Quick turnaround
Large reach
Focused analysis
TEST TYPES

X10
•Newsletter acquisition
- Color and copy variations
- Segmented targeting
- Size variations
- Activation triggers
- Registration campaigns
- Custom code
- Color and copy variations
ACQUISITION TESTING
CRO
Lifecycle management
FOCUS
Quick turnaround
Large reach
Automated analysis
TEST TYPES

REVENUE OPTIMIZATION
Revenue modules
Templates
Ad Footprint
FOCUS
Longitudinal
Small reach
Deep analysis
TEST TYPES
•Optimize template monetization
- Custom code
- Placement on template
- Full downstream analysis
- Revenue impact analysis

PRODUCT TESTING
FOCUS
New Product features
UX changes
TEST TYPES
Longitudinal
Small reach
Deep analysis

HOW WE GOT
HERE

X14
ORGANIZATIONAL EVOLUTION
•Creation of Audience Development
• Focus on growth and habit
• Lifecycle management
•Consumer Science approach
- Road towards COE
•Product’s adoption of cross functional
team approach
•Build up of UX Research team
•Scaling the organization
CHANGING HOW TEAMS INTERACT

X15
CULTURAL EVOLUTION
• Discovery driven Product Development
• Right people, right problems
• Hiring the right team
• Audience segmentation awareness
• Cold, Casual, Core
• Audience metrics
• Building habit
• Measuring satisfaction
• Retention Proxy metrics
GROWTH OF PRODUCT AND ENGINEERING

X16
PROCESS
• Starting with what we know
• Data / observations / feedback
• Earlier involvement of
Consumer Science team
• Relentless focus on refining the
process
EMPHASIS ON TEST & LEARN
DATA
QUAL
TESTING
RELEAS
E
Validate
and repeat
QUANT
TESTING
“Test
against
Data”
“Test
against few
volunteers”
“Test at
scale”
Fidelity & effort

X17
TECHNOLOGY
• Optimizely Web
• Optimizely Full Stack
• Improved event instrumentation
• List Vars on Adobe
• Custom metrics on Optimizely
• Program documentation and
measurement
• Snowflake, Databricks, Tableau & Looker

X18
ENGINEERING
EXPERIMENTATION WITH CONFIDENCE

X19
ENGINEERING
RUTHLESS FOCUS

X20
ENGINEERING
ROLLING “CANARY” RELEASE

X21
7
APP TEST: “FOR YOU” PERSONALIZED FEED
Test VariationControl Preview

X22
5
APP TESTING
•Important to monitor adoption
rates before launching
experiment
•Launch to larger percentage of
app audience to reach
sufficient sample size faster
•Simultaneously test on mobile
web to help inform app testing
WHAT WE’VE LEARNED

X23
8
NEXT STEPS
•Continue optimizing iOS mobile app test
process
•Launch Android mobile app tests
•Evaluate OTT testing requirements
•Enable server-side testing on both web
and apps

X24
8
LONG TERM
•Dynamic experience testing (Server-side)
•Automated and faster analysis
•Complex editorial testing
•Cross platform experiments
•Continued evolution of user experience

X25
•Trustworthiness is fundamental
•Move fast and methodically
•Get involved from inception
•Leverage qualitative to refine
solutions before coding
•Define success before testing
•Communicate broadly
•Focus on process
KEY
LEARNINGS