17. 17
Scenario 1
Hours per week 200
Hours per experiment 40
# experiments p/w 5
# experiments p/y 260
% winners 25%
# winners 65
Value per winner €100K
Revenue uplift €6,5M
18. 18
Scenario 1 2
Hours per week 200 200
Hours per experiment 40 25
# experiments p/w 5 8
# experiments p/y 260 416
% winners 25% 20%
# winners 65 83
Value per winner €100K €100K
Revenue uplift €6,5M €8,3M
Extra revenue € 1,8M
Lower chance, increase frequentie
19. Success= chance *frequency
• Does the design have to be pixel perfect?
• Does the experiment need beautiful code?
• Do you need hours of data mining?
• Do you really need to attend that meeting, or use that time to setup an experiment?
• Etc.
24. First success
24
Chance
Frequency
Google Optimize Free
HTML, CSS, Js course €10,-
Google Analytics Free
Scientific research Free
Website of competition Free
Information from customer support Free
1,5 month Hotjar €30,-
CRO/ optimization course €10 or educational budget
Total €50,-
Question: Who has an experiment driven culture?
I dare to say NU.nl has an experiment driven culture. We can still make a lot of improvements, but the basis, the culture is definitely there.
Ruben – manager CO & UX Sanoma at the central data & performance department. NU, CRO & U team
Free time, own business for courses in CRO and setting up experiments, writing book on psychology and websites and help charity
For those who dont know Sanoma, these are a number of famous brands.
In the next 20 minutes tell something about how an experiment driven culture started at Sanoma ad NU, how to get the ball rolling and how we are organized today.
An experiment driven culture starts with the first proof of success. Proof that experimentation contributes to the development of an organization. Without the first proof of success, companies get stuck on a few tests a month or even a year. This is how it started at NU and Sanoma.
A few years ago there was one manager who hired a CRO specialist at NU en he had rapid success. NU is a brand of Sanoma which gets noticed, and so did the success and proof that experimentation works. From there on, experimentation became bigger and bigger at Sanoma.
But what is success, what does it look like, and how do you get to the first success? How do you get the ball roling, and keep roling, without hitting obstacles?
I belief that realising the first success is of great importance, and not all companies realize this.
Recently report CXL and VWO about status of CO world wide. And that really proofs getting to first success is not that easy
Two important conclusions…
With less than 5 tests -> no cultural shit. This is not enough, you won’t get enough proof.
Especially realize in most cases a team or at least one dedicated person responsible
This is literally stated in the research
Somewhere wrong choice. But we do have lots of traffic and a really fun team to work with. But lets stick to the first point, where companies dont test enough
First success and success of experimentation -> simple formula
Success – winning experiments
Change to get success – spend as much time as you want. Perfect code, perfect design, research
And frequency
Higher chance & higher frequency is more success
For those who dont know about experimentation
For those who dont know about experimentation
First success and success of experimentation -> simple formula
Success – winning experiments
Change to get success – spend as much time as you want. Perfect code, perfect design, research
And frequency
Higher chance & higher frequency is more success
Must sound very straigh forward, but by being aware of this, we will start spending our time differently and we are more aware of success.
For instance: do you really need to attent meeting or set up an experiment
QA at NU
Inspiration section vtwonen
Tom van den Berg – online dialogue
Bit less research, code less perfect, analysis afwards bit shorter. Bit lower chance, more time for experiments
Of course also otherway around. No success = spend more time on chance
If you take this into account during every day work, much more succesful tests
But I think you can get even more out of this, which will lead to even more success.
Happens with the right mindset. Seen in GH blogs, speak about growth mindset
Growth mindset is ‘I dont fail, I learn’. When others successful = not jealous, inspire. I want to challenge myself
We really need a growth mindset, and see it as a fun challenge. Research shows we have many challenges / frustrations.
Another one I often here is IT capacity
Are you creative with bottlenecks?
high chance low frequency due to IT capacity. Lower fr for two weeks and learn coding yourself
Of course not most fun thing to do, but helps get started and build business case for dedicated front-end developer
We can elaborate on this.
For €50,- start experimenting and not dependent on IT or budgets
This is how we kickstart a culture and get the ball roling
By being aware of our success, and being creative with this regarding bottlenecks, I truly belief there are many possibilities to start an experiment driven culture
As Tom said at DiDO 27…
Stop talking and start testing
Make sure that with the right chance, as many experiments as possible, gather proof and start rolling that ball
That is how it starts, and we are still working with this mindset. So how do we do this today?
Add NU we test almost everything, not just design, but also algoritms, AI, features and new businesses
Not just A/B testing but also in our own usability lab. UX lab is extremely useful to get valuable insights, but also great for marketing and getting colleagues to think more from a user perspective
That doesnt mean it is going well for every brand at Sanoma, but we are trying our best. Two redesign gone wrong – no testing. Step by step we are learning
How? We have a centralized D&P team. See as inhouse agency, with 20 experts helping the top online brands, including NU. Nu gets noticed, what works there can be tested or rolled out to other brands
Work together with product teams, performance teams designers and developers, to improve most important KPIs
Wekelijkse meetings, waarbij we testen en inzichten bespreken. Brainstorms voor kleinere titels
In EE houden we alles bij en documenteren we alles
Someone in the team responsible for sharing knowlegde with the whole organization: Newletters, cases, workshops, presentations and online lectures at Sanoma Academy
Consultation hour every Tuesday
Several people are responsible for setting up experiments. Besides CO and redesign, also personalization AI video.
Also web analists and development
Weekly Optimization meeting
Occasionally invite others, like chefs
Test results, upcoming tests and innovation newletter
Everyone is encouraged to contribute and come up with ideas.
And of course we celebrate our failures big, with campaign and glitters.
In other words, at NU there is a real experiment driven culture. But is all started with one manager who hired one CRO specialist who got to that first success. The first proof that experiments contribute to the success of the organization.
And I belief when we are aware of obtaining this success. With the right mindset, we get the ball rolling and in time, create an experiment driven culture.
