Fluxx-The Tested User event-011112-main presentation

Why this event?


The 5 ages of User Experience
1. Total ignorance
2. Blind obedience (The Jakob years)
3. User-Centred Design & Usability
4. Let’s go crazy – multi-touch is here!
5. Anything goes, so long as users say it’s ok
(AKA nobody gets fired for doing what customers ask you to do)

And most recently:
Let’s test everything. It will make our lives so much easier.
We will never have to make a decision again.


What is ‘User Testing’ anyway?

Has it evolved?
Have we evolved our understanding and
usage of it?
Is it always the right thing to do?


Designed a phone
that inspired a
generation

? Re-invented a
familiar household
object for a new era

Made the world feel
like a smaller place

Pioneered a focus
on user testing


Designed a phone
that inspired a
generation

Re-invented a
familiar household
object for a new era

Made the world feel
like a smaller place

Pioneered a focus
on user testing
Henry Dreyfuss, 1904-1972

“If I’d asked
people what they
wanted, they
would have said
faster horses”
Henry Ford


So our questions for tonight are:
• When should I place emphasis on customers’ opinions, and
when should I put down the shutters and just show them
when we’re ready?
• What sorts of means do we have at our disposal for
establishing what customers will and won’t do?
• What is the future of validating our approaches with
customers?


Rajiv Arjan Paul Dawson Gavin Gordon
City Fluxx Mintel UK
Interaction Lab


Importance of user
testing
Rajiv Arjan
City Interaction Lab


Importance of user testing
Raj Arjan, City Interaction Lab

Agenda
• Introduction of City Interaction Lab and Me
• User Centered design process
• Usability testing process
• Usability testing enhancements
• Benefits of user testing
• Questions

@rajarjan

About Me
• Managing the interaction lab since 2008.
• Responsible for;
– Business development
– Managing UX projects
– Supporting Academics/Students
– Promoting Interaction Lab and HCID
@rajarjan

City University London

• Courses Taught
– MSc Human Centred Systems
– MSc Creativity Innovation and Leadership
– Can be undertaken Part time/ Full time

@rajarjan

• Facility used for student
and commercial projects
in areas of;
– Usability
– Accessibility
– Interaction design
– User research
@rajarjan

User Centered Design

@rajarjan Source : kitdigital - http://www.kitd.com/what-we-do/services/design/

Usability Testing Inputs

Wireframe Live
Site

Powerpoint/Keyn
Paper Prototype
@rajarjan ote

Usability Testing Process

Morae Usability Testing Apps

Tobii Studio – eye tracking

Silverback - for MAC OS

@rajarjan

Usability Testing Outputs
• Usability problems
• User opinions
• Task completion times
• Other Metrics (eye tracking, mouse clicks,
key presses)

@rajarjan

Paying attention to what users do

@rajarjan

“If I had asked
people what they
wanted, they would
have said faster
horses.”
@rajarjan

Paying attention to how users react

@rajarjan


• Movement
• Skin Conductance
• Temperature
@rajarjan

Observations

Opinions Emotional
Response
@rajarjan

Take Away’s
• User Testing is always important
• Observing what users do is often more
important than what they say.
• The user is not the designer.
• Maximise the value of user testing by asking
the right questions.
• What are you trying to test? The interface or the
concept?

@rajarjan

Thank you
Rajiv Arjan
@rajarjan

Facts, not opinions
Paul Dawson
Fluxx Labs


Fluxx Labs is our methodical
approach to validating new product
and service propositions, using lean
startup techniques.

labs

Why?

In 15 years of designing and delivering digital products
big and small, we’ve seen a lot of great ideas and
realised a few things.

labs

1 : BIG BETS

Don’t always pay off


2 : There is no shortage of big
ideas in our client base

labs

3 : Some people are pretty
opinionated

And they often spell the end to a strong concept

labs

http://www.youtube.com/watch?v=E1IarxtP4aA

labs

insight idea feasibility

design &
manufacture distribution
development


d
i

Easy to do, and no risk Starting to invest time & money Hard to do, and a lot of risk
s
t
r
i
b
u
man t
ufac i
design & ture o
development n

feasibility

insight

Idea


The MVP is at the heart of Fluxx Labs
In ‘Lean Startup’ land, the fastest route to getting facts is to
launch a ‘minimum viable product’

Q: What is the absolute minimum we can do to see if people
actually do value our product?

labs

Dim and distant past Painful, but we got there…. Dragon’s Den

insight Bankrupt
factory People
bought
it
concept
Give it
manufacture to the
design & press
development distribution

Licensee Dodgy
feasibility liquidated catches


He believed in it so much
because he generated facts by
putting his product into the
hands of potential customers

The toughness of his journey
was testament to the fact that his
MVP could not be that minimal

labs

Fortunately…

We live in a world where you can make anything seem
real, to make the learnings as real as they can be.

We set Fluxx Labs up to generate evidence-based
learning, to help our clients get their ideas right and
make good decisions based on facts.

labs

Basic Insight
We find it really hard to find a whiteboard the
size we want and think that other people will
have the same problem.

labs

Insight leads to a value hypothesis that we
can begin to test

“We think that people will value a premium
product that makes and installs whiteboards to a
precise set of dimensions.”

labs

1. Make a site that
calculates the cost and
puts customers in touch
with us.
2. Get a phone!

If people value this
service as much as • Volume of people
we do. searching for it on Google
• Requests for pricing
• People who buy

labs

The first MVP
(2 days)

labs

The Lab’s Learnings

This is a viable, small, local
business.
It would be a good idea to make
the whiteboards erasable!
It is possible in a week to prove
out a digital & physical
business in a lean way.
labs

The High Street Bank
A real life case study

labs

Amongst others, we created one
experiment designed to test
something that had been debated
for ages at the bank

labs

It was a thorny subject.
It involved using customers’
personal data to do something we
thought they would like and value.

labs

But would they accept it, or would
they go nuts?
We put it live in a few weeks in an MVP to
several thousand real customers who didn’t
know it was just an experiment.

labs

But before we did…

labs

It was usability tested
It panned!
Only 24% of people said they would do it

labs

How many ACTUALLY did it in
real life?

96% of people just jumped right on in!

labs

“What people say they will do and
what they actually do is often
very different and can be greatly
affected by their level of
awareness of the task at hand”

Thank you, and see you in the Lab

Paul Dawson - @poleydee
embrace@fluxx.uk.com


Mintel’s approach to
research
Gavin Gordon
Mintel


A typical Mintel Oxygen report contains…

Market Drivers Size of the Market and Brands, Companies and
Forecast Innovations

Advertising and Distribution Trends Consumer Insight
Marketing Activity

67
mintel.com

Different elements of research

Desk research Trade research

Company reports
Trade input – formal and
Press releases
informal
Media reports
Legislative and regulatory Understanding a Industry events
developments market

Statistical forecasting Consumer research
Mintel and analyst
experience
Mintel’s market size Mintel’s own consumer
database research

Other data ONS, Treasury,
TGI data
Bank of England etc.

Qualitative research
68
mintel.com
Trade

Desk research

• Press reports – trade publications and national media

• Regulatory/legislative developments

• Gathering and analysing reliable and trusted data sources

• Typical sources include ONS, Ofcom, trade associations and other
industry data

• Company reports and press releases

69
mintel.com

Trade research

• Adds relevance and understanding, as well as a source of data

• Trade interviewing and attending industry events

• How is the market performing? Checking market size and our
forecasts.

• What challenges and issues is the market facing?

• Future trends, innovations, possible developments

• What consumer insight do industry players want?

70
mintel.com

Forecasting – Statistical modelling

Mintel’s own market size Analysis of relationships
5-year forecasts based on
database is supplemented between, Mintel market
an advanced statistical
by macro- and socio- sizes and a selection of
technique known as
economic data sourced key economic and
‘multivariate time series
from organisations such as demographic factors to
auto-regression’ using the
the Office for National identify those which have
statistical software
Statistics, HM Treasury and the most influence on the
package SPSS
the Bank of England. market.

71
mintel.com

Forecasting – Tackling uncertainty

The Mintel fan chart

The fan chart Equity Release Schemes, 2011-16
illustrates the
probability of
various outcomes
for the market
value/volume over
the next five years.

At a 95%
confidence interval,
we are saying that
95 out of 100 times,
the forecast will fall
within these outer
limits.

72
mintel.com

Forecasting - Qualitative insight

Forecast of games consoles and games software sales by value, 2012-17

Slimline,
cheaper
4,000
models
Sony/ Best case (£m)
3,792
3,500
Microsoft
next gen
3,000
Market value (£m)

Mintel forecast (£m)
2,811
2,500 Wii U
Worst case (£m)
2,000 2,074

1,500 (£m)
1,651

1,000
Confidence intervals
500
95%
90%
0
0
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 70%
Est. 50%
Actual Forecast

73
mintel.com

Consumer research planning

• Deciding what to cover – insight from desk and trade research

• Questionnaire design – collaborate with Research Services
Team

• Looking at ways to provide interesting analysis

• Qualitative consumer research

74
mintel.com

Qualboard:

1. To add context to rigorous quant data
2. To point to emergent trends
3. To provoke additional quant analysis

75
mintel.com

Consumer research planning

Online Face-to-face Specialist panel

• Internet users • Nationally • IFA research
representative
• Increasingly popular • HNW groups
form of research • Reaching those (Affluent Banking or
groups not engaged SIPPs)
• More detailed with the internet
exploration of issues • Youth surveys
• Annuities / Equity
Release Schemes

• Travel Agents

76
mintel.com

Bringing it all together

Desk research Consumer research

Company reports Mintel’s own consumer
Press releases research
Media reports
TGI data
Legislative and regulatory Understanding a
developments market Qualitative research

Statistical Forecasting Trade research
Mintel and analyst
Mintel’s market size experience
database Trade input – formal and
informal
Other data ONS, Treasury,
Bank of England etc. Industry events

77
mintel.com

Transforming information into actionable insight: What makes it work

• Understanding/evolving the role of syndicated research

• Faster

• More accessible

• More consultative

• More disruptive.

78
mintel.com

Updating our Model
“To ensure Mintel remains timely”
Move away from the 2-year publishing cycle?
Modified approach to report writing and publishing?
Analyst updates (market data/company information/consumer)
Auto updates (background data and existing databases) “Found relevant report
easily but not
Consistency with US?
recent enough”

“There's a real lag between
some reports being updated”

“The reports were too out-of-date
- I wanted something more
current and relevant”

79
mintel.com

Communication
“Building relationships and tapping into client knowledge”

Making clients one of the first ports of call:
Modifications to the current system?
Developing the analyst’s role to become more than a picture on the website
Listening to clients on a regular, on-going basis
“More consultation
Pro-active development led by research. with clients
for their specific needs”

‘I didn’t realise I
could talk to the analysts’
– 46%

80
mintel.com

Consistent and Insightful
“Usage is high, but frequency is not high enough”

Driving frequent usage up:
Grounding Inspire in more ‘facts’ and making What’s Hot to be more than just a visual treat
We’re guilty of ‘hiding away’ insight (reports too long, too much text)
More flexible approach to report scheduling (hot topic reports)
Consistent approach across Oxygen (both within UK and UK/US).

“Keep the same level of detail
and analysis but just make it
clearer and easier to find
what you are looking for”
“More raw data “Opinions/viewpoints always
where possible. interesting/thought provoking”
Let me make decisions!”

81
mintel.com

Thinking about your needs, why do you typically use Mintel?

When I have been given a brief for a project 67.6

As evidence to back up our ideas 63.0

To prep for a client meeting/ presentations 51.5

For a strategy meeting 40.4

When I need a quick answer 30.2

When somebody recommends a report internally 21.2

When I cannot find data elsewhere 20.1

Other 3.1

82 Base: 646 Mintel Oxygen clients mintel.com

Thank you
Gavin Gordon
Mintel


The fluxx ‘Tested User’ Framework

Idea/Concept

Product/Service
Maturity Level

Partial/Fully
Developed

Is the User
Unaware Aware
aware of the test?



Will people find the What are people’s opinions,
Idea/Concept product/service of value? expectations and ideas?

Product/Service
Market
Maturity Level
Intelligence

How well does our
Partial/Fully Which is the best way to
user interface
implement this?
Developed support our product?

Is the User
Unaware Aware
aware of the test?



MVP/Lean Market Research
Idea/Concept e.g. Fluxx Labs companies

Product/Service Market Intelligence
Maturity Level e.g. Mintel

Partial/Fully Product Development Usability Testing
e.g. Fluxx e.g. City Interaction Lab
Developed

Is the User
Unaware Aware
aware of the test?


Where product is at an early concept stage, and users
are unaware of the test itself.

Questions Trying to Answer:
Will people find the product of value? Lab-like
When to use: validation
When there is a perceived barrier or
high degree of uncertainty about
willingness of consumers to adopt
Where the product paradigm is
substantially different from the norm
Technique(s) Used:
Minimum Viable Product (MVP)


An early stage concept, and users are aware they are
being asked for their opinions and input

What are people’s opinions, expectations and
ideas?
Opinions
When to use: & ideas
When there is a wide range of possible
concepts that you want to understand more
Where you want early stage insight into how
specific consumer groups behave and think
You want consumer ideas and input
Technique(s) Used:
Focus Groups, Market Research


Where product is at a later stage, but you don’t make
consumers aware that they are part of a test

Question Trying to Answer:
Which is the optimal way to implement
this?
When to use:
Where there are clearly defined
potential solutions with no strong
supporting evidence as to which is the A/B
best approach
& MVT
Technique(s) Used:
A/B Testing, Multi-variate testing (MVT)


Where product is well developed and people know they
are being asked to test it

How well does our user interface
support our product? Can we make it
better?
When to use:
When you are clear on your proposed
design solution and want to uncover any User
potential barriers to adoption
Technique(s) Used:
Testing
Usability Testing, prototype testing, eye
tracking

Intelligence and data on consumer activities, behaviours and
opinions in specific sectors, industries and markets

Is this the right thing to do?
Is there a market for what we want to do?
When to use:
When you need clarity around target Market
segments or market
To provide external evidence of key Intelligence
adoption assumptions
To inform and validate the business case
Technique(s) Used:
Surveys, Polls, Intelligence Gathering


Appendix
of common
Testing Techniques


Fluxx Rapid Start
• A multi-day multi-disciplinary hugely intensive exercise
unique to Fluxx, where business teams are taken from early
stage ideas to launchable products ready for circulating to
consumers
• Typically this is done in two days and key side effects include
– Rebuilding teams’ belief in their ability to deliver
– Promotes collaborative design and ways of working
– Incorporating the voice of the customer into the build process
– Most rapid way to get something into the hands of customers


Focus Groups
• Definition:
– A focus group is a form of qualitative research in which a group of
people are asked about their perceptions, opinions, beliefs, and
attitudes towards a product, service, concept, advertisement, idea, or
packaging. Questions are asked in an interactive group setting where
participants are free to talk with other group members.


Usability Testing
• Definition:
– Usability testing is a technique used in user-centered interaction
design to evaluate a product by testing it on users. This can be seen as
an irreplaceable usability practice, since it gives direct input on how
real users use the system.


Ethnographic Research
Ethnography is technically about observing a particular society or culture through a
passive participation in it, and without impacting on the society itself, and this is still
the strictest definition of it.

However, nowadays, it is more widely used to categorise techniques that are about
trying to understand how people live their lives. Typically this involves visiting people
in their home or work environments using a series of highly practical questions
designed to uncover what is happening and why, Their goal is to see people’s
behaviour on their terms, not in an artificial way.

The latter approach is often confused or used interfchangeably to mean what is
known as a ‘contextual inquiry’ or ‘contextual analysis’ , the definition of which is
observivng someone’s interactions with your product in their usual context or
surroundings.


A/B Testing
• A/B testing or split testing compares the effectiveness of two
versions of a product/service in order to discover which has
better response rate or better sales conversion rate
• Two alternate designs or approaches are randomly served to
people, and the resultant outcomes of those differently
served customer bases compared to each other, and a control
group.


Heuristic Evaluation
• A Heuristic Evaluation, or Usability Audit, is an evaluation of
an interface by one or more Human Factors experts.
Evaluators measure the usability, efficiency, and
effectiveness of the interface based on 10 usability
heuristics.
• There are many heuristics available, but a popular one was
created by Jakob Nielsen in 1994. Many argue that this
heuristic is now outdated by modern interaction techniques,
and takes no account of user attitudes.


‘Depths’
• Structured, one-on-one interviews help researchers learn
about users’ attitudes and beliefs surrounding a website or
application and specific tasks that the website or application
supports. In order to prevent the researcher from introducing
bias and to ensure that every participant is asked the same
questions, the researcher prepares a Discussion Guide,
which is a list of questions that will be asked of participants
in a particular order.


Card Sorting
• Card sorting is a useful tool to determine how users
categorize the information that will appear on a website. The
name “card sort” comes from the original exercise, which
used index cards with a word or phrase written on one side,
and in some cases, a definition or additional information on
the reverse side


Electroencephalography (EEG)
• EEG involves measuring electrical activity in different parts of
the brain in response to certain stimuli
• It can now used for the assessment of user experiences to
see how people react at an emotional level to stimulus or a
particular experience
• Detailed and specific design recommendations can be made
by recording reactions at different stages of interaction with
a product/service


Minimum Viable Product (MVP)
• Minimum Viable Product is a strategy used for fast and
quantitative market testing of a product or product feature
• The product is typically deployed to a subset of possible
customers, such as early adopters that are thought to be
more forgiving, more likely to give feedback, and able to
grasp a product vision from an early prototype or marketing
information
• This is the technique employed by the Fluxx Labs method


Multi-variate testing (MVT)
• Multivariate testing is a technique for testing hypotheses on
complex multi-variable systems, especially used in testing
market perceptions
• It is a process by which more than one component of a
product/service may be tested in a live environment
• When automated it can be very powerful in finding the
optimal combination of multiple component parts


Prototype Testing
• Prototypes can be tested for aspects like design flaws and
ease of use, two things that are critical if your product is
going to be a success
• You need to make sure everything works the way it should -
and that your customers can figure out how to make it work
too


Eye Tracking
• Eye tracking measures the point of gaze and eye movement
• It can detect the direction in which users are looking and the
duration for which they are focusing on a specific part of a
product e.g. web page
• This is helpful knowledge if you are trying to find out if users
are actively focusing on particular content on your page or if
they miss it all together


Quantitative Market Research
Market research is the systematic gathering and interpretation of information about
individuals or organisations using the statistical and analytical methods and techniques
of the applied sciences to gain insight or support decision making

For example:
• Mass participation surveys
– Usually aimed at specifically targeted segments. This type of research relies on volume of
participants to give its results significance. Questions usually have binary or multiple choice
answers.
• Omnibus Survey:
– A regular survey designed to determine what the general population thinks on a range of topics and
issues. Brands can pay for questions on this survey alongside the other questions. Thought to have
derived from the phrase “The Man on the Clapham Omnibus’ which was used as a euphemism for ‘the
common man’.


About Mintel
Leading global market intelligence company with more than 500 employees worldwide
Guide for leaders in marketing, product development and research for more than 35
years
Clients rely on our authoritative offering to drive strategic decisions

What we do
Provide fresh insight into your marketplace and across industries
Identify opportunities for profit and growth
Formulate insights into consumer behavior, product innovation and competitive
marketing strategies
Create high-quality data, evocative analysis and tailored consulting

Fluxx-The Tested User event-011112-main presentation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (15)

Semelhante a Fluxx-The Tested User event-011112-main presentation

Semelhante a Fluxx-The Tested User event-011112-main presentation (20)

Mais de Fluxx

Mais de Fluxx (6)

Fluxx-The Tested User event-011112-main presentation