What Can Be Learned From A One-Minute Online Survey

What Can Be Learned From a
One-Minute Online Survey
How people really feel about Amazon.com
(and what does this mean)?

An Illustrative Case Study Using

Foreword: Web teams need constant feedback
By Gerry McGovern

Living systems get constant feedback from their external environment. To truly
succeed, web teams need constant feedback from their customers.

You're a manager in a restaurant. It's raining. A customer walks in and almost
slips on the mat in front of the door. You're very busy at this stage, but you make
a mental note: "I must change that mat." About 15 minutes later another
customer comes in. She, too, almost slips on the mat. You rush up to her,
apologize profusely and then change the mat.

People are slipping on our websites right now but, because we don't see them
slip, we don't change the mat. I'm one of the biggest offenders. Over the years I
have left content and applications on my websites that had problems that I was
vaguely aware of, but they just didn't seem important enough to warrant any
action. Even when I became clearly aware of the issue I didn't react with enough
urgency.

Why was that? Why was I so complacent? I would like to think that if I was
running a restaurant I would have apologized to the customer and changed the
mat. Why don't I do that when it comes down to managing a website? I think a
core part of the problem is the lack of real feedback.

I'm not actually seeing the customer slip. I don't actually see real people use my
websites.

Customers are hugely impatient on the Web. When they slip, their first impulse is
to hit the Back button. Jared Spool wrote an excellent article in 2009 called the
"The $300 Million Button." In it he explained how the removal of a registration
button from a particular step in a purchase process resulted in a dramatic
improvement in sales.

The Web team had created the registration button so as to make it easier and
faster for regular customers to buy. But people absolutely hate registration. New
customers felt they would be spammed if they registered. One potential customer
summed up their feelings as follows: "I'm not here to enter into a relationship. I
just want to buy something."

The regular customers didn't feel much happier. "45% of all customers had
multiple registrations in the system, some as many as 10," Jared wrote. "We also
analyzed how many people requested passwords, to find out it reached about
160,000 per day."

The Web is so important today. And yet many of the web teams I deal with are
way down the management hierarchy. Intranet teams, in particular, tend to get
negligible resources. That needs to change because the reality is that the Web is
central to the present and future success of most organizations.

One of the ways we make that change happen is that we start developing much
better feedback mechanisms for our websites. At a most basic level, we must find
ways to regularly (weekly at minimum) observe our customers carry out top tasks
on our websites. That's how Jared Spool discovered there was a problem: by
watching customers trying to buy.

According to Wikipedia, "Living things are systems that tend to respond to
changes in their environment." Let us embrace our customer environment. Let us
observe and evolve. The rewards are very substantial.

Gerry is the founder and CEO of Customer Carewords. He is widely regarded as
the worldwide authority on increasing web satisfaction by managing customer
tasks. Gerry has spoken, written and consulted extensively on web content
management issues since 1994. His new book, The Stranger's Long Neck: How to
Deliver What Your Customers Really Want Online, was published in July 2010.
Read the first chapter of the book at http://www.gerrymcgovern.com/sln-ch1.htm

What Can Be Learned From a One-Minute Online Survey

Executive summary
In order to demonstrate what can be learned from a carefully-designed survey
that is very quick to complete, we asked a combination of consumers and Web
professionals to rate Amazon.com (chosen as it is very familiar to many people).
As expected, the site was rated very positively yet there was still much to learn.
The most compelling characteristic of the site for most visitors was the ratings
and reviews feature. But while ratings and reviews are a strength now, they
might also be under threat. The growth of social media has diminished the
advantage of ratings and reviews by strangers and Amazon is arguably no longer
at the leading edge in this respect. Other findings demonstrate that female
visitors like to ‘get things done quickly’ and are more troubled by the difficulty of
‘contacting a person’ while men value ‘complete information’ a little more.

Perhaps surprisingly, Web professionals complained more than consumers that
Amazon.com had a cluttered layout, hinting that expert opinions cannot be relied
on exclusively without the input of regular Web users.

4


Table of contents
Introduction..........................................................................................6
Key findings..........................................................................................7
Results.................................................................................................8
............................................................................................................8
Analysis................................................................................................9
Ratings, reviews and recommendations are popular but under threat. . .9
......................................................................................................11
People like Amazon’s search but there is no room for complacency ....11
Clutter and layout were problems, but not for all...............................12
Some people find it hard to contact a person.....................................13
Visual appearance not of overriding importance.................................15
Methodology.......................................................................................18

5


Introduction
Consumers have opinions on websites; they express them to friends and family,
not in the technical and analytical terms that professionals might use, but in quick
summary statements that capture the essence of the sites.

Typically, Web surveys ask consumers to rate sites in technical terms, but this is
expecting consumers to think the way we experts do, to tell us what and where
site issues exist.

. We need to simulate being present at hundreds of social gatherings and
eavesdropping on conversations about websites. The approach used in this
We instead need to collect
Amazon review, tried and trusted over many years, puts the onus on the analyst
what people feel strongly
to interpret consumer sentiment, and makes it easier for them to express their
about; what they would feelings. It was our goal to run through a illustrative example from start to finish
say over the barbecue or to show you the level of actionable information can be gleamed from this simple
at the water cooler and comparatively inexpensive method. You can be the judge of the success of
that objective.

This technique does not ask questions in company-centric language (e.g. ‘Please
rate the options that are available for you to navigate this website?’), but instead
simply asks people to express what they felt strongly about. This ensures that
their instinctive reactions are captured. It also helps that the survey was very
quick to complete — it genuinely took less than a minute and nobody abandoned
it.

The success of the survey depends of course on the results. You have your
opinion of Amazon, so how well do the findings match your expectations, but also
surprise and provide ‘food for thought’? Ultimately, what value does this survey
bring to website analysis?

This study was conducted by The Customer Respect Group using the Customer
Carewords methodology. It is far from perfect. We would have liked larger
participant numbers and we did invite web professionals with their different
preconceptions to get involved, which may have skewed overall results a little.
But a lot of data and analysis emerged from what was, superficially, a very simple
survey.

6


Key findings
A very clear trend emerged from the results – the top five attributes accounted
for over half of the overall impressions. The strong trend showed that the survey
successfully identified the aspects of Amazon.com that are most compelling. The
positive sentiments were dominant and the site recorded an extremely high rating
of 86 out of 100. The general rating for websites tested with this methodology is
typically in the 50-60 range.

The top five attributes were:

1. Site has ratings, reviews and recommendations
2. Helpful search results
3. Fast to do things
4. Simple layout/easy to read
5. Accurate information

An important consequence of the survey is that it highlights site strengths just as
much as weaknesses. So while ratings, reviews and recommendations are a
much-loved feature this strength can become a threat should competitors
neutralize their value.

Although the negative sentiments were in the minority, a significant minority is
not happy with some aspects of Amazon.com.

Even a successful site like Amazon can be improved. Given the volume of traffic,
a small improvement could have a significant effect. The three main areas of
concern were:

1. Cluttered layout
2. Hard to contact a person
3. The site was visually unattractive

7


Results
Participants came from a pool of demographically evenly-spread consumers
(approx 60% of the total) and web professionals. The graph below reflects the
importance of each of the 26 attributes in the survey. A red bar indicates a
negative sentiment.

8


Analysis
The data is a signpost and tells us where to look; the interpretation brings the
data to life and leads to actionable tasks and strategies to pursue. With websites
so complex, it is critical to garner consumer sentiment continuously to reflect
changing expectations as much as changes in functionality. When so many
companies focus on customer retention and growth, sentiment towards websites
is a critical metric that up to now has been difficult to harness.

Ratings, reviews and recommendations are popular but under
threat
84% of US customers prefer The presence of ratings, reviews and recommendations was the top-rated
the opinion of other customers attribute. It was once what differentiated Amazon.com but the landscape has
versus experts. changed. Most retailers have adopted, and even improved, reviews and for very
good commercial reasons:
—Marketing Sherpa 78% of Internet users rate recommendations as most credible form of
advertising (Neilsen)
84% of US customers prefer the opinion of other customers versus
experts (Marketing Sherpa)
96% of online retailers rank customer ratings and reviews as an effective
or very effective tactic for driving conversion (Forrester)
65% trust ‘friends’ recommendations while 33% trust company-supplied
recommendations (emarketer)
69% of consumers who read reviews then go on to share them with
family and friends (Deloitte)
Products that have reviews show a 35% increase in conversion rates
(Bazaarvoice)

One weakness of the Amazon implementation is that it does not allow for
recommendations by friends. We can read that hundreds of people rated this
product as 5-star, but it’s not immediately clear if they share our tastes or
preferences. A clear finding of various studies is that consumers want to hear
from “someone like me”.

9


Best Buy’s Facebook page, which has almost 1.2 million fans (compared to
Amazon’s 90,000) includes a ‘Shop + Share’ tab on which consumers can browse
products and traditional reviews. A key enhancement, however, is the ‘Ask
Friends’ button which allows fans to collect input on their decision from friends’.
This makes it very easy to get the opinions of trusted peers. Now, friends can
offer an opinion before we buy, in a manner similar to bringing a friend to a
bricks-and-mortar store.

There are many other examples of how ratings and reviews have evolved and
developed. Once considered a unique feature of Amazon, this could be a game-
changing feature and Amazon can’t afford to fall behind the innovation curve in
an area so ingrained in its users’ experience. If a key strength is perceived as
being inferior to systems in use elsewhere, there is a danger that Amazon might
lose some of its popularity.

10


Best Buy’s Facebook page brings reviews and ratings to a new level by allowing people to
ask their friends for advice on selected products

Best Buy has recruited 15
times more ‘fans’ on
Facebook than Amazon and
more likely to benefit from
recommendations from
friends

People like Amazon’s search but there is no room for
complacency
Good search results was considered to be the next most important attribute.
Search is sometimes considered to be ‘Plan B’ when navigating through menus
and links has not resulted in success, but in this case it seems unlikely. Very few
people chose ‘confusing menus and links’, so it seems that Amazon’s search is
actually particularly good – good enough to be a Plan A when arriving on the site.
Customers know that searching for that book or DVD will produce really accurate
and fast results.

Amazon search has customer-oriented features that can help them find what they
Amazon cannot afford need very quickly. Sub-categories on the left menu allow customers to filter
to neglect improving results if the default results are not quite right, ratings are visible, and related
searches are suggested.
its search facility...
34.5% of traffic to top
retail sites in July
2010 came directly
from [external] search
11


Amazon cannot afford to neglect improving its search facility. There are many
web shoppers that are not wedded to an Amazon-like one-stop shopping
experience. According to Hitwise, 34.5% of traffic to top retail sites in July 2010
came directly from search engines.

Google’s new instant search introduces the concept of consumers fine-tuning
search terms dynamically. If this proves popular, Amazon will need to respond.
There are also examples of reviews being included in search results (Bazaarvoice
is supplying reviews to Google that includes content from sites like BestBuy and
Macys).

As Facebook becomes more and more a destination site for consumers, its search
feature might drive traffic to rival sites, particularly if their Facebook presences
are more advanced or feature-rich.

Negative sentiments were in the minority but can’t be
ignored
The important thing about negative sentiments in a one-minute survey is that
they help companies isolate general issues and provide the roadmap of where to
start ‘digging’. Furthermore, we have found that Web teams welcome some
negative findings, which are often seen as a validation of their own opinions. A
more typical ratio of positive to negative is 65/35 or 70/30, and in these cases
the discussion of how to rectify problems would dominate the analysis.

Clutter and layout were problems, but not for all
The most significant negative aspect of Amazon.com was ‘cluttered layout, hard
to read’. Clearly, page layout matters - it has direct impact on the ability to
complete tasks. Too many long blocks of text, too many links, clutter, not enough
white space, lack of a coherent structure; these can all slow people down.
... ‘you guys are so
According to a consumer for whom this is the most important facet of
primitive, you are
Amazon.com:
like cavemen. Don’t
I’ve used the site many times and have found it confusing to use. It has taken
you have any sense
me too long in the past to get to the information I need or to find out that I
of aesthetic?’... had to do something else to get where I wanted to be. Just a hassle!
In all the complaints - Female consumer, 25-34 years old, quite familiar with the site
and requests we get
The bigger takeaway story on this factor was that this was much more important
from users, this is
for professionals than consumers. It was the only factor where professionals
never one of them. differed from consumers to any degree of significance. It may indicate that web
professionals tend to judge layout more than consumers do, years of experience
— Jim Buckmaster, having trained them to look at websites differently. Very few Web professionals
CEO of Craigslist would not criticize Craigslist yet it remains a very popular site. According to Jim
Buckmaster, CEO of Craigslist:

12


“I hear this all the time, ‘you guys are so primitive, you are like cavemen.
Don’t you have any sense of aesthetic? ‘ But the people I hear it from are
invariably working for firms that want the job of redoing the site. In all the
complaints and requests we get from users, this is never one of them. Time
spent on the site, the number of people who post—we’re the leader. It could
be we’re doing one or two things right.”

To reinforce this view, far more people chose ‘simple layout, easy to read’, with
consumers more satisfied with the layout than the professionals.

Although subjectivity is also a factor in this attribute, the difference of opinion
shows the importance of getting the opinions of non-expert consumers, who
vastly outnumber the Web professionals in the online space. Without intending
to, we were able to show that web professionals may not always be the best
predictors of consumer sentiment.

Some people find it hard to contact a person
Sears offers chat, call At one time, it was extremely difficult to get in touch with Amazon support staff.
back and email on Things have improved, but it still is not that easy to engage with a
every page to assist the representative.
consumer complete the The issue is the lack of real-time escalation – being able to contact an agent by
buying process — this phone or chat without leaving the page. Many competitive sites offer telephone
is a standard of numbers, live chat, instant callback and click-to-call to ensure that consumers
availability of help that don’t lose momentum by having to navigate to another page to resolve an issue.
sets expectations for According to Art Technology Group, 58% of consumers now look for live help if
missing information while online, while 53% will do so if there are problems
visitors to sites like
checking out. It was interesting that men looked for help less than women,
Amazon’s.
reinforcing the general perception of men having an aversion to asking for
directions.

Sears, a retailer that has announced major plans to expand its online offering
with the adoption of an ‘Amazon-like’ marketplace that offers products from third
Chatters that engage parties, provides prominent options for consumers to reach out. Service agents
via proactive invitation can view the contents of the consumer’s shopping cart and answer any questions.
are 6.3X This helps to reduce cart abandonment, a major issue for online retailers. Sears
more likely to convert offers chat, call-back and email on every page to assist consumers complete the
than visitors who don’t buying process — this is a standard of availability of help that sets expectations
for visitors to sites like Amazon’s.
chat.

— Bold Software Live
Chat Performance
Benchmarks October
2009

13


Live chat with an agent from product pages allows Sears customers to escalate issues or
get answers without interrupting their tasks

It would be very revealing to drill down with the participants that expressed
concern about contact options to find out, for example:

Why is this an issue – what was the reason they needed to contact
Amazon when they found it difficult?
Was the issue they tried to escalate to do with a site issue, or perhaps an
offline issue like late delivery or a billing discrepancy?
Did the issue cause them to abandon a purchase and go elsewhere?
What were their expectations of Amazon – a telephone number on every
page, or just an easier path to an email contact form?

14


Visual appearance not of overriding importance
Visual appearance is of course another subjective characteristic, but what is
striking is that the ‘look’ of the site was not important to most people, whether
they liked it or not. (There was only a slight difference here between consumers
… the ‘look’ of the site was
and professionals.)
not important to most people,
whether they liked it or not Functionality seems to be what matters — with the appearance of Amazon not
getting in the way of people browsing products and buying. This indicates that
once the appearance of a website is not strikingly bad (and Amazon’s certainly
isn’t), it does not really register prominently with visitors. Once customers can do
what they need to do, they’ll not be too concerned about colors, fonts, imagery
or other design features. That’s not to say that any design will do – it still needs a
professional to design an interface that is visually coherent and serves the brand
well. But once people can get where they need to go quickly, and complete their
tasks, the visual appearance is very much secondary to functionality. Consumers
are probably too short on time to notice the visual design of site that is designed
to get things done.

This finding also raises further questions:

What type of site visitor finds Amazon.com unattractive?
Are new visitors more critical of the appearance, perhaps it becomes less
important as they find out that the site is functionally excellent?
What exactly about the appearance is causing an issue – is it related to
clutter, or is it a purely subjective attribute like the choice of colors
and fonts?
Why do they feel strongly enough about appearance to pick this attribute
rather than one of the other 24 available?

15


Report Summary
Consumers want choice and they want low prices. That’s a big reason for the
success of Amazon.com. Alongside its own merchandise, Amazon also offers
goods from other retailers and the revenue from this ‘marketplace’ in the first half
of the year was roughly $4.2 billion.

Major online competitors including Sears and Wal-Mart are aggressively building
their own marketplaces and industry experts say several more large retailers will
launch similar efforts in the coming months.

So, does our study help understand Amazon’s strengths, weaknesses,
opportunities and threats?

Clearly, ratings and reviews are a critical attraction for Amazon but they
need a significant upgrade to exploit new channels of social media,
especially Facebook. Sears has already developed a sophisticated
online community which looks to be more advanced than Amazon’s.
This is critical to consumers and in a world of diminishing loyalty may
serve to weaken Amazon’s dominant position
Customers increasingly see online retail as a multi-channel experience and
it is likely that Amazon will need to provide better options for
speaking to a service person in real time. The study indicated a small
(but what is likely to be a growing) dissatisfaction in this area. A
continuous review process would be able to monitor that. Both Wal-
Mart and Sears already offer better capabilities as well as store pick-
up options and are experimenting with same-day delivery in major
cities
Search is highlighted as critical and while Amazon’s is good, there are
sites which already surpass its capabilities. There is the added threat
of Facebook and Google search as the most popular starting points
for consumers.

In summary, Amazon received a very positive rating so on the surface it has little
to put right. But as we have shown significant challenges are emerging.

16


Conclusions
We produced this report to step through a complete process; we used Amazon
simply because many people are familiar with the site and have impressions.
Many readers of the report will have provided input so you know that it did take
the 1 minute we say. You all have regular visitors to your sites; what are their
impressions, what effect does that have on viral recommendations or on
repetitive use.

The benefit of the methodology is the combination of a unique survey technique
in combination with the expert interpretation from a range of partners that have
extensive online experience.

Already the methodology has been applied to sites across multiple industries and
countries, it provides great value for a comparative inexpensive commitment.

17


Methodology
We created an online poll and asked participants to tell us which three of 26
statements most closely described how they felt about Amazon.com, and allocate
‘3’ to the strongest sentiment, ‘2’ to the next, and ‘1’ to the third (mirroring Gerry
McGovern’s widely-used Customer Centric Index).

The statements were displayed in a randomly-ordered list. Although there were
26 statements, they were in fact positive and negative sentiments of 13 core
attributes.

Visual/Architectural factors
Clear menus and links Confusing menus and links

Helpful search results Poor search results

Simple layout / easy to read Cluttered layout / hard to read

Looks attractive / appealing Looks unattractive / unappealing

Fast to do things Slow to do things

Content factors
Accurate information Inaccurate information

Complete information Incomplete information

Up-to-date information Out-of-date information

Plain language Full of jargon / corporate speak

Social factors
Gives me the facts / transparent Misleading / not transparent

Easy to contact a person Hard to contact a person

Easy to participate / give feedback Hard to participate / give feedback

Has ratings, reviews, Has no ratings, reviews,
recommendations recommendations

18


The statements as they were presented to participants (randomized for each)

19


USA

The Customer Respect Group
978.412.4047
http://www.customerrespect.com

Net Reflector
206.462.4240
http://www.netreflector.com/

Bob Johnson Consulting
248.766.6425
http://www.bobjohnsonconsulting.com

Canada
Customer Carewords is a methodology
Neo Insight, Inc
that helps you to truly understand what
613.271.3001
the top tasks of your customers are when www.neoinsight.com
they come to your website and the critical
impressions they take away. Europe

By continuously improving the perform- Ireland
ance of your customers' top tasks, you will Customer Carewords, LLC
maximize the performance of your web- www.customercarewords.com
site. +353 87 238.6136

United Kingdom
We have been developing customer cent-
ric strategies for websites since 1994, Barry Hagan
identifying the top tasks of thousands of +44 77 1470.0066
customers and employees for organiza- www.customercarewords.com
tions such as Microsoft, Cisco, Tetra Pak,
Thrivent Financial, US Internal Revenue Brian Lamb
Service, NHS Choices, Rolls-Royce, BBC, +44 79 8070.0075
www.customercarewords.com
Innovation Norway, etc. We have partners
in the UK, Holland, Sweden, Norway, Sweden
Canada and United States. Webbrådgivaren Sverige AB
+46 040.134.200
http://www.wpr.se

Netherlands
Sabel Communicatie
+31 088.227.2240

20


http://www.sabelcommunicatie.nl

Norway
NetLife Research
+47 9240.3165
http://www.netliferesearch.no

21

What Can Be Learned From A One-Minute Online Survey

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (17)

Último

Último (20)

What Can Be Learned From A One-Minute Online Survey