This document summarizes a presentation on conducting user needs assessments. It discusses why user-centered design is important, how to sell needs assessments to project sponsors, and techniques for gathering user information, including surveys, interviews, observations and contextual inquiries. Case studies are provided to illustrate how these techniques were applied to understand users of a webcast study tool. The document concludes with recommendations for analyzing findings and convincing teams to use results.
2. Agenda
I. Why user-centered design and user
needs assessment is important
II. Selling user needs assessment
III. Gathering good information about
your users
IV. Understanding the information you
gathered
3. Adapted from Usability Professionals’ Association website,
http://www.upassoc.org/usability_resources/about_usability/definitions_of_usability.html
I. Why User-Centered
Design?
• Increased customer satisfaction
• Increased user productivity, efficiency, and
accuracy
• Increased service/site usage and adoption
• Decreased support and training costs
• Reduced development time and costs
– Create only the features users need
• Reduced maintenance costs
– Do it right the first time
4. User Centered Design
Process Overview
3. Specify user
and organizational
requirements
2. Specify the
context of use
4. Produce
design solutions
1. Plan the user
centered design
process
5. Evaluate
designs against
user requirements
Complete
Adapted from ISO 13407: http://www.ucc.ie/hfrg/emmus/methods/iso.html
5. How does needs
assessment help?
• Focuses on understanding:
– Who are the users?
– What are their goals?
• Goals drive a person’s actions
• Tasks are things a person does in order to accomplish
his goals
– What are their pain points?
• Uses a variety of observational techniques to
gather this data
• This understanding drives design
6. II. Selling user needs
assessment to project sponsors
• This is hard to do
• Think like a salesperson:
– You need to get them to buy (and buy in) to user
needs assessment
– Use the sponsor’s terminology and background to
shape your arguments
– Quantify where possible (e.g. reduced training &
support or development costs)
– Discuss user’s expectations (Google, Amazon)
– Start small and sell up
7. Talking your sponsor’s
language
• UCD processes include activities sponsors
may call:
– Business Analysis
– Requirements Definition and Verification
– Information Architecture
– Web Design and/or Development
– User Testing
– Proactive User Support
– Other ideas?
8. Why sponsors aren’t users
• Sponsors have:
– great familiarity with the existing solution
– great familiarity with underlying business processes
– too much experience receiving and responding to requests
for assistance, complaints, or suggestions from more vocal
users
• Sponsors know (or think they know):
– all about the service area that needs improvement
– “how it's always been done”
– what “can't be changed”
– constituencies that use or are impacted by the service
– what financial and personnel resources are available to do
the work
9. Set yourself up for success
• A good project to choose is/has:
– One you can get involved with early
– Real users you can talk to
– A receptive project team
– A scope that is the right size for you/your team to
handle
– A short project allowing you to show quick wins
– Improving an existing product
• Learn how to say no
10. III. Gathering good
information about your users
• Case study: Webcast Study Tool
• Selecting representative users
• User Needs Assessment techniques
11. Case study: Webcast study tool
• Many Berkeley lectures are webcast
• Allow students to apply known effective
study techniques to opaque media
• Surveyed students & faculty
• Interviewed stakeholders
• Interviewed & observed students:
– in traditional study environments
– using webcast to study
12. Selecting representative
users
• Determine which characteristics define your
target user base
• Ensure that each characteristic is
represented in (approximately) the right
proportion
– Don’t test just one end of the spectrum (e.g.
vocal users who are calling customer support)
• Can use a profile matrix to keep track of all
the characteristics that matter
15. 1. User needs assessment tips
• Be a good listener
• Remain neutral: don’t react
• Focus on goals first, tasks second
• Don’t limit yourself to a fixed set of questions
• Encourage story telling
• Distance yourself from the product
• Avoid making the user a designer
• Categorize notes = easier analysis
• Analyze your notes within 48 hours
• Ideally should be performed in teams
16. From Jakob Nielsen, “Field Studies Done Right: Fast & Observational,” http://www.useit.com/alertbox/20020120.html
1. User needs assessment tips
• Don’t use questions that can be answered
with “yes” or “no”
• Don’t ask leading questions
• Don’t use jargon
• Don’t draw attention to specific issues that
you care about
17. Keeping yourself on track
• It’s not always easy to determine what is
relevant
• Your problem statement likely describes your
project’s focus
– A starting perspective, lens, or viewpoint
– Is present whether articulated or not
• Focus structure document
– Clusters of questions or pieces of information you
are looking to explore, grouped categorically
18. 2. Surveys
• Strengths
– Getting general feelings about an existing product
– Acquiring demographic data on target market
– Understanding what users “think” is important (marketing
data)
• Weaknesses
– Survey design is very difficult to get right
– Constrained responses
– Typically limited time & focus for response
– Limited or no ability to follow-up for clarification
– Relies on user to self-report accurately
• Resource for designing surveys:
http://www.ssri.psu.edu/survey/educ.htm
19. 3. Focus Groups
• Strengths
– Can talk to many people at once
– Allows users to feed on each others’ ideas
– Understanding users' attitudes, beliefs, desires
– Getting users' reactions to ideas or to prototypes
• Weaknesses
– Understanding what people REALLY do with a product
– Understanding what features people will really use in a new
product
– “Group think” drives people toward consensus with the
loudest opinion
– Relies on user to self-report accurately
20. 4. Interviews
• Strengths
– Understanding how users understand their work
– Analyzing goals of work
– Ability to follow-up and clarify
– Builds relationships
• Weaknesses
– Relies on user to self-report accurately
– Experts often have an inability to describe what
has become subconscious (unconscious
competence)
– More time intensive for facilitator
21. “Users are perfectly
capable of expressing
their latent needs. They
just can’t do it verbally.
That’s why we do
ethnography and
empathic research!”
-Rich Sheridan, Menlo
Innovations
5. Observation
• Strengths
– Allows you to watch what people do rather than
rely what they say (self-report)
– More likely to discover unmet user needs
– Truly understanding how users get their work
done in context
– Observing subtleties of work (e.g. post-it notes,
cheat sheets, interruptions)
– Overcomes experts’ inability to describe what
has become subconscious
• Weaknesses
– Time commitment
– Difficult to be “a fly on the wall”
– Relies on observers’ interpretation
– Hard to know what to pay attention to
22. Interview
(Process
influenced more
by designers)
Observation
(Process
influenced more
by end-users)
Contextual Inquiry
(Process influenced by
both designers and
end-users)
6. Contextual Inquiries
• Combines strengths of interview and
observation
• Interview in the context of where the work
happens
• “Show and tell”
• Find “pauses” to ask questions; Don’t
interrupt their thought processes
23. Case study:
surveys told us
• Bookmarking and video download are the features
that are of greatest interest across the board
• Searchable captions, chaptering, and Powerpoint
sync are the features most highly rated by
webcast.berkeley students.
• Annotation is less popular than bookmarking.
• Interest in knowledge sharing tools is relatively low.
• The general webcast.berkeley audience is the only
one highly interested in being notified about posting
of video.
24. Case study: interviews &
observations told us
• Greatest pain points are finding specific spots in
webcast lectures
• Powerpoint slides are often-used reference point for
finding spot
• There’s administrative overhead in marking down
time code for getting to or returning to specific points
• Students replay specific segments to aid in
understanding, creating study sheets, etc.
• Students jot down notes while watching
• Students look at more than one webcast in a sitting
25. IV. Understanding the
information you gathered
• Who are users and how do they
accomplish their goals now?
– Personas
– Task analysis
– Activity Diagrams
• What do users need?
– Scenarios
26. Personas
• Research-based user archetypes representing needs of a set of
constituents
– Based on patterns from user research
• Allow designers & developers to put themselves into the shoes
of “real” users
• Can help build consensus & commitment to the design
• Should be specific, real & memorable
– Pictures, posters
– Include details about their life—humanize them
• Keep us from designing for:
– The elastic user
– The mirror persona (ourselves)
– Design edge cases
27. Academic goals:
• Get into Med school
• Feel confident walking into exams
• Be as efficient as possible
Personal goals:
• Stay healthy
• Have time to spend with friends
Example Persona - Webcast Study Project
Lisa Ng: Conscientious Student
• 2nd year undergraduate
• Planning to go to med school, so doesn’t feel she can
take risks with classes
• Rarely uses webcast as a replacement for class
• Relies on computers in lab on campus
• Use of webcast is primarily for studying for exams
• Good study skills: When studying with text, uses
highlighters to mark parts she’ll want to be able to find
again & to identify key points or points of confusion.
• When doesn’t understand what happened in class,
uses webcast to review
• Refers to PowerPoint slides when studying
28. Modeling what users do now
• Task Analysis
– Decomposes tasks in order to understand procedures better &
provide support for these tasks in the interface
• Helps ensure necessary features aren’t overlooked
– Define the task and the goal of the task and then list the steps
involved
– Can rate tasks on frequency, importance, difficulty
• Tells you what functionality is important
• Can help you combine similar personas
• Help you choose which tasks to include or emphasize in scenarios
• Activity Diagrams
– Modeling existing user behavior and interaction with their
existing system
29. Task analysis matrix
Persona/
Task
Katie Richardson Harold Jackson Sally McNeil Nina Sanchez
Create a
Calendar
Frequency Importance Frequency Importance Frequency Importance Frequency Importance
Design
calendar
appearance
LOW LOW MEDIUM HIGH MEDIUM HIGH LOW MEDIUM
Set up
calendar on
website
LOW LOW MEDIUM HIGH MEDIUM HIGH LOW MEDIUM
Manage
Events
Frequency Importance Frequency Importance Frequency Importance Frequency Importance
Add Event LOW HIGH HIGH HIGH HIGH HIGH MEDIUM HIGH
Edit Event LOW MEDIUM MEDIUM HIGH HIGH HIGH MEDIUM HIGH
Delete
Event
LOW MEDIUM MEDIUM HIGH HIGH HIGH MEDIUM HIGH
Approve
Events
LOW MEDIUM HIGH HIGH HIGH HIGH LOW LOW
32. Scenarios
• A design technique used to envision future
use of a system
– Focusing on how users can achieve their goals
– Helps designers & developers understand how
system will really be used
• A story about user interacting with the system
• Categorize scenarios as Daily, Necessary,
and Edge Use
• Can be used for usability testing
• Iterate on scenarios
33. Case study context scenario:
Studying for exam
• Lisa has an exam coming up and wants to create a study sheet she
can use for the next week while on the elliptical @ the gym.
• She gets out notepaper, her textbook, and her binder with PPT “notes”
pages and gets comfy on the couch.
• She starts reviewing the powerpoints and notes from the lectures after
the last exam. As she does this, she’s making notes (summarizing
important topics) on her notepaper. (This will become her study sheet).
• As she’s making her way through the slides she decides it would be
useful to hear the instructor’s explanation of DNA replication again.
• She goes to … a point in the webcast where that ppt slide is, and
listens. One sentence he says seems to encapsulate the concept for
her, so she tries to get it down word for word. Since her prof talks fast
and does not always use lay terms, she relistens several times.
• After she feels like she understands, she adds some notes in the study
sheet.
• She sees that there were a number of segments that she’d
highlighted
34. Convincing project team to use
needs assessment results
• Include project team (sponsors, developers,
designers) in needs assessment process:
– Have team members attend interviews,
observations, user tests
– Share interesting results with team while needs
assessment or usability tests are ongoing
– Share and get feedback on tools, questions, and
suggestions you’re developing
– Offer introductory training in UCD and needs
assessment techniques
35. Recommended books
• The Inmates are Running the Asylum and
About Face 3.0 – Alan Cooper
• The Design of Everyday Things and
Emotional Design – Don Norman
• Don’t Make Me Think – Steve Krug
• Usability Engineering – Jakob Nielsen
• User Interface Task Analysis - Joann T.
Hackos and Janice Redish
• Designing for Interaction - Dan Saffer
• Other recommendations?
IAN
Fewer lawsuits--ADA, etc.
Better design=better maintainability
POLITICS: Buy-in
・Start with a good design, know what your users need, don't waste time developing unnecessary features/fixing problems in the future
・if a system is not usable or seems to be designed without regard for the needs of the users, users may:
・be less productive
・be less accurate (in terms of data entered)
・be unhappy that they must use inefficient tools that make it more difficult for them to do their jobs
HIDDEN
Focuses on understanding:
Who are the users?
What are their goals?
Goals drive a person’s actions
Tasks are things a person does in order to accomplish his goals
What are their pain points?
This understanding drives design
IAN
Focusing on parts 2 and 3 in this talk
IAN
Goals:
Are what the user wants to do, but not how the user achieves them
Help you not to make any assumptions about the system interface
Can be used to compare different interface design alternatives in a fair way
Can be personal, practical, or false (don't focus on false goals!)
tasks:
Describe the steps necessary to achieve the goals
Can vary with the available technology
Are broken down into steps for task analysis, and are recombined into sequence of steps for scenario development
The main emphasis on our process is to understand who our users are, what there goals are and what their current pain points are.
So, who are we building the system for? It’s likely not the designers, developers, testers, support folks or anyone else on the development team. I’ll got into a little more depth in minute about how we learn about our users.
We want user goals to drive our system definition…hence goal-directed design.
Goals help us think out of the box and truly figure out how we can make users lives easier with technology.
For instance, I have a daily goal of getting to the office.
My task used to be to drive in traffic to meet that goal. Now most the time I ride my bike to meet my goal. In the future, maybe I’ll be able to simply press a button to get the office.
So the goal stays the same but the tasks change with context, with technological capabilities, with innovation…
We also want to focus on understanding where users feel the most pain with their current processes? Can technology take on some of the burden? Rather than replicate exactly what they do now in the new system, we need to figure out how we can make things better for them.
All this is meant to help us design systems that truly make our users lives easier
IAN
This is admittedly a difficult challenge--we’re still learning how best to do this, and we can’t give you a magic wand, but we can give
you a few good tips
Think like a salesperson--this is a sales/marketing process--UCD does cost real money, so you need to find ways to sell it in their language
Start with a small project “to show concrete results“
Or use a few techniques on a project to get some user input
Quantify where possible:
Increased productivity
Increased usage and adoption
Decreased support and training costs
Reduced development time and costs
Reduced maintenance costs
Increased customer satisfaction
Adapted from Usability Professionals’ Association website, http://www.upassoc.org/usability_resources/about_usability/definitions_of_usability.html
IAN
Proactive user support instead of reactive user support – solving problems for users before they happen
These are our ideas, be thinking as we talk about these, will ask you for ideas
IAN
This knowledge is very useful, and is something you should certainly pay attention to as you do your work,
but this all distinguishes the sponsor from a real user and prevents them from being able to accurately represent what the user needs
IAN
Start small!
With a small set of users
Non-disruptive technology
Under the radar (not politically loaded, non-mission-critical)
Learn how to say no
Choose a few projects you can do really well rather than spreading the team too thin and doing nothing well.
Doesn’t have to have all of these things, these are just some suggestions based on our experience. If you can get a project with 3 or 4 of these criteria, you’re on the right track.
DAPHNE
Next we’ll talk about how to gather good information about users.
Goal to understand what they do now so you make sure a new system can support their needs
Using case study about project done in ETS to demonstrate the activities we describe.
Allison will talk briefly about selecting users to study
And then we’ll discuss some user needs assessment techniques that we’ve found useful
DAPHNE
ALLISON
ALLISON
ALLISON
Distance: It's often best not to describe why you have a vested interest in the project so the user feels comfortable being frank with you.
Interview where the interaction happens
Avoid discussions of technology
ALLISON
ALLISON
While it’s important to explore different topics brought up by the users, it is important make sure you don’t end up with lots of irrelevant data
In interview situations, you need to know when to stop users from talking about things they love, but are not pertinent to the your project – this is a judgment call because you may not know for sure whether or not it is relevant, but if your time is limited you likely only have enough time to explore selected lines of questioning/focus
In an observation, you need to know what to look for, what behaviors will inform what you are researching
Note-taking short-cuts/abbreviations may also help you organize your thoughts
OLD
Getting information that isn’t relevant
Problem statements
Who are the users
ALLISON
Have you ever been asked to take a survey and can’t figure out where you fit on the given responses/the response you want to give isn’t a choice?
ALLISON
a traditional market research technique
Need a skilled moderator to ensure everyone participates
OLD
People have trouble accessing what they do and how
Good for gauging initial reaction to form of a product, or getting feedback on visual appearance
Industrial design
ALLISON
HIDDEN
May start out with structures and comprehensive set of questions
Good goal is to get to where you are using a “guide” of information you’d like to know about rather than a fixed set of questions. Will make it seem more conversational.
“Think of yourself as a therapist” and draw the user out without imposing your own ideas or preconceptions.
Ask users:
What are their overall goals? - Goals remain constant even when the technology changes
What tasks are they performing? How often is each task performed?
How easy is it to perform each task using the current system?
Ask yourself: How critical is it that the system support each task? – Your job is to determine whether the tasks they are performing are really necessary or can be done in a different way to achieve their goals
How can the current system be improved? (If possible, observe the user using this design.)
Tasks vs. Goals:
-If a stated goal involves a particular technology, it’s probably not the real goal--dig deeper
Phases:
-May not be able to do all phases in a project--that’s OK.
-Is also possible to combine first two phases in some cases.
HIDDEN
HIDDEN
6. It's often best not to describe why you have a vested interest in the project so the user feels comfortable being frank with you.
DAPHNE
Observations are popular technique
Don’t have to rely on users to remember and be able to tell you the details and subtleties of their work
Post-it notes and cheat sheets give us clues about where the system could support them better
Research shows as people become experts much of their work becomes automatic -- it’s done at a subconscious level.
Observations allows you to overcome this
Talk about weaknesses
One way to overcome interpretation is to ask questions at the end
DAPHNE
Cross between interviews and observation and combines strengths of both
With contextual inquiries you still go the user and interview them in their normal context
Idea is to interview users in the context of their work and ask them to show you what they describe along the way
Tip: Important to look for natural pauses in their answers and as they show their work so you don’t interrupt their natural flow
Talk about contuim
DAPHNE
More surfacy, high level information
What they are interested in and what’s popular
Talk to bullets
As mentioned they are good for helping you focus further studies
DAPHNE
Don’t expect you read the entire slide, important points are in bold.
Interviews & observations gave us more detailed information.
Talk through the bullets
ALLISON
personas and scenarios can (from UIE seminar):
1) bring focus
2) common language [Daphne’s idea]
3) build empathy
4) encourage consensus
5) create efficiency
Common language is important--users may know a great deal about their field, but not about computers and therefore may use different terms than you would. Figuring out what they really mean is critical. Ian’s department is currently working on a project to talk to Arts and Humanities scholars and those folks often fall into this category. Getting the vocabulary clear and normalized across users is critical.
Assuming all this work will lead to a new system design, it’s worth mentioning, the importance of sharing out what you learn along the way before any design even happens. If you don’t gain common ground around who the users are and what they need, by the time you get to the design stage there is tends to be an assumption that the disagreement is about the specific implementation when it’s more likely about one of the previous two.
“Bring the rest of the team along” in your understanding
Create models to help understand and analyze the data
ALLISON
Personas can:
Determine what a product should do an how it should behave
Communicate with stakeholders, developers & other designers
Measure the design’s effectiveness
Contribute to other product-related efforts such as marketing & sales plans
Bullseye model of personas
HELPS AVOID THE ELASTIC USER - “the user in the developer’s head often changes to match what can be done, not what the user really needs”--WE’RE ALL VULNERABLE TO THIS
Limit number personas--one per important user category
Don’t use real people
Think about the target audience for your personas
Developers will find it useful to know different things about a persona than marketing people would, but both can find personas helpful
ALLISON
Flickr Creative Commons is a good resource for pictures
Role-playing - interview a persona
Post short versions in the designers’/developers’ workspace & keep the longer descriptions handy (e.g. on a website).
ALLISON
Tasks can later be recombined as part of scenario development
DON’T READ
Definition - a set of methods for decomposing people's tasks in order to understand the procedures better and to help provide computer support for those tasks. The basic approach is to define the task and the goal of the task and then to list the steps involved. The level of detail in decomposing the steps is determined by how the analysis is going to be used.
Task analyses are useful for making time predictions for how long a task will take and for spotting potential errors (steps in the process which are extremely difficult or confusing). Task analyses are also good for spotting areas in a user interface that may have been overlooked or oversimplified. A task analysis is a fundamental part of a cognitive model of user performance, such as GOMS. From http://www.usabilityfirst.com/glossary/term_294.txl
ALLISON
Different users perform tasks with differing levels of frequency & importance
May tell you about how to expose functionality in the interface (allowing functional minimalism)
Can also create weighted matrix using numbers (0-3) multiplied by the importance of each persona if you need to prioritize features in order to choose which ones to implement
DAPHNE
As Allison mentioned activity diagrams can be a good way to think through what you learn from users
The idea is to diagram the work you saw and/or heard about it
Again, this from webcast project
The highlighted area shows how cumbersome the activity of getting to a particular point in the webcast based on a certain point in time in lecture they want to get back to.
This led us to build in automatic time adjustment from the lecture time they wrote down or remembered to right time in the video
DAPHNE
2nd half of the same diagram
Visualizes parallel work
Our design allowed them to have all of this at once with the ppts synched to the video and several ways to help them add notes like highlighting parts of the video, adding bookmarks and annotations, and in 1 click making an assignment at segment to come back to.
DAPHNE
Writing scenarios technique to start thinking about how the new system needs to support what you learned about users and their needs.
Scenario is a hypothetical chain of events through activities that meet users goals
So, if the user’s goal is to give students feedback on their latest assignement, the scenario describes the most effective and efficent way they can do that.
Another helpful technique is to categorize scenarios
Scenarios can be used throughout the project. When doing user testing you can ask users to walk through a similar scenario to see how well the new system supports them.
Another tip is to iterate on scenarios. Next I’ll show you an early scenario.
HIDDEN
DAPHNE
One of the scenarios from the webcast project.
Don’t expect for you read all this, I’ve highlighted some important aspects in the scenario that I’ll talk about in a minute.
This is early in the project. At this point we like to describe the work without any implementation details so the focus on is what kind of information the user needs along with the natural flow of the work.
By not constraining yourself with how the system will support users at this point, it allows you to think out of the box.
Walk through what we learned.
DAPHNE
And lastly, we wanted to briefly touch on the importance of getting buy-in from the rest of the team
Besides convincing sponsors to let you do user needs assessment in the first place, you may need to convince your team to use the results.
I find this is most often the case when it comes to making decisions about what features are in and out and how much time to spend giving user a particular way to accomplish a task -- even though it might require additional implementation time.
“Bringing the team along with you” and gaining buy-in along the way while you are doing the research and creating models will help ensure they respect and understand how the results drive your system design.
Some ways to do this are:
DAPHNE
DAPHNE
TPO site for all the links above, as well as many other resources
Accessibility is a related topic; more resources about that on TPO site too