User Surveys: STC, Seattle, April 2002
•Transferir como PPT, PDF•
1 gostou•242 visualizações
This presentation to the Society for Technical Communication conference in Seattle, US, covers processes and common pitfalls in survey design.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Semelhante a User Surveys: STC, Seattle, April 2002
Semelhante a User Surveys: STC, Seattle, April 2002 (20)
Mais de Caroline Jarrett
Mais de Caroline Jarrett (20)
Último
Último (20)
User Surveys: STC, Seattle, April 2002
- 1. Creating Effective User Surveys Caroline Jarrett, Effortmark Karen Bachmann, Seascape Consulting
- 2. 7 May 2002 User Surveys - Jarrett, Bachmann2 Outline Introduction When to do a user survey 10 steps to a survey Feedback
- 3. 7 May 2002 User Surveys - Jarrett, Bachmann3 Benefits of user surveys Define the characteristics of users Get direct input from users Can have a big sample size
- 4. 7 May 2002 User Surveys - Jarrett, Bachmann4 Challenges of user surveys Deciding between interviews and questionnaires Getting a good design Interpreting the responses
- 5. 7 May 2002 User Surveys - Jarrett, Bachmann5 Interview or questionnaire? Interview Talk to the user Interviewer captures the answers Requires small samples Flexible Questionnaire Do not talk to the user Receive written answers from user Supports large samples Inflexible
- 6. 7 May 2002 User Surveys - Jarrett, Bachmann6 An example Look at ‘before’ example from a survey of project managers What does this tell you about the project managers?
- 7. 7 May 2002 User Surveys - Jarrett, Bachmann7 Our activity example New company SuperIndex™ Plan to launch SuperIndex™ at STC 2003 Opportunity here to find out: about users when they’ll use the product what they want to see in it
- 8. When to do a user survey
- 9. 7 May 2002 User Surveys - Jarrett, Bachmann9 When to do a user survey To specify users To establish user goals To characterise user groups Interviews: Find out about users Questionnaires: Confirm numbers, not discovery
- 10. 7 May 2002 User Surveys - Jarrett, Bachmann10 Pragmatic choice of method No direct contact with users Direct contact with users Little time to collect information Short questionnaire Telephone or electronic interview Plenty of time Detailed questionnaire Face-to-face interview
- 11. 7 May 2002 User Surveys - Jarrett, Bachmann11 Ideal choice of method Face-to-face interview Telephone or electronic interview Detailed questionnaire Short questionnaire
- 12. 7 May 2002 User Surveys - Jarrett, Bachmann12 Let’s try it Fill in ‘use of surveys’ questionnaire How many? Top tip?
- 13. 7 May 2002 User Surveys - Jarrett, Bachmann13 When NOT to do a user survey Don’t yet have a business definition of the users Forced to use a method that doesn’t match the time available
- 14. 10 steps to a user survey
- 15. 7 May 2002 User Surveys - Jarrett, Bachmann15 10 steps Preparation 1. Set goals 2. Decide on target group 3. Interview target users Production 4. Create questions 5. Decide on delivery 6. Test 7. Revise for final survey Delivery 8. Conduct 9. Compile and analyse responses 10. Publish results
- 16. Preparation 1. Set goals 2. Decide on target group 3. Interview target users
- 17. 7 May 2002 User Surveys - Jarrett, Bachmann17 1. Set goals for the survey What question are you trying to answer? How do you need to use the information? When do you need the information?
- 18. 7 May 2002 User Surveys - Jarrett, Bachmann18 2. Decide on the target group Who are they? Do they all own the same information? Census or sample? Do they want to tell you the information? How will you find them?
- 19. 7 May 2002 User Surveys - Jarrett, Bachmann19 3. Interview some target users Check that you have the right group Establish correct language Find out what they want to tell you Validate and refine your goals
- 20. 7 May 2002 User Surveys - Jarrett, Bachmann20 Activity: Steps 1, 2, 3 Set goals: market for SuperIndex™ Decide on target group: STC attendees Activity: Create an interview plan Try using it on your neighbor
- 21. 7 May 2002 User Surveys - Jarrett, Bachmann21 Typical first attempt Would you use SuperIndex? When would you use it? What benefits would you see from it? When would you NOT use it? Check against step 3: Interview some target users
- 22. Production 4. Create questions 5. Decide on delivery 6. Test 7. Revise for final survey
- 23. 7 May 2002 User Surveys - Jarrett, Bachmann23 4. Create the questions “similar to driving in freeway traffic while drinking a cup of hot coffee and answering an emergency call on a cellphone” “Many things are competing for attention and failure to heed any of them can spell disaster”
- 24. 7 May 2002 User Surveys - Jarrett, Bachmann24 4. Create the questions Mix question types: choice and open Use appropriate, unambiguous language Avoid leading questions Present one question at a time Keep questions concise Keep positive
- 25. 7 May 2002 User Surveys - Jarrett, Bachmann25 4. Organize the questions Keep to one topic at a time Start with “easy” questions Move from less invasive to more invasive Minimise requests for personal information
- 26. 7 May 2002 User Surveys - Jarrett, Bachmann26 Activity: Create and organise questions Here is a list of questions Rewrite them as good questions Organise into topic order
- 27. 7 May 2002 User Surveys - Jarrett, Bachmann27 4. Write a preamble Provide sufficient and clear instructions Tell participants what the survey is for Can they receive results if interested? Are responses anonymous or confidential?
- 28. 7 May 2002 User Surveys - Jarrett, Bachmann28 Activity: Comment on preamble You have been selected to take part in a survey of Project Managers to identify which approaches are being used to run projects. This information will be used to re-develop the current Life Cycle to meet the needs of Group Technology. Please answer the questions below and return to ????? by ??/??/2002. It should only take a few minutes to complete.
- 29. 7 May 2002 User Surveys - Jarrett, Bachmann29 5. Decide on delivery format Good for Problem Mail anonymity slow Website anonymity sampling Email interested respondents privacy spam Telephone direct interaction time-consuming Face-to-face direct interaction time-consuming, costly
- 30. 7 May 2002 User Surveys - Jarrett, Bachmann30 6. Test the survey Is the time involved to complete the survey appropriate? Are questions clear and unambiguous? Are you targeting the correct group? Is the survey interesting for the respondents?
- 31. 7 May 2002 User Surveys - Jarrett, Bachmann31 Activity: Create Survey Write out your questions in topic order Swap surveys with someone at your table When complete, compare your questions and responses with your partner
- 32. Delivery 8. Conduct 9. Compile and analyse responses 10. Publish results
- 33. 7 May 2002 User Surveys - Jarrett, Bachmann33 7. Prepare the final survey Use the test finding to refine the survey Try analysing the results Check the results against the survey goals Eliminate off-topic questions Confirm delivery method for target group
- 34. 7 May 2002 User Surveys - Jarrett, Bachmann34 8. Conduct the survey Schedule the necessary time for delivery of the survey Plan to send a reminder message to target users Deliver the survey Thank participants
- 35. 7 May 2002 User Surveys - Jarrett, Bachmann35 9. Compile responses Responses: Compile using: Up to 100 Spreadsheet 100 to 1000 Database, e.g. Access Over 1000 Consider specialist software e.g. SPSS
- 36. 7 May 2002 User Surveys - Jarrett, Bachmann36 9. Analyse responses Calculate numerical responses beware of false statistics Group qualitative responses don’t get too hung up on minor differences Be prepared for surprises some of your assumptions will be challenged
- 37. 7 May 2002 User Surveys - Jarrett, Bachmann37 Activity: Analyze responses Review the responses in the handout Provide conclusions based on the responses Create a report about the responses Present your report to the group
- 38. 7 May 2002 User Surveys - Jarrett, Bachmann38 “I’ve finished the report for you”
- 39. 7 May 2002 User Surveys - Jarrett, Bachmann39 10. Publish results Don’t surprise people with bad news Make sure publication is timely Keep reports short It’s OK to have some gaps in the results, “more work needed”
- 40. 7 May 2002 User Surveys - Jarrett, Bachmann40 Example questionnaire ‘after’ Distributed to 300 very busy project managers Obtained 100 responses Showed that the two existing methodologies are used very differently
- 41. 7 May 2002 User Surveys - Jarrett, Bachmann41 Questions Caroline: caroline.jarrett@effortmark.co.uk Karen: karenlynn@mindspring.com Handouts: http://karenlynn.home.mindspring.com/ http://www.effortmark.co.uk
- 42. 7 May 2002 User Surveys - Jarrett, Bachmann42 References Don A. Dillman (2000) Mail and Internet Surveys: The Tailored Design Method John Wiley & Sons Inc, New York ISBN: 0-471-32354-3 A. N. Oppenheim (1992) Questionnaire Design, Interviewing and Attitude Measurement Pinter Publishers Ltd, London ISBN: 1-85567-044-5 JoAnn T. Hackos and Janice C. Redish (1998) User and Task Analysis for Interface Design John Wiley and Sons, Inc. New York. ISBN: 0-471-17831-4
Notas do Editor
- KB: Should questionnaire use an outline to mirror the interview plan of interviews?
- Make sure that the survey is the right tool for the job. Understand the limitations of surveys. Users may misremember information or may misrepresent information to meet what they think are the goals of the survey or to make themselves look better (social desirability bias). Most will be biased in some way, especially in cases where all members of the target group are not equally motivated to respond. If you receive only negative feedback about a product, that may be because only unhappy customers were motivated to reply and slanted the results rather than an accurate reflection of where the product needs enhancement.
- A user survey can be very effective when used to collect the right information at the right time and in the right format. The type of survey used—questionnaire vs. interview vs. test, lengthy vs. short, anonymous vs. confidential, self-administered vs. investigator-administered—depend on a number of factors.
- A user survey can be very effective when used to collect the right information at the right time and in the right format. The type of survey used—questionnaire vs. interview vs. test, lengthy vs. short, anonymous vs. confidential, self-administered vs. investigator-administered—depend on a number of factors.
- Before even deciding to conduct a survey, you must determine what information you need to capture. If the information is best captured using a user survey, then setting well-formed goals will keep the survey focused and increase the likelihood of collecting the right information. Set well-defined and concrete goals. Instead of “Learn whether users understood the documentation” set a goal to “Learn whether human resources personnel were able to use the documentation to perform all basic application tasks.”
- Rarely will you really need or want to poll all possible users, unless you have a very small target population. Understanding which users will provide the desired information will result in quality responses. It may also increase the response rate if the target group has a particular interest in the information being collected (for example, senior users being asked about on-screen legibility). Instead of targeting all users of the product consider targeting specific groups, such as HR reps, women between the ages of 25 and 40, or users over the age of 55.
- Once you know the target population is, increase your understanding of those users by interviewing sample members to help validate and refine your goals as well as give you a start on creating questions.
- Using the information gathered, create a list of questions that capture the information. Organize the questions so that respondents are drawn into the survey, saving the more personal or difficult questions for later in the survey. Note: Including 7 choices does not a Likert-type scale make. Survey format Structured: Every participant answers the same questions; good for remote delivery Semi-structured: Every participant answers the same core questions, but the interviewer can expand areas as needed; good for telephone and direct interviews Unstructured: The interviewer asks questions freely; best used for exploration not reaching concrete conclusions
- Using the information gathered, create a list of questions that capture the information. Organize the questions so that respondents are drawn into the survey, saving the more personal or difficult questions for later in the survey. Note: Including 7 choices does not a Likert-type scale make. Survey format Structured: Every participant answers the same questions; good for remote delivery Semi-structured: Every participant answers the same core questions, but the interviewer can expand areas as needed; good for telephone and direct interviews Unstructured: The interviewer asks questions freely; best used for exploration not reaching concrete conclusions
- Example of last item: In the blood donation interview, all answers should be No except two questions near the end.
- Anonymous is when the survey administer does not know the respondents’ identity. Confidential means that the survey administer knows who the respondents are, but will not convey that information—this requires more trust from participants and, therefore, a higher vested interest in the survey.
- The survey should be delivered in a format that encourages accurate and complete information as well as high response rate. The higher the vested interested of the group, the more questions you can ask and the more likely you are to elicit direct interaction.
- Verify that the survey is usable by the target users. Areas to check is whether the time involved to complete the survey is appropriate, whether the questions are clearly understood, and if the target group is the correct sample to survey. Example for point 4: User surveys at the end of user guides and the response rate.
- Using feedback from the test step, refine the survey and prepare the final version. Depending on the length and importance of the survey, you may want to iterate through steps 4-6 several times.
- Schedule the necessary time for delivery of the survey, whether a mailed questionnaire or an on-site interview. For questionnaires, depending on the deadline for responses, plan to send a reminder message to target users.
- Capture the information collected in the survey in a format that is unbiased and useful to stakeholders (management, development team members, and so forth). Circulate the results and any pertinent analysis as soon as possible after the response deadline