Rethinking user research for social design
•Download as KEY, PDF•
18 likes•1,432 views
This is a talk I gave at IA Summit 11 about how the methods we've been using for a generation to evaluate UIs are imperfect for social interfaces.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Rethinking user research for social design
Similar to Rethinking user research for social design (20)
More from Dana Chisnell
More from Dana Chisnell (20)
Recently uploaded
Recently uploaded (20)
Rethinking user research for social design
- 1. Rethinking User Research for the Social Web Dana Chisnell dana@usabilityworks.net @danachis IA Summit 2011 #rethinking 1
- 2. Shelly 2
- 3. The state of the art is a generation old Time for a new generation of user research methods. 3
- 4. Usability testing isn’t telling us what we need to know for SxD Me, Inc. 4
- 5. We don’t know how to find out about things we don’t know about 5
- 6. People don’t live in the world doing one task with one device out of context WebEx - meetings Marriott.com - reserving rooms Me, Inc. - retirement planning eHarmony - coded messages in Libya Facebook & Twitter - organizing demonstrations in Egypt 6
- 7. 7
- 8. H+C=I 7
- 9. 7
- 10. H <== (computer) ==> H 7
- 11. 8
- 12. Methods and measures were probably always imperfect. 8
- 13. Methods and measures were probably always imperfect. But they used to be enough. 8
- 14. Methods and measures were probably always imperfect. But they used to be enough. No more. 8
- 15. Everything is different now. But not different. All this has happened before. 9
- 16. The nature of being online is social It always has been. email calendaring sharing documents --> Name some online transaction you think is not social. http://www.flickr.com/photos/holeymoon/ 10
- 17. Scale is a game changer Caution: large sample sizes 20,000 googlers didn’t expose real issues. Quora moderating overburdened Google Buzz http://www.flickr.com/photos/daguerreotyped/ 11
- 18. Tasks aren’t what you think Activities = goals that emerge and change Effectiveness Efficiency ... aren’t meaningful http://www.flickr.com/photos/jeremybrooks/ 12
- 19. Satisfaction is correlated with task completion Instead: control, engagement But how do you measure control and engagement? What does engagement mean? 13 http://www.flickr.com/photos/litebriteneonstudio/
- 20. Users How do you design effective tasks in this case? /cc @ continuously product reviews that turn into literary projects design your UI Using comments in Facebook for a conversation Hey, where did everybody go? - user-created help page to describe in real time a hack to get around a feature they don’t like Coded messages in Egypt on dating sites Workarounds = hacks Etiquette and norms http://www.flickr.com/photos/juleshabib/ 14
- 21. And we have to be careful not to violate either of those things: Etsy Circles Facebook engagement ring story 15
- 22. Social is about context. And we have to be careful not to violate either of those things: Etsy Circles Facebook engagement ring story 15
- 23. Social is about context. And relationships. And we have to be careful not to violate either of those things: Etsy Circles Facebook engagement ring story 15
- 24. Methods are not robust for understanding context and relationships. 16
- 25. Cultivate polymaths Decision sciences Communication Anthropology sciences Psychologies Sociolinguistics Behavioral economics Organizational behavior Psychobiology Sociology Social neuroscience Social networks 17
- 26. The most successful user research methods for SxD combine field and testing techniques 18
- 27. 19
- 28. Methods 19
- 29. Methods Reviewing online profiles, connections in groups in interviews 19
- 30. Methods Reviewing online profiles, connections in groups in interviews Stories of how you met 19
- 31. Methods Reviewing online profiles, connections in groups in interviews Stories of how you met Multi-user sessions with people who have strong and loose ties 19
- 32. Methods Reviewing online profiles, connections in groups in interviews Stories of how you met Multi-user sessions with people who have strong and loose ties Video diaries with retrospective review 19
- 33. Methods Reviewing online profiles, connections in groups in interviews Stories of how you met Multi-user sessions with people who have strong and loose ties Video diaries with retrospective review Experience sampling with SMS 19
- 34. Takes more time recruiting, prep, session, analysis 20
- 35. Takes strong research design rather than templated, commodity usability testing 21
- 36. Takes deep thinking about questions to answer, and what the evidence will be that you have answers 22
- 37. Takes study of cohesiveness collaboration, connections, networks, relationships Strong and weak ties Dunbar 23
- 38. Where are you with planning? what we asked answered a business question 24
- 39. Who do you ask when you don’t know? How do you make financial decisions was what we wanted to know - which is about context and relationships 25
- 40. Would have told us Who is part of planning and deciding Who is trusted Why these people are important Why they are trusted How planning happens 26
- 41. To build a model that supports 27
- 42. Rethinking user research 28
- 43. Rethinking user research We’re not getting the answers we need. 28
- 44. Rethinking user research We’re not getting the answers we need. Experimenting is limited because we’re pressured to go to market. 28
- 45. Rethinking user research We’re not getting the answers we need. Experimenting is limited because we’re pressured to go to market. We’re looking for things we know about. 28
- 46. We’re missing things we don’t know about. 29
- 47. It’s time to rethink user research for the new (old) social web 30
Editor's Notes
- \n
- \nThe story of the session. \nEnding: she and her father would have done this together. \n\nThis tool is a piece of the social web \n\nNothing unusual about this test. \none person, one computer\npretend to do a task \n\n
- In use at least 20 years \n\nMethod failed Shelly desperately \n\nStarted thinking about this with Buzz\nFTC\n\n
- \nBuzz testing \n\nAfter I blogged about it... \n\n
- We&#x2019;re not sure how to do user research that will tell us what we need to know about relationships online\n Something&#x2019;s not working, and I think it&#x2019;s this: \n
- People simply do operate in real life the way we ask them to behave in usability tests. \n\nWebEx - observe meetings? No \n\nMarriott.com - take into account the others who decide what hotel? No \n\nRetirement planning tool - husbands + household expenses; wives + children about healthcare decisions \n\nEgypt and Libya, using eHarmony and Facebook to pass coded messages: Who knew a social network would *ever* be used that way? \n
- \n(talk) \n
- \n(talk) \n
- \n(talk) \n
- The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
- The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
- The methods were probably *always* less than perfect. \n Considering SxD magnifies the imperfections of user research methods\n \n
- \nSeems that with ambient technology, everything has changed. \n\nBut it hasn&#x2019;t. \n\nThere are 5 factors to user research that I&#x2019;m convinced we should rework, rethink to avoid things like the Buzz and Etsy privacy violations. \n
- \nAnything one person does to change the behavior of another is social. \n\nName one thing you can do online that isn&#x2019;t social. \n
- \n From one person + one computer... \n Large sample sizes won&#x2019;t tell you everything you need to know. The Google 20,000 didn&#x2019;t reveal the privacy problem. Why not? \n Looking at scale another way, let&#x2019;s think about Quora and it&#x2019;s massively user-generated IA. How do you design for that? How do you test for that? How do you do user research that will tell you how to go? \n
- Usability testing and user research methods use tasks - but users don&#x2019;t come to technology with task goals in mind. \n ephemeral, changeable + context, timing, space, what just happened, what happens next\n \n Tasks are probably activities where goals emerge and change depending on context, timing, space, etc. \n
- This is probably the wrong measure. \n We should look at control by the user and engagement. People want to be the boss of their data. Buzz and Facebook&#x2019;s Beacon, and now Etsy&#x2019;s changes take that control away. \n We need to take a hard look at conventional metrics. \n What is effectiveness and efficiency with Twitter and Facebook open all day? What&#x2019;s the task? \n Measures of success don&#x2019;t translate to SxD\n time?\n task completion? \n success? \n
- People use your UI in ways you didn&#x2019;t intend, but that help them do what they want to do.\n Individual workarounds now surface to everyone in a network. \n Etiquette and social norms adjust over time and scale. \n Design is effectively open sourced to the user community\n
- Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
- Onlineness, ambient technology is about context and\n\nwho is zooming whom \n\nWe must not violate the context of the relationships. Otherwise, problems. \n\nClients learn embarrassing purchases - Etsy\nGirlfriend learns source of engagement ring - Facebook\n
- Something built for HCI doesn&#x2019;t cover the expanded view of this one person interacting online and the implications of that. \n\nLab testing & structured field visits are lacking. \n
- \n
- Borrow, borrow from, combine. \n\nI interviewed smart people.... \n\nThis is what they told me: The key is combiningmethods. \n
- \nMet: Twitter road trip\n
- \nMet: Twitter road trip\n
- \nMet: Twitter road trip\n
- \nMet: Twitter road trip\n
- \nMet: Twitter road trip\n
- \nMet: Twitter road trip\n
- \n
- \n
- \n
- \n
- And so, we&#x2019;re back in the lab in Waltham. \n\nShelly is crying as she looks out at her future self. \n\nWe asked a business question. \n
- \nThe business would have been better served by our asking more about Shelly. \n\nWhat&#x2019;s the right question? \n
- \n
- \n
- With our generation old methods, we really only know how to find things we&#x2019;re looking for. \n\n\n
- With our generation old methods, we really only know how to find things we&#x2019;re looking for. \n\n\n
- With our generation old methods, we really only know how to find things we&#x2019;re looking for. \n\n\n
- We don&#x2019;t have good ways of learning about things we don&#x2019;t know about. \n
- \nTime to rework the old standard methods into new remixes. \n
- Thank you. \n