1. Rating Scales for Collective Intelligence in Innovation Communities Why Quick and Easy Decision Making Does Not Get it Right > Christoph Riedl IvoBlohm Jan Marco Leimeister Helmut Krcmar

2. 1. Problem Setting

6. So, there are large data pools… How do you select the best ideas?

7. 2. Theory Background

10. Organization’s absorptive capacity is limited (Cohen et al. 1990; Di Gangi et al. 2009)

12. Dimensions of Idea Quality An idea‘s originality and innovativeness Ease of transforming an idea into a new product An idea‘s value for the organization An idea‘s concretization and maturity Source: [1, 2, 3]

13. 3. Research Model

14. Research Model Judgment Accuracy Rating Scale H1+ Rating Satisfaction H2+ H1: The granularity of the rating scale positively influences its rating accuracy. H2: The granularity of the rating scale positively influences the users' satisfaction with their ratings.

15. Research Model User Expertise Judgment Accuracy H3a Rating Scale H1+ Rating Satisfaction H2+ H3a: User expertise moderates the relationship between rating scale granularity and rating accuracy such that the positive relationship will be weakened for high levels of user expertise and strengthened for low levels of user expertise.

16. Research Model User Expertise Judgment Accuracy H3a H3b Rating Scale H1+ Rating Satisfaction H2+ H3b: User expertise moderates the relationship between rating scale granularity and rating satisfaction such that the positive relationship will be strengthened for high levels of user expertise and weakened for low levels of user expertise.

18. Multi-method study

19. Web-based experiment

20. Survey measuring rating satisfaction of participants

22. Participant Demographics N = 313

23. Participant Demographics

24. Participant Demographics Additional Slide (not partofthepresentation)

25. Screenshot of system

26. Research Design Promote/Demote Rating 5Star Rating ComplexRating

27. So much for the data space and its attributes. Next, we have to think about who our users are and what they want to do. All lifelogging applications are first of all about 5. Results

28. Correct Identification of Good and Bad Ideas

29. Error Identifying Top Ideas as Good and Bottom Ideas as Bad

30. Rating Accuracy (Fit-Score)

31. Factor Analysis of Idea Quality Additional Slide (not partofthepresentation)

32. Participants’ Rating Satisfaction

33. ANOVA Results N = 313, *** significant with p < 0.001, ** significant with p < 0.01, * significant with p < 0.05

34. ANOVA Results Post-hoc comparisons: Complex rating scale leads to significantly higher rating accuracy than promote/demote rating and 5-star rating (p < 0.001)

37. Regression Results There is no direct and no moderating effect of user expertise. The scale with the highest rating accuracy / rating satisfaction should be used for all user groups.

38. Correlations of Expert Rating and Rating Scales Additional Slide (not partofthepresentation)

39. 6. Contribution

40. Limitations Expert as base-line Forced choice

44. Design and test of a model to analyze the influence of the rating scale on rating quality and user satisfaction.

46. Simple scales have low rating accuracy and low satisfaction  Design recommendations for user rating scales for idea evaluation

47. Rating Scales for Collective Intelligence in Innovation Communities > Christoph Riedl IvoBlohm Jan Marco Leimeister Helmut Krcmar riedlc@in.tum.de twitter: @criedl

48. Image credits: Title background: Author collection Starbucks Idea: http://mystarbucksidea.force.com/ The Thinker: http://www.flickr.com/photos/tmartin/32010732/ Information Overload: http://www.flickr.com/photos/verbeeldingskr8/3638834128/#/ Scientists: http://www.flickr.com/photos/marsdd/2986989396/ Reading girl: http://www.flickr.com/photos/12392252@N03/2482835894/ User: http://blog.mozilla.com/metrics/files/2009/07/voice_of_user2.jpg Male Icon: http://icons.mysitemyway.com/wp-content/gallery/whitewashed-star-patterned-icons-symbols-shapes/131821-whitewashed-star-patterned-icon-symbols-shapes-male-symbol1-sc48.png Harvard University: http://gallery.hd.org/_exhibits/places-and-sights/_more1999/_more05/US-MA-Cambridge-Harvard-University-red-brick-building-sunshine-grass-lawn-students-1-AJHD.jpg Notebook scribbles: http://www.flickr.com/photos/cherryboppy/4812211497/ La Cuidad: http://www.flickr.com/photos/37645476@N05/3488148351/ Theory and Practice: http://www.flickr.com/photos/arenamontanus/2766579982 Papers: [1] Amabile, T. M. (1996). Creativity in Context. Update to Social Psychology of Creativity. 1 edition, Westview Press, Oxford, UK. [2] Blohm, I., Bretschneider, U., Leimeister, J. M. and Krcmar, H. (2010). Does collaboration among participants lead to better ideas in IT-based idea competitions? An empirical investigation. In Proceedings of the 43th Hawaii Internat. Conf. System Sci. p. Kauai, Hawai. [3] Dean, D. L., Hender, J. M., Rodgers, T. L. and Santanen, E. L. (2006). Identifying quality, novel, and creative ideas: Constructs and scales for idea evaluation. Journal of the Association for Information Systems, 7 (10), 646-698.