Presented by Kurt Luther (GVU Center, Georgia Tech) in the session "Create, Donate, Collaborate" at the 2010 ACM Conference on Supporting Group Work (GROUP 2010), in Sanibel Island, GA.
Exploring the Future Potential of AI-Enabled Smartphone Processors
Success Factors in Online Creative Collaboration
1. Kurt Luther¹
Kelly Caine²
Kevin Ziegler¹
Amy Bruckman¹
¹Georgia Tech
²Indiana University
Why It Works (When It Works):
Success Factors in
Online Creative Collaboration
1
8. Kurt Luther¹
Kelly Caine²
Kevin Ziegler¹
Amy Bruckman¹
¹Georgia Tech
²Indiana University
Why It Works (When It Works):
Success Factors in
Online Creative Collaboration
8
10. Background
Online Creative Collaboration
People meet and communicate primarily over the Internet
with the shared goals of working together and creating
new artifacts.
a.k.a Yochai Benkler’s peer production
Canonical examples are Wikipedia, open-source software
(OSS) projects
10
11. Background
Newgrounds
Oldest and largest host of Flash-animated movies and
games (founded 1995)
Top 1,000 most-visited websites (Alexa.com)
2,200,000 registered members
180,000 member-uploaded movies and games
Solo projects: single-author, individually made
Collabs: multi-author, collaboratively made
Accepted movies and games can attract 800,000-
1,000,000 views or more
11
13. Background
Collabs on Newgrounds
Hundreds organized each year on Newgrounds discussion
forums
Each collab has 1 leader and 2 to 50+ artists
Leader starts a collab thread on the forum, describes the
idea in the first post, tries to recruit artists
Production is modularized, not specialized
Multi-author system allows leader to “co-author” up to
10 collaborators
13
14. Traditional definitions of success unsuitable because OSS
projects are ongoing and iterative (no concept of
“completion”)
Success definitions tend to be complicated, often multi-
dimensional
English & Schweik’s 6-part classification
Crowston et al.’s 18 measures of success
Many attempts to identify success factors
Self-selection for tasks, charismatic leader, meritocratic or rational
culture, modular division of labor, use of collaborative
technologies…
Success Definitions & Factors
Open-Source Software
14
15. Success defined as completing a collab; incomplete
collabs are “failed” or “dead”
Ratings, popularity, learning often viewed as secondary
to completion
Successful collabs…
get exposure (via hosting on Newgrounds), increase the
creators’ Batting Average,
and may generate sponsorship income
What factors influence success in collabs?
Success Definitions & Factors
Collabs
15
16. Study 1 (Qualitative)
Methods
Semi-structured interviews with 17 animators
14 interviewed via phone, 3 via email
Most participants were leaders, some were artists, but all
had collab experience
Ranged from novice to expert, amateur to pro
Represented six countries
16
17. Planning
& Structure
Study 1 (Qualitative)
Results & Discussion
17
Reputation
& Experience
Communication
& Dedication
Collab just started
Collab in progress
18. Successful leaders have a clear
goal and a plan for achieving it
Collaborations fail because people get an
idea in their head and they can’t accurately
convey their vision to the people they want
to participate. And the collaboration
doesn’t appear very appealing because the
person doesn’t seem organized. –J.R.
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
18
19. Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Technical specifications (“specs”)
deemed especially important
You have to give people really concrete
boundaries in terms of how to put their
movie together—not the creative part, but
the technical aspects of it—in order for it to
succeed to begin with. –L.C.
Study 1 (Qualitative)
Results & Discussion
19
20. Study 1 (Qualitative)
Results & Discussion
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
P1: Collabs with initial planning
and structure, especially technical
specifications, are more likely to
be successful.
Initial planning and structure:
Number of planning/structural elements
provided by leader in first post of collab
thread
20
21. Leaders with standing in the
community are more likely to
succeed
It definitely helps when the leader is
someone who already has the respect of the
other users… When a popular artist asks
people to get involved, they already assume
it’s going to have a certain level of quality
to it. If it’s some unknown person who
wants you to be involved in a collab, you
don’t know what the quality is going to be
like. –T.F.
Reputation
& Experience
Planning
& Structure
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
21
22. P2: Animators who are well-
known in the community are more
likely to lead successful collabs.Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Well-known:
Number of forum posts, number of awards
won, and Batting Average
Study 1 (Qualitative)
Results & Discussion
22
23. Leaders should be skilled and
experienced animators, first and
foremost
I think it also is important that people have
a substantial amount of experience on their
own with Flash and animation before they
take the leadership role in a project of that
nature. Just because it leads to a lot of
frustration when you know more
than the person who’s essentially going to
be giving you orders. –L.C.
Reputation
& Experience
Planning
& Structure
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
23
24. P3: Animators who have
experience with Flash and past
collabs are more likely to lead
successful collabs.
Experience with Flash:
Number of solo projects hosted on
Newgrounds
Experience with past collabs:
Number of collabs hosted on Newgrounds
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
24
25. Successful leaders are
communicative, dedicated, and
personally invested
A collaboration can’t succeed without
[communication]. At least, not a complex
one. A movie director doesn’t walk onto the
set and say, “I want you to do this, this,
and this. I’ll be back in a few hours to
check on you.” A movie director is
constantly
there giving his guidance. –J.R.
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
25
26. Artists look for collabs that
generate high activity and
discussion in the community
The thing about collabs, it’s all about
community. If only one person’s interested,
they’ll just regularly bump up the thread,
but after awhile they’ll just give up or
somebody just comes along and tells them,
“No, this is rubbish.” But if it’s a really
good idea, loads of people will take part
and the thread will just keep being bumped
and bumped up to the top page. –R.W.
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
26
27. P4: Collabs whose members
frequently communicate are more
likely to be successful.
P5: Collabs whose leaders
frequently communicate are more
likely to be successful.
Communication frequency:
Number of replies by each leader/member
in collab thread
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 1 (Qualitative)
Results & Discussion
27
28. Study 2 (Quantitative)
Methods
Scraped 137,328 forum threads on Newgrounds
Randomly sampled 892 threads into 3 bins:
high activity (>50 replies), medium (10-50),
low (<10)
Collected ID, title, reply count for each thread; date,
author & message for each reply in thread
Collected completed solo projects & completed collabs,
awards won, Batting Average for each thread creator
28
29. Study 2 (Quantitative)
Methods
1) Content analysis of collab thread first posts
(themes, specs, gatekeeping policies, authorship
policies, restrictions, leader contact info)
2) Categorization of collab thread outcomes
(either success or failure)
3) T-tests comparing first posts
(successful vs. failed collabs)
4) T-tests comparing leaders
(successful vs. failed collabs)
5) T-tests comparing activity dynamics
(successful vs. failed collabs)
29
30. Study 2 (Quantitative)
Results & Discussion
87.4% of collabs failed
Only 112 of 892 collabs succeeded
Success is difficult—what makes it more likely?
30
31. Study 2 (Quantitative)
Results & Discussion
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Leaders of successful collabs put
more planning/structural elements
in the first post
Themes** 0.94 vs. 0.73
Specs** 3.74 vs. 2.03
Gatekeeping policies* 0.39 vs. 0.25
Authorship policies** 0.29 vs. 0.11
Restrictions* 0.63 vs. 0.40
Leader contact info** 0.83 vs. 0.52
*p < 0.05, **p < 0.01
31
32. Planning
& Structure
Reputation
& Experience
Communication
& Dedication
P1: Collabs with initial planning
and structure, especially technical
specifications, are more likely to
be successful.
Study 2 (Quantitative)
Results & Discussion
Supported
32
33. Communication
& Dedication
Leaders of successful collabs have
more completed solo projects and
collabs, more awards, more forum
posts, and higher Batting Averages
Past solo projects** 2.80 vs. 1.70
Past collabs** 5.42 vs. 3.35
Awards won** 1.60 vs. 0.60
Forum posts* 986 vs. 586
Batting Average** 3.17 vs. 2.92
*p < 0.05, **p < 0.01
Planning
& Structure
Reputation
& Experience
Study 2 (Quantitative)
Results & Discussion
33
34. P2: Animators who are well-
known in the community are more
likely to lead successful collabs.Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 2 (Quantitative)
Results & Discussion
Supported
P3: Animators who have
experience with Flash and past
collabs are more likely to lead
successful collabs.
Supported 34
35. Leaders of successful collabs post
333% more replies
All members of successful collabs
post 228% more replies
Posts per day**
(leaders only)
4.24 vs. 1.86
Posts per day**
(all members)
24.22 vs. 7.28
*p < 0.05, **p < 0.01
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 2 (Quantitative)
Results & Discussion
35
36. P4: Collabs whose members
frequently communicate are more
likely to be successful.
P5: Collabs whose leaders
frequently communicate are more
likely to be successful.
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Study 2 (Quantitative)
Results & Discussion
Supported
Supported
36
37. Comparisons & Contrasts
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
There are only two ways we know of to make
extremely complicated things. One is by
engineering, and the other is by evolution. –
Danny Hillis
Collabs are engineered
The leader sets expectations early on and
reduces conflicts later on by devising a
plan and sticking to it.
OSS projects evolve
The leader’s problem definition is open to
distributed iteration; members organize
themselves.
37
38. Comparisons & Contrasts
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Successful collabs are run by
leaders with technical skills and a
record of success.
Successful OSS projects have
similar attributes, but members
also emphasize “soft skills” (e.g.
charisma).
A certain base level of design and coding skill
is required… but it’s far from the whole story.
–Eric Raymond
38
39. Comparisons & Contrasts
Planning
& Structure
Reputation
& Experience
Communication
& Dedication
Successful collabs and OSS
projects are both characterized by
frequent communication among
leaders and all members.
Collab leaders go down with the
ship, while OSS leaders are
vulnerable to “forking.”
A dissatisfied [OSS] community… can always
leave and start again under new leadership. –
Joseph Reagle
39
40. Implications
Online creative collaboration expanding to new domains
(e.g. video, music, animation)
Which lessons learned from Wikipedia and OSS transfer,
and which don’t? Why?
Clear similarities and differences, just within two
domains (collabs vs. OSS)
Are broader principles of online creative collaboration
possible?
40
41. Implications
Possibility 1: Skilled, experienced leaders
In both collabs and OSS projects, leaders with skill and experience in
their domain are more successful.
Possibility 2: Frequent communication
Both successful collabs and successful OSS projects had leaders and
members who communicated frequently.
Caveat: Must test these principles in other domains, to
avoid assumptions of generalization
41
43. Thank You
Co-authors:
Amy Bruckman, Kelly Caine, Kevin Ziegler
Study participants
Georgia Tech ELC Lab members
Newgrounds staff
NSF CreativeIT award #0855952
Flickr photos by ian_munroe, macinate, and kmndr used with permission
43