This presentation was hold on the ECIS 2013 on the paper "METHODS AND METRICS FOR MEASURING THE SUCCESS OF ENTERPRISE SOCIAL SOFTWARE – WHAT WE CAN LEARN FROM PRACTICE AND VICE VERSA". Abstracht: With Enterprise Social Software (ESS) being increasingly used for internal communication and collaboration, an important question is how their success can be measured. In this paper we contribute to answering this question by investigating which methods and metrics organizations are currently applying for measuring ESS success. To do so, we have conducted 26 interviews with persons responsible for the deployment and use of ESS in their company. We found that measuring ESS success in the studied companies mainly focuses on the analysis of usage. We attribute this to the fact that ESS is primarily characterized by an abundance of user-generated content that can be easily analyzed. At the same time, companies still face difficulties in the evaluation of the business value of the platforms. We attribute this to some characteristics of ESS that we will discuss in this paper. Our study is supposed to provide a comprehensive overview of the methods and metrics applied in practice to measure ESS success.

1. www.kooperationssysteme.de
Methods and metrics for
measuring the success of ESS
What we can learn from practice
and vice versa
Christian Herzog
21st European Conference on Information Systems
June, 2013 Utrecht

2. www.kooperationssysteme.de
• Companies use Enterprise Social Software (ESS) for their internal
communication and collaboration
• The positive impact has been recognized but the measurable effects are still
not obvious and easy to prove
• IT executives are under pressure to justify the investments and to
demonstrate the benefits of ESS by appropriate methods and meaningful
indicators
Motivation
User
Stakeholder
Interested in: Seeing the
benefits of the platform for
their own work (What is in
for me?)
2Christian Herzog
„What kinds of methods and metrics are currently being used to measure
the success of Enterprise Social Software?”
Platform owners
Interested in: Improving the
platform and its usage
(What can we do to improve
collaboration?)
Management
Interested in: Justification for
investment
(Is it worth to sponsor?)
The impact of different stakeholders should be considered during success measurement and the
choice of success metrics.

3. www.kooperationssysteme.de
What do we know?

4. www.kooperationssysteme.de
Success measurement models
4Christian Herzog
• Differences of ESS to other IS (create own content, connect with other users;
mostly used to support unstructured tasks; no clearly defined objective)
 not possible to adopt IS success models without modification
• Studies on IS success are missing detailed success measurement methods
and proposals for the collection of these metrics
• ESS success models provide more specific methods for measuring the success
• ESS studies frequently focus on measuring the usage ( user-generated
content)
• Weblog success model (Du and Wagner,
2006)
• Return On Contribution (ROC) (Muller et
al. 2009)
• Measuring the value of social software
(Cooper et al. 2010)
• Impact of organizational social web site
usage on individual and team
performance (Raeth et al., 2011)
• Theory-Based Measuring Model (Lehner
and Haas 2011)
Enterprise Social Software
• Technology Acceptance Model (Davis,
1989)
• IS Success Model (DeLone & McLean,
1992)
• The construct space for IS effectiveness
(Grover et al., 1996)
• Unified Theory of Acceptance and Use of
Technology (Venkatesh et al., 2003)
• IS-Impact Measurement Model (Gable et
al., 2008)
Information Systems

5. www.kooperationssysteme.de
Our interview
study…

6. www.kooperationssysteme.de
Interview Study
Methodology
Interviews (n=26)
with “Enterprise 2.0”
companies (n=24)
6
Topics / Objectives
What kind of methods do you use in
order to measure the ESS success?
Christian Herzog
• Qualitative approach by semi-structured interviews; interview guide with 32
questions  exploratory assessment
• Companies from German-speaking area (<10 till >400.000 employees);
all have practical experience in the application of ESS
• Conducted by telephone, one was held face to face (Dez. 11– Jun. 12); 35-50 min;
recorded, transcripted and coded
• Discussion in two focus groups (10 and 3 participants) with experts from practice
and research ( reduce subjectivity, discuss interim results)
Do you measure the success of your
ESS?
What metrics are examined in the
success measurement?
Are there any barriers or limitations?

7. www.kooperationssysteme.de 7
Results
A practice-oriented set of methods for measuring ESS
success
Christian Herzog
• Focus of ESS success measurement can change with the maturity of the system
i25: “In first place we really wanted to know if there is activity on it [the platform]. But once if there is regular
activity on it, then you are suddenly interested in how it can bring economic benefits.
[…] interests also move the more mature a system is.”

8. www.kooperationssysteme.de 8
Results
ESS success measurement is applied in practice…
i22: “So when you measure the success of the tools, it means I take my staff and users seriously. I'm interested
in what you are really interested in, what is working, what is not working, what do you actually want and
what should be different.”
i18: “How much do I invest in the measurement? The problem is simple: To have something ... to argue
something waterproof, I need to invest in the measurement so much that I know the profit is almost
gone.”
• No definition of success or success measurement was given to the interviewees
 Different views of success and its evaluation
i13: “…what is difficult in this issue, is first per se the definition: What is success? So right now we are still
based on the criteria for success, which we can measure.“
• Different opinions on the point of when success measuring begins
i11:“Yes surely you can quantitatively measure the usage, but I think what matters at the end of the day,
especially in Enterprise 2.0 projects - is what has changed qualitatively.”
Christian Herzog

9. www.kooperationssysteme.de
• 20 of 24 of organizations collect
usage statistics
i08: “Someone who uses the tool intensively will
produce so much more benefit and also recognize
more benefit than someone who does not use it. So
those are certainly self-reinforcing cycles”
• Objectives influence whether rather
qualitative or quantitative methods
are used
i25: “It depends on the target horizon […]. Using
quantitative methods on a relatively new field of
research is very difficult. Suitable for this are more
qualitative evaluations.”
9
Results
Usage
Christian Herzog
Method Metrics
Content and
usage
analysis
adjusted ideas; intensity of
collaboration; degree of cross-linking
Database
queries /
Log file
analysis
no. of: blog-posts, communitiy spaces,
authors, attachments, visits, edits, log-
ins, user, new users, messages per day,
unique visitors, comments, blog
followers, average comments on a blog
or discussion forum, unique users and
hits per time period, sessions, wiki
pages per day, posts, readers of a post,
praises per post;
development of the use; posts with
most readers; average time per user
per visit; session time
User
Interviews
user requirements for the plattform;
usage behavior; use case validation /
user satisfaction with the processes or
tools; review of the tool
User
surveys
user satisfaction with the plattform;
usage types of the tool; frequency of
use; applicability of the tools;
knowledge of the users about the
possibilities of the tools; satisfaction
with the availibility through the new
tools; self-assesment of the affinity in
using new tools; usability benefit

10. www.kooperationssysteme.de 10
Results
Business Value
Christian Herzog
Method Metrics
Calculation of
costs
savings of travel costs; savings of
hardware costs
Imputed
assumptions /
estimates
opportunity proceeds of projects;
ROI; cost-benefit ratio
Measurement
of email
traffic
email frequency; no. of emails;
correlation of email frequency and
blog posts from one person
Output
measurement
no. of implemented ideas; no. of
awards
Process and
usage analysis
time spent for reading, writing and
answering
User
Interviews
ROI of use cases
User surveys
effort for working with the tools;
individual business value; ROI; saved
money; generating new revenue;
new customer acquisition; percieved
benefits for the organization;
business value
• Difficult and complex; often based on
estimates or subjective perceptions
i16: “[…] In ROI yes. So, this is very difficult for us.
There is talk of reducing travel costs, speed of
innovation, time to market, cost to market, etc.
Unfortunately we cannot yet.
• Malleability makes the measurement
difficult
i25: “The experience is that over time you will find
whole new use cases that were not planned and
which nevertheless generate a benefit.”
• Precise definitions of use cases,
application scenarios or specific
objectives are needed
i25: “[…] the more it goes toward financial metrics,
the more accurate you have to know the
targets.“

11. www.kooperationssysteme.de
So what?

12. www.kooperationssysteme.de
Discussion
12Christian Herzog
Usage
• Usage is of central importance for the
success measurement of ESS as it is driven
by user-generated content
• Almost all companies collect usage statistics
 simplicity of collecting usage data via
analysis tools at low costs
 meets the need for quantifiable success
metrics of an IT manager
 participation in an ESS platform is
voluntary: just happens if users accept
platform and find benefits for their own
work
BUT
• Usage statistics alone have a limited
informative value regarding the company’s
success and must be considered in context
Business Value
• Subjective estimates and imputed
assumptions for collecting monetary
metrics  makes it difficult to obtain valid
data
• With increasing maturity the focus shifts
more to business value
• ESS is mostly used to support unstructured
tasks and can be characterized as
malleable end-user software with no
clearly a priori defined usage scenarios
• Email traffic is seen as an indicator of
information processing workload
BUT moving to ESS does not simultaneously
bring a reduction of labor costs
• Number of different metrics which are not
considered by practitioners

13. www.kooperationssysteme.de 13
Conclusions
• ESS success measurement is applied in practice by a variety of different
methods and metrics
• Different views on ESS success: The dimensions usage and business
value are distinguished and supplemented by a set of methods
• Usage is an important indicator for ESS success ( characteristics of ESS)
but usage statistics require additional interpretations
• Measuring the monetary or economic output is not easy due to the lack
of detailed objectives and missing definitions of processes in specific
use cases as well as due to the malleability
• Further research in terms of a quantitative study and in order to develop
the results towards comprehensive guidelines
Christian Herzog

14. www.kooperationssysteme.de
Thank you very much!
?QuestionsPlease don´t hesitate to contact me if
you have any further questions
afterwards:
christian.herzog@unibw.de
Christian Herzog
Alexander Richter
Melanie Steinhüser
Uwe Hoppe
Michael Koch
Cooperation Support Center Munich
Institute for software technology
Bundeswehr University Munich
Institute for Information Management
and Information Systems
University of Osnabrück
14