The document analyzes discourse around nonprofit evaluation from three social worlds: civil society, science, and management. It finds that a combination of these languages has led to a growing emphasis on measurable impact and managerial criteria for nonprofits. The study analyzes online discourse to determine whether these languages co-exist, combine, or if one colonizes the others. It identifies key terms from each language and finds evidence of an "interlanguage" combining aspects of all three that is used by most entities discussing nonprofit evaluation.

1. Co-exist, Colonize, or Combine?
Accounting for patterns of discourse on nonprofit evaluation
Carrie Oelberger, Achim Oberg, Karina Kloos, Valeska Korff, Woody Powell
Academy of Management
Cultural (Ac)counting: The rise of formal organization in social and cultural domains
7 August 2012, Boston, MA

2. Co-exist, Colonize, or Combine?
• Our study analyzes the current discussion of
nonprofit evaluation with respect to contact between
three different social worlds:
– Civil Society
– Science
– Management
• Each of the three worlds have played varying roles
throughout the history and development of the
current nonprofit sector

3. Co-exist, Colonize, or Combine?
• The confluence of financially-driven managerial criteria,
combined with the progressive era’s lasting focus on
measurable impact, has led to a growing instrumental
orientation for the nonprofit sector.
• At the same time, there is concern that what we measure
will influence the shape of civil society.
• Our study analyzes these conversations and asks whether
scientific and managerial language is co-existing,
colonizing or combining with the more traditional
associational language of civil society.

4. Three Distinct
Social Domains

5. Civil Society
Trust Compassion
Social Change Justice
Participatory
Transparency
Commitment Three Distinct
Social Domains

6. Civil Society
Trust Compassion
Social Change Justice
Participatory
Transparency
Commitment Three Distinct
Data Social Domains
Methods Assessment
Survey
Quantitative
Framework
Indicators
Evaluation
Science

7. Civil Society
Trust Compassion Management
Social Change Justice Performance
Participatory Impact
Transparency
Commitment Three Distinct Efficiency
Data Social Domains
Outcomes
Methods Assessment M&E
Survey Lessons Learned
Quantitative
Framework Effectiveness
Indicators Best practice
Evaluation
Science

8. Research Questions
Who is contributing what
to nonprofit evaluation discourse?
1. Who is participating in online discourse regarding
nonprofit evaluation?
2. What kind of discourse patterns form when different
languages come into contact: co-existence,
combination, or colonization?
3. What organizational features influence an entity’s
discourse patterns?

9. Research Questions
Who is contributing what
to nonprofit evaluation discourse?
1. Who is participating in online discourse regarding
nonprofit evaluation?
2. What kind of discourse patterns form when different
languages come into contact: co-existence,
combination, or colonization?
3. What organizational features influence an entity’s
discourse patterns?

10. Website Discourse
• Website discourse of entities talking about nonprofit
evaluation
– Open to the public, the information presented is not
tailored to one particular audience
– Purposeful self-representations

12. Webcrawler Methodology
• Snowball sampling approach:
Websites are added based on number of
incoming references by identified members
of the relevant sample.
• Inclusion/exclusion decision:
Collective analysis of website content to
appraise extent of contribution to non-
profit evaluation discourse
• Website “Scraping”:
The entire text from each website is
“scraped” into our off-line database to
enable analysis of discourse patterns

13. Research Questions
Who is contributing what
to nonprofit evaluation discourse?
1. Who is participating in online discourse regarding
nonprofit evaluation?
2. What kind of discourse patterns form when different
languages come into contact: co-existence,
combination, or colonization?
3. What organizational features influence an entity’s
discourse patterns?

14. Sample Characteristics
• Our methodology produced 419 highly interconnected
entities involved in nonprofit evaluation
• Entities were then coded for core organizational
features including:
– Demography
– Institutional Properties
– Resources and Constituencies

15. Sample Characteristics
Demography
• Age: 2 - over 200 years old
• Size -> Scale: one person blogs - 250,000 employee global organizations
• Size -> Scope: local, regional, national and international
Institutional Properties
• Form: 56% nonprofits, 13% for-profits, 3% branches of state or national government, 14%
transnational organizations, and 14% non-organizational forms
• Activity: evaluation, funding, consulting, networking, media, advocacy, research, social
services
Resources and Constituencies
• Revenue Streams: foundation grants, government grants, corporate funding, individual
donors, fee-for-services, membership fees, endowment, public equity market and taxes
• Target Audiences: social service beneficiaries, donors, nonprofits, for-profits,
(transnational) government and the public

16. Research Questions
Who is contributing what
to nonprofit evaluation discourse?
1. Who is participating in online discourse regarding
nonprofit evaluation?
2. What kind of discourse patterns form when different
languages come into contact: co-
existence, combination, or colonization?
3. What organizational features influence an entity’s
discourse patterns?

17. Co-exist
• Do entities co-exist in this field of nonprofit
evaluation, but retain distinctly separate discourse –
associational, scientific, or managerial – about how
to approach nonprofit evaluation?
– Similar to the “salad bowl” metaphor of immigration,
where individuals remain monolingual with their
traditional language.

18. Combine
• Do we observe a combination of vocabularies and
the emergence of some sort of shared language
around nonprofit evaluation?
– Similar to the “melting pot” story of immigration, where
we would find entities drawing equally on a combination of
all of three languages, dissolving the boundaries that
previously existed.

19. Colonize
• Has science or management colonized the nonprofit
evaluation debate and crowded out the less powerful
domain of civil society and its related associational
discourse?
– Similar to the classic story whereby immigrants “colonize”
indigenous languages.

20. Analyzing Discourse:
Developing Vocabulary
Keywords are “significant, indicative words in certain forms of thought” that
make up a distinctive, domain-specific vocabulary (Williams 1969: 14)
Iterative process of identifying keywords to develop a vocabulary of nonprofit
evaluation:
• Mined the discourse on the websites and consulted experts
• Created word clusters that resulted in three different social domains
• Fleshed out clusters based on extant knowledge of domain
• Co-occurrence analysis to affirm validity of clusters
Process resulted in 196 terms categorized in 3 clusters

21. Analyzing Discourse:
Clustering Terms
196 terms categorized into the following 3 clusters:
Social Domain Discourse
Civil Society Associational
Scientific Research Scientific
Management: Managerial
Business & Government

22. Analyzing Discourse:
Counting Keywords
Examined each website to calculate the relative
percentage of each of the three languages:
associational, scientific, and managerial
• For example, if there were:
– 50 occurrences of managerial terms
– 30 occurrences of scientific terms
– 20 occurrences of associational terms
• The entity would be 50% managerial, 30% scientific, and
20% associational

23. Relative language use
thesroinetwork.org
swtgroup.net
cerise-microfinance.org
mullagofoundation.org
efqm.org
corostrandberg.com
ladb.org
sphereproject.org
wdi.umich.edu
usaid.org
fbheron.org
rainforest-alliance.org
organizationalresearch.com
eandco.org
compasspoint.org
arnova.org
iisd.org
robinhood.org
broadfoundation.org
joycefnd.org
gistfunders.org
gmfus.org
americanprogress.org
hfpg.org
alliance1.org
aecf.org
seechangeevaluation.com
ncvo-vol.org.uk
All entities involved in nonprofit evaluation
nonprofitquarterly.org
unstats.un.org
onphilanthropy.com
gatesfoundation.org
worldofgood.org
usip.org
cofinteract.org
africagrantmakers.org
unwomen.org
komen.org
sunlightfoundation.com
350.Org
cafonline.org
Observed Distribution of Discourse

24. Relative language use
thesroinetwork.org
swtgroup.net
cerise-microfinance.org
mullagofoundation.org
efqm.org
corostrandberg.com
ladb.org
sphereproject.org
wdi.umich.edu
usaid.org
fbheron.org
rainforest-alliance.org
organizationalresearch.com
eandco.org
compasspoint.org
arnova.org
iisd.org
robinhood.org
broadfoundation.org
joycefnd.org
gistfunders.org
gmfus.org
americanprogress.org
hfpg.org
alliance1.org
aecf.org
seechangeevaluation.com
ncvo-vol.org.uk
All entities involved in nonprofit evaluation
nonprofitquarterly.org
unstats.un.org
onphilanthropy.com
gatesfoundation.org
worldofgood.org
usip.org
cofinteract.org
africagrantmakers.org
unwomen.org
komen.org
sunlightfoundation.com
350.Org
cafonline.org
Does a coherent, shared language exist?

25. Does a coherent, shared language exist?
Interlanguage (Galison 1997)
• pidgins and creoles that emerge in the interstices between
social domains
• facilitates local communication across social and linguistic
boundaries
• enables coordination of action across place, time and context

26. Does a coherent, shared language exist?
To investigate whether a coherent interlanguage exists,
we examined whether there are keywords that both
occur:
• frequently across the majority of entity websites
• in combination with keywords from the other two “parent”
languages
This identified 24 (out of 196) terms that represent an
interlanguage on nonprofit evaluation.

27. Operationalizing Interlanguage
• Nonprofit evaluation
interlanguage spans the Methods Impact
Assessment
boundaries between the Accountability
Outcomes
Lessons learned
domains of civil Quantitative
Evaluation
Effectiveness
Performance
society, scientific Framework Transparency
38% What works Best practice 39%
research, and management Indicators Certification
Survey
• Approximate 2:1 ratio Trust
Evidence
Social change
implies that there is slight Commitment
Participatory
colonization by managerial Transparency 23%
and scientific languages

28. Including our Interlangauge in the analysis
• These 24 keywords become
our interlanguage Methods Impact
Assessment Outcomes
Accountability Lessons learned
• If we remove them from their Quantitative Effectiveness
Evaluation Performance
“parent language” and create Framework Transparency
What works Best practice
a fourth language cluster an Indicators Certification
interesting picture emerges Survey
Trust
Evidence
Social change
Commitment
Participatory
Transparency

29. Observed Distribution of Interlanguage
Relative language use
Entirety of entities involved in nonprofit evaluation

30. Conclusion
• A diverse array of entities discuss nonprofit evaluation
• Entities are multi-lingual, combining three distinct “parent”
languages
– Furthermore, there is extensive use of a coherent interlanguage
– Managerial and scientific terms outnumber associational in
interlanguage by a factor of 2:1
– Associational terms remain highly relevant, if less
standardized, across all entities

31. Thank You!
Gerhard Richter
Bach (4)
1992
300 cm x 300 cm
Oil on canvas
Catalogue Raisonné: 788