This workshop carried out by Alisa Zomer (MIT GOV/LAB, US), Erhardt Graeff (Olin College of Engineering, US), Luke Jordan (Grassroot, South Africa) & Marci Harris (POPVOX, US) at the Impacts of Civic Technology Conference (TICTeC 2019) in Paris on 20th March 2019. You can find out more information about the conference here: http://tictec.mysociety.org/2019

1. ALISA ZOMER / AZOMER@MIT.EDU
@AZOMER / MIT GOV/LAB
MITGOVLAB.ORG / @MITGOVLAB
[design sprint
workshop]
TICTEC 2019
ENGAGEMENT
METRICS FOR
SOCIAL IMPACT
Nepal (Alisa Zomer)

2. Philippines (Nina McMurry)
today, we are
design sprinting
1. CRASH course on theory + process
2. LEARN from POPVOX + Grassroot
3. SPRINT in small groups
4. SHARE in the report back
Nepal (Alisa Zomer)

3. SOCIAL CAPITAL
// shared knowledge,
norms, rules, expectations
EMPOWERMENT
political efficacy // the belief
that you can influence politics
COLLECTIVE
ACTION
// group of people taking
voluntary action
SOCIAL
MOVEMENTS
// networks with collective
identity; “sustained campaign
of claim-making”; in/formal

4. Designing for
Empowerment with
Better Impact Metrics
Erhardt Graeff, Olin College of Engineering
@erhardt
egraeff@olin.edu
erhardtgraeff.com
The Impacts of Civic Technology Conference (TICTeC)
OECD, Paris, France, 19–20 March 2019

5. impact?

6. How do we measure the impact of civic
technologies in terms of empowerment
and civic learning? And what is the
ethical responsibility of technologists
as stewards of democracy?
erhardtgraeff.com

7. Empowerment: accruing the
knowledge, skills, attitudes, and
habits necessary to have voice
and influence

8. 2016 Stats
• 1M+ users
• 300+ local government partners
• 931K+ issues reported
• 818K+ issues fixed

9. external
political
efficacy
internal
political
efficacy
platform
political
efficacy

10. Questions use a 7-point
Likert scale:
1 - Strongly Disagree
2 - Disagree
3 - Disagree Somewhat
4 - Neither Agree nor Disagree
5 - Agree Somewhat
6 - Agree
7 - Strongly Agree
example questions
internal political efficacy (IPE)
I consider myself well-qualified to participate in local affairs.
external political efficacy (EPE)
People like me DON'T have any say about what my local
government does.
platform political efficacy (PPE)
SeeClickFix and its partner platforms help make my local
government pay attention to what the people want.

11. study design
Invited by email all users “active” on a SeeClickFix
platform in the past year to take the survey and take
the same survey again one month later
Correlated survey responses with platform activity
Invited Survey 1 Survey 2
115493 9166 3482

12. key findings
• Women and those
identifying as other
gendered have lower
efficacy on average
• High IPE people are
likely also early adopters
• Engagement is related
to efficacy and platform
efficacy may predict
future engagement
• Closing issues and
response times really do
matter for a user’s EPE
and PPE

13. response from SCF staff
• Long-held assumptions
made “visible and
tangible” by findings
• Want to find more ways
to show users their
impact, but must
overcome local gov
resistance to more
transparency
• See opportunities for
ongoing measurement
of political efficacy, but
also see opportunities
for doing deep,
qualitative research in
key cities

14. a simple idea.started with
In 2011

15. a simple idea.
The problem:
Staffers overwhelmed
Don’t know if they are hearing from real constituents
Public doesn’t understand the legislative process
What if we…?
Listed all bills online
Required constituents to register with their real address
Delivered input in an easy-to-process format
Made the whole process transparent
POP·VOX
from “vox populi” (voice of the people)
Let’s call it

16. Original mission (2011):
To connect people and government, empower effective participation,
and create a transparent record that informs policymaking and fosters
accountable, responsive governing.
A mission-driven “civic startup”
To create technology that informs and
empowers people and makes
government work better for everyone.
Updated mission (2019)

17. that technology can help us be better
informed, more connected, and make
government work for everyone…
We believe

18. INFRASTRUCTURE for governing
Message Delivery as a Service
serving advocacy vendors
‣ Ensures messages
delivered electronically
‣ Lays groundwork for
better analytics tools
within lawmaker offices
Launched August 2017
Secure dashboards for
legislative staffers
‣ Provides workflow tools
and opportunities for
staffers to connect and
interact
‣ Resource hub to make
work by other good
government groups more
accessible
In beta
‣ Allows local officials to post
agenda items and receive
public input
‣ Provides electronic public
notice system
‣ Familiarizes constituents with
“who represents me” and
local issues
Civic engagement infrastructure
for local governments
Coming Q3-Q4 2019
Civic network
‣ Constituents “follow”
lawmakers, jurisdictions,
and organizations
‣ Provides non-ad social
network for sharing
information on issues and
connecting with others
‣ Allows for constituent-
only public posts
Coming Q2-Q3 2019
Nonpartisan platform for civic
engagement and governing
‣ Lists all pending bills
‣ Organizations create
profile, post positions
supporting or opposing
bills
‣ Constituents create
profile, post positions
supporting or opposing
bills
‣ Messages delivered to
lawmakers
Launched August 2017

19. LOCAL Civic engagement infrastructure for local governments

20. LOCAL

21. LOCAL (Planned) 2019 Pilot Jurisdictions
San Mateo County, CA
Fayetteville, AR
Madison County, TN
Chicago Public Schools
Chicago, IL Philadelphia, PA
(partnership with
OpenAccess PHL)
Oakland, CA
Sebastopol, CA

22. ‣ An observable, transparent
network (allowing anyone to
view activity but limiting
interactions to constituents)
‣ Prioritizing individual control
of data, without algorithmic
manipulation, and require real
names at sign up with
verification
LOCAL

23. The Goals
‣ Increase awareness and understanding of local issues
‣ Increase the number and diversity of participants engaging on
local issues
‣ Improve resident satisfaction with interactions with officials
‣ Increase awareness of “who represents me” and “how local
government works”
Improve tone, truth, and trust in online civic engagement
LOCAL

24. Design Constraints
‣ Must stay within basic POPVOX structure
‣ Most changes will also impact broader POPVOX structure (state
and federal)
‣ We do not currently collect demographic data – should we?
‣ Must work for varied jurisdictions: urban and rural; large city vs.
small city council
LOCAL

25. Problem Statement
LOCAL
How might we measure participants’ understanding of
government processes, comfort engaging, and sense that their
voice matters?

26. Grassroot
TICTeC 2019

27. Background
● Grassroot was founded in South Africa in late 2015
● We enable communities in South Africa to take collective
action and give effect to participatory democracy by
building and deploying simple technology
● Our team is half development, half field-work, working
with partner communities as we build, deploy and iterate
● We’ve reached over 350k people acting 2k+ times / month

28. Platform
● Our platform runs on USSD, Android, responsive web and
now WhatsApp, all syncing to a common back-end
● We cross the digital divide, to allow anyone on any phone,
enabling them to call meetings, ask for volunteers, sign
petitions, take polls, and issue press alerts
● Our ‘net promoter scores’ are similar to Amazon and
Netflix, with 80% of growth by word of mouth

29. Platform Screenshots & Description (1/2)
● USSD allows even feature phones, or smart
phones without data, to set up a community
group, recruit others, call meetings, etc
● Simple ‘join codes’ allow people to sign up
for alerts in large numbers
● Easy to use menu flows heavily user tested,
and available in multiple (5) languages

30. Platform Screenshots (2/2)
● Web app for complex
functions, e.g., votes,
petitions, involving
tens of thousands
● WhatsApp bot under
development, allows
signing petitions,
getting info on sexual
assault clinics, etc.

31. Goal
● We focus on two problems:
1. It’s too costly and difficult for the poor to organize
2. It’s too hard for local leaders to access information they can act
on and connect to each other in a meaningful way
● Grassroot tries to address those problems by creating
simple, mobile-first tools that increase the density and
effectiveness of collective action

32. Design Constraints
● Our traditional definition of impact has been:
● How much more often are people acting together? I.e., how much
has the density of self-organizing among the poor increased?
● How much more often are they getting results? I.e., how much
has the effectiveness of that organizing increased?
● The first we can measure using data and basic surveys
(‘lean data’), but we are finding it is a very poor proxy for
the second (i.e., our theory of change is broken)

33. Problem Statement
● We are pretty sure that our existing tools increase the
density of collective action but not the effectiveness
● That breaks our theory of change, so we are trying new
ideas to create impact, e.g., via short content in
messaging groups connecting organizers
● Need to know quickly if we are having impact, but hard to
measure quickly the impact, esp of changes in capacity

34. Design prompt
● How might we measure the ways in which social
movement organizations grow and build capacity?
● (Constraints: structurally unresponsive government and
organizations with early activity but subsequent decline)

35. Design Prompts
• POPVOX / How might we measure participants’
understanding of government processes, comfort
engaging, and sense that their voice matters?
• Grassroot / How might we measure the ways in which
social movement organizations grow and build capacity?
(constraints: unresponsive government and organizations
with strong initial momentum and steep decline)

36. [thank you for sprinting]
Alisa Zomer / azomer@mit.edu / @azomer
Erhardt Graeff / egraeff@olin.edu / @erhardt
Marci Harris / marci@popvox.com / @marcidale
Luke Jordan / luke.jordan@grassroot.org.za / @grassrootza