Transformational Change for Evaluation of SDGs

InDEC Discussion for EvalSDGs
Jakarta, 1 November 2019
Sponsored/Supported by:Organized by:

• BACKGROUND
• PERSPECTIVES:
• HOW DO INDONESIAN EVALUATORS SEE AND
PERCEIVE?
• HOW DO INDONESIAN EVALUATORS THINK
ABOUT THE METHODS FOR CAPTURING
TRANSFORMATIONAL CHANGE?
• LEARNING FROM EXAMPLES:
• LEARNING FROM IDEAS 2019:

Financial
Audit
Performance
Audit
Impact
Evaluation
Learning
Focused
Evaluator as
Accountant / Auditor
Evaluator as
Researcher
Evaluator as
Facilitator
Evaluation
as choice
Evaluation as
compliance
Evaluation
as culture
LogFrame, Indicators, & Result-Based
Management
Outcome Harvesting and
Impact Hunting
2010
TRANSFORMATION
2019

Registered Member of InDEC: 78 pax
• Indo-MONEV listserv: ± 600 subscribers
• Facebook Page: 2,244 followers
Pasive Member
• International Development Organisation (Bilateral,
Multilateral, INGOs.)
• Government officials
• Individual/Independent Practitioners
• CSOs – Local NGOs
• Academia & Research Centre
Members from:

DRIVERS &
AGENT OF
CHANGE
STRUCTURA
L CHANGE
COME WITH
RESISTENCY
CHANGED
RELATIONS
(CONNECTIONS)
WITH EDUCATION/
AWARENESS
RAISING
CHANGE OF
FAITH/BELIEF
ENGAGEMENT /
PARTICIPATION+
COLLABORATION
PERMANENT
COMPREHENSIV
E
POLITICAL
ECONOMY
DEMOCRACY
SUSTAINABLE
NOT
BUSINES AS
USUAL
Based on online survey & discussion
FOCUS ON
PROCESS??
CONTINUOUS
LEARNING
STAKEHOLDER
DYNAMICS
INTEGRATED
PLANNED/
BY DESIGN
REVOLUTIONARY
STRATEGIC

(OVER)
REGULATION /
BUREAUCRACY
INERTIA
(INSTITUTIONA
L)
CLOSE/
NARROW
-MINDED
WANT TO
MAKE OTHER
HAPPY WITH
STATUS QUO
BAD
COMMUNICATION
LACK of
KNOW-
HOW

MIXED – METHODS
PARTICIPATORY, GROUP TECHNIQUE
 EMPOWERMENT EVALUATION?
REALIST EVALUATION? THEORY-
BASED EVALUATION?
PROBLEM DRIVEN ITERATIVE
ADAPTATION?
STORIES OF TRANSFORMATIONAL
CHANGE (WITH MSC)? CASE STUDY
APPROACH?

•
•
•
•
•
•
•
•

• innovative technologies are piloted, with the potential to demonstrate new ways of doing
things, which could lead to wider and sustained change
INNOVATION
• approaches which have proved successful in one location are made widely available and
lessons on their usefulness are credible and shared widely
EVIDENCE OF EFFECTIVENESS IS SHARED
• Countries (government, communities, private sector, and others) have the capacities and
capabilities necessary to bring the change about
CAPACITY AND CAPABILITY CAN BE INCREASED

• Good ideas piloted by the ICF are replicated by others in the same country and more widely
REPLICABLE
• The costs of climate action are reduced to the point that acting on climate is a sensible decision
for commercial firms and private individuals. These cost reductions may need to be steep
enough to overcome behavioural inertia;
LEVERAGE / CREATE INCENTIVES FOR OTHERS TO ACT
• Need for the change is agreed locally and the process is locally owned. For widespread changes,
notably changes to the patterns of development, this will require high level political buy-in and
broader support from across society;
POLITICAL WILL AND LOCAL OWNERSHIP:

• Interventions (such as national, sectoral or regional programmes) that have sufficient reach
to achieve institutional and policy reform, or drive down costs of technology deployment
AT SCALE
• Change is likely to be sustained once ICF support ends
SUSTAINABLE
• Ultimately, many truly transformational changes will require a critical mass, to overcome
political, market and other sources of inertia.
CRITICAL MASS

LAND USE & TENURE CONFLICT
MANAGEMENT / RESOLUTION
REGULATORY AND GOVERNANCE
SUSTAINABLE FINANCING
PRIVATE SECTOR AND
COMMUNITY-LEVEL PRACTICES
LEARNING, COMMUNICATION &
ADVOCACY

Local partners’ stories
selection
PROJECT LEVEL
Quality Assurance
FLAG EMU
LEVEL
Final
Selection
LEARNING
EVENT
3 Stories is Expected from
each of 6 Lead
Organization
1-3 Stories is
expected to be
selected

CONTRIBUTOR/
INFLUENCING
FACTORS
WHAT WHO
WHAT
HAS CHANGED?
Domain/Area
BEFORE AFTER
What did it
looks like?
Since when?
Until when?
Where?
Where else?
What does it look
like now?
When di started
to change?
WHY IS IT
SIGNIFICANT?

CRITERIA Score
6)
Narrative/Description to support the scoring
(with examples, reference or basis of evidence)
1. INNOVATION
2. EVIDENCE OF EFFECTIVENESS
IS SHD
3. CAPACITY & CAPABILITY CAN
BE INCREASED
4. REPLICABLE
5. LEVERAGE / INCENTIVE
6. POLITICAL WILL AND LOCAL
OWNERSHIP
7. AT SCALE  Critical Mass
8. SUSTAINABLE

Stories Harvested:
1 on Social Forestry / Community-Based
Forestry Management
1 on Small-Holders preparing for ISPO
1 on Sustainable Financing – engaging
financial institutions
2 on Innovation for
livelihood/agroforestry
2 on Conflict management/resolution
2 on Access to Information and Joint
Review for Advocacy
1 Story Selected:
Institutional Set-Up for Conflict
Management/Resolution at Sub-National
Level

• THE FRAMEWORK FOR BUILDING,
DOCUMENTING AND ASSESSING A STORIES OF
TRANSFORMATIONAL CHANGE IS USEFUL IN
PUSH EVERYBODY TO THINK BEYOND A
PROJECT/PILOT
• BUT SCORING IS NOT USEFUL  THE WHOLE IS
GREATER (OR LESSER) THAN THE SUMS OF ITS
PARTS  NEED TO LOOK AT SYNERGY, AND
HOW EACH CRITERIA LINK AND INFLUENCE THE
OTHER CRITERIA
• EACH CRITERIA IS A BIG CONCEPT, THAT IS NOT
EASY TO GRASP – ESPECIALLY BY LOCAL NGOS
• SCALABILITY TOWARDS CRITICAL MASS AND
SUSTAINABILITY IS IMPORTANT
• WITH APPROPRIATE SCALING STRATEGY, SCALING
AGENT AND SCALING DIRECTION

•
•
•
•
•
•
•
•
•
•
•

SMART-FISH FOCUS ON : 3 VALUE CHAINS, 6 COMPONENTS (2014-2019)
SIX COMPONENTS:
1. Public-Private Dialogues
2. Centre of Excellence
3. Education Program
4. Traceability
5. Sustainability-Eco labelling
6. Promotion

Indonesian Evaluators need to explore
more the conceptual elements that
construct TRANSFORMATIONAL
CHANGE  need to be sensitive and
observant to the elements and inject
them in the discussions  need to be
aware of different tools that could help
them in capturing and influencing
transformation.
Development Actors
(mainly the Program
Implementer – local
level) need to be
exposed to the concept
 need more awareness
Indonesia Need more
examples of
transformational change
from local practice, that
could give more clarity on
what it means that relate to
the complex context of
Indonesia  Local
ownership to the
Transformation Change

TRANSFORMATION IS NOTHING NEW
 CARBON
 WAR
 INCARCERATION
 APARTHEID
 CENTRALIZED
 WORLD IS FLAT

MANY DISCIPLINES, MANY PERSPECTIVES –
AN OPPORTUNITY












TRANSFORMATIONS ARE USUALLY NOT
CONTROLLED OR FORESEEN

BASIC TYPES OF TRANSFORMATIONS



7 TransformAbilitiesessentialobligatory
©Paradigm-shiftAbility
ScalAbility
SustainAbility and resilienceAbility
©MultidimensionAbility
Social-ChangeAbility
ComplexAbility and adaptAbility
ReciprocAbility

3 TYPES OF CHANGE
Daniel Kehrer
Incremental:
More of the same
(size, number…)
Pictures:
lifecycle.onenessbecomesus.com
https://www.butterfliesandmoths.org

3 TYPES OF CHANGE
Daniel Kehrer
Reform:
Different ‘look’,
system changes…
Pictures:

3 TYPES OF CHANGE
Daniel Kehrer
Transformation:
New identity, purpose,
paradigm…
Pictures:

Types of Change – examples
Types of Change
Examples Incremental Reform Transformation
Waste
Less waste
(waste regime)
Waste recycling
(waste regime)
Cradle to Cradle
(no waste regime)
Racism
Reduction of
discrimination
(racial segregation
regime)
More rights for the
discriminated
(racial segregation
regime)
Same rights for all
(no racial segregation
regime)
Mitigation of
Climate
Change
Increasing energy
efficiency
(lower carbon regime)
Promoting renewable
energies
(low carbon regime)
Abandoning fossil
energy, using 100%
renewables
(carbon neutral
regime)

INCREMENTAL CHANGE VS. PARADIGM / REGIME SHIFT
From: Joachim Hafkesbrink
https://www.slideshare.net/johaf/technological-paradigm-shifts-updated

Von: http://mediaimpact.org/production/callaloo
Paradigm-shiftAbility
Questions / Indicator Verification
Does the project question,
recent systems, paradigms,
regimes or major narratives
by its goal frame, indicators
and activities, offering
fundamentally different
ones?
yes/no
+ description

“
”
TransformAbility 2:
ScalAbility©

WINDOWS OF OPPORTUNITY
ICAD. (2017).
Transformational
Change Guidance
- First Draft.

SCALABILITY
Will the intervention be able to contribute reaching a
certain mainstream (piloting vs. scaling)?
yes/no
+ description
How broad is the scope (e.g. single technology / local
social norm up to technological regime / sets of societal
norms)?
Levels of scope to be
operationalized
What would be the highest system level that can be
addressed (niche to global paradigm / narrative)?
System levels e.g. according to
Figure 5
In how far can the biggest possible actor coalition be
involved?
e.g. X of the relevant actors
etc.
In how far does the intervention link up with other similar
or complementary interventions on various system levels?
e.g. number, quality and system
levels of connected
interventions

“
”
TransformAbility 3:
SustainAbility and
resilienceAbility©

Transformation as a reaction to disturbances, disrupting a system and
leading to a new adaptedness and resilience (from Nelson, Adger, and
Brown 2007).
RESILIENCE PERSPECTIVE

From: Müller et al. 2014
RESILIENCE PERSPECTIVE

SustainAbility and resilienceAbility
Does the intervention have measures in place to
weaken the resilience of established regimes,
paradigms and narratives (e.g. abandoning of
subsidies) and to strengthen the resilience of
alternative ones (e.g. investments and policies creating
new path dependencies or improving societal
acceptance)?
yes/no
+ description (including
expected tipping
points)
Are measures in place to strengthen social resilience of
the new system by addressing various forms of
justice, like between social milieus, local to global,
intergenerational, historical, mutual agreement,
subordinate value (utilitarism), distributive justice…?
yes/no
+ description
+ number of different
forms of justice
Are there measures in place to facilitate deliberation
and agreement about which forms of justice to
address?
yes/no
+ description

Environment
Social sphere from individual to (global) society, involving:
values, cultures, social norms, attitudes, memes, beliefs, knowledge, behaviors, practices
Politics,
legal
norms,
incentives,
institutions
Economy, financial market,
production and consumption
patterns
Science,
technology,
infrastructure
Most of our
interventions are
here
Daniel Kehrer
DIMENSIONS AND KEY ASPECTS OF TRANSFORMATION

MULTIDIMENSIONABILITY
How many of the dimensions (Figure 6) are deliberately
addressed and how many actor groups / societal milieus within
them?
Listing and explanation of various
forms of diversity like richness,
proportional abundance… (compare
diversity indices)
How do actors perceive the credibility, salience (or relevance)
and legitimacy of processes and their outcomes at the
multidimensional interfaces?
Regular assessments of these
criteria
To which degree can various forms of knowledge (e.g.
scientific, local, traditional, tacit, regulatory knowledge or goal,
transformation and system knowledge) be integrated?
Description
To which degree can knowledge be balanced with various
values and norms (positivist vs. constructivist perspective,
scientization and technocracy vs. politicization)?
Description
How strong is the facilitating capacity for integrating different
dimensions (e.g. using change agents, boundary workers,
facilitators, boundary organizations…)?
Description including the amount
of competences and resources
invested

3 DIMENSIONAL ROOM OF TRANSFORMATION AND SOCIAL CHANGE
Daniel Kehrer
Social levels Individual Family Friends Milieus Nation Continent Global Society
Social layers values cultures social norms attitudes memes beliefs knowledge behaviors practices
Transformation Phases Pre-development Take-off Acceleration Stabilization
Organizations Institutions, Networks

IndividualFamilyFriendsMilieusNation
Institutions,
Networks
Organisations
values social norms attitudes memes beliefs knowledge behaviors practicescultures
Global
Society
The case 4 social change:
Coordinates of Social Change 4 Transformation
Sociallevels
Social layers
Transformative
Intervention

INFLUENCER THEORY
Change – Stellschauben (Grenny et al., 2013) adaptiert von Sarah Lubjuhn
Motivation Ability
Personal
Personal motivation:
Do people actually want to act
differently?
Make undesired things desirable
Personal skills:
Do people have the ability and strength
to act differently?
Help others to learn
Social
Social motivation:
Are there other people close who
motivate or inhibit certain behavior?
Use group dynamics
Social skills:
Do other people offer support,
information, etc. that is needed at times?
Involve social support
Structural
Structural motivation:
Do structures enable desired behavior or
do they discourage?
Develop systemic measures
Structural skills:
Which things, structures etc. empower
people to act in certain ways?
Use or develop respective structures
1 2
3 4
5 6

Open for cognitive communication
Open for affective communication
Daniel Kehrer
Precontemplation
Unaware of the problem
Contemplation
Aware of the problem and of
the desired behavior change
Preparation
Intends to take action
Action
Practices the desired
behavior
Maintenance
Works to sustain the
behavior change
Relapse
Fall back to
old patterns of
behavior
The Stages of Behavior Change
Learning cycle & upwards spiral – one stage at a time

SOCIAL-CHANGEABILITY
How deliberate and complete is the social change approach towards
anchoring innovations within society?, i.e.
See below
How many social layers and levels of social change (Figure 8) are
addressed?
Number and description
How many target groups are addressed? Same
How differentiated and adequate are target groups addressed (using
diverse strategies like formal and informal education, co-production,
positive deviance, campaigns, infotainment, entertainment education…)
Are the strategies based on thorough target group research?
Description
Target group analysis (qualitative and
quantitative, including citations of
world views etc.)
Does the intervention go beyond the ‘knowledge to action’ paradigm
and e.g. addresses various conditions of social change:
1. Knowing (awareness, information, knowledge...)
2. Capacity / ability (Skills, action knowledge, experience...)
3. Motivation (Values, social norms, beliefs, attitude, routines...)
4. Enabling environment / structural conditions (Legal and societal
norms, physical/geographical structures, incentives...)

TRANSFORMATION DEALS WITH ‘SUPER COMPLEX’ SYSTEMS
Unordered
Ordered

COMPLICATED VS. COMPLEX
Conventional aid thinking
(complicated world)
New perspectives
(complex world)
Systems and
problems
System and problems are closed, static,
linear systems; reductionist (parts would
reveal the whole)
Systems are open, dynamic, non-linear
systems far from equilibrium. Macro
patterns emerge from micro behaviors
and interactions
Human
agency
Individuals use rational deduction;
behavior and action can be specified
from top-down; perfect knowledge of
future outcomes is possible
Heterogeneous agents that mix
deductive/inductive decisions, are
subject to errors and biases, and which
learn, adapt, self-organize and co-evolve
over time
Social
structures
Formal relations between actors are most
important; relationships are ahistorical
and can be designed; actors can be
treated as independent and atomized
Interpersonal relationships and
interactions matter in form of culture,
ties, values, beliefs, peers. Informal
matters, relationships are path
dependent and historical
The nature
of change
Change is direct result of actions;
proportional, additive and predictable;
can hold things constant; simple cause
and effect
Change is non-linear, unpredictable, with
phases of transitions
Ramalingam, Ben. 2013. Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World. OUP Oxford.

COMPLEXABILITY AND
ADAPTABILITY
In how far is the intervention adjusted to dealing with
(super) complex problems? i.e.
How strong will the understanding of the complexity
involved be (e.g. using tools for analyzing complexity /
interlinkages)?
Description of methods and
tools
In how far is the intervention designed and implemented
in an iterative and adaptive way (process vs. output
orientation, i.e. the degree to which processes and
methods are emphasized whereas goals and indicators
are tentative)?
Assessment of importance of
processes vs. goals and
indicators
How flexible and open for unforeseen goals and
solutions as well as unpredictable behavior of the entire
system is the intervention and its design (expectation of
control and classical vs. adaptive management)?
Description of flexibility for
changes of goals, indicators
and means

TRANSFORMATIONS ARE ON THEIR WAY OR ANTICIPATED
•
•
•
•
•
Daniel Kehrer

THE CASE FOR DELIBERATELY ORGANIZING SCIENCE, SOCIETY, POLICY
INTERFACES







Daniel Kehrer

RECIPROCABILITY
In how far does the intervention build on and promote reciprocal relationships vertically and horizontally between actors from
different dimensions of transformation within and across countries (interdependencies vs. unidirectional dependencies and
relationships)? i.e.
In how far do these actors share and expect accountability and
responsibility from each other (e.g. between funding source, implementer
and beneficiary)?
E.g. direction of reporting obligations…
In how far are these actors able to learn with and from each other within
and across countries (co-productive mode vs. knowledge transfer)?
E.g. mode of project design and regular iterative
exchange
In how far do these different actors benefit in a just and fair way (‘just
transition’)?
Explicit deliberation on justice…
In how far is the intervention able to question power structures,
hierarchical cultures and top down processes?
E.g. Mode of exchange across hierarchies
Which levels of participation are reached between these actors from up
to joint decision-making?
E.g. participation strategy
Extent to which different types actors / target
groups can choose a transformation and ist nature
How much equal capacity and influence do these actors have in defining
and framing world views, goals, indicators, means, metrics and processes?
Capacity building, co-productive modes…
In how far does information flow freely between these actors? E.g. use of shared platforms…

Transformational Change for Evaluation of SDGs

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Transformational Change for Evaluation of SDGs

Semelhante a Transformational Change for Evaluation of SDGs (20)

Último

Último (20)

Transformational Change for Evaluation of SDGs