What is developmental evaluation and when would you use it?

1. Developmental Evaluation
Fostering collective learning & effective
collaboration through developmental evaluation
07/13/12 Christopher Wilson & Associates 1

2. Traditional Evaluation
• Traditional evaluation is typically designed to
assess whether an intervention has reached its
intended target either through a formative or
summative evaluation.
• Formative Evaluation - improves
– provides interim feedback to various initiative
stakeholders, such as funders, staff, or community
residents during the implementation period to ensure that
the implementation aligns with the initial plan.
• Summative Evaluation – validates
– provides evidence of progress towards program & policy
goals over a period of time by looking at outcomes and
impacts over the longer term. Obviously, the outcomes and
impacts must be achievable within the time frame being
evaluated.
07/13/12 Christopher Wilson & Associates 2

3. Traditional Assumptions
• The connection between ends and means
is well established
• The situation being evaluated remains
stable
• Key variables are known and controllable
• Outcomes are predictable and measurable
07/13/12 Christopher Wilson & Associates 3

4. Traditional Orientation
• Solutions are centrally defined, then evaluated in
field tests for their validity and then ‘best
practices’ are implemented broadly
• Evaluation is conducted in the middle (formative)
or at the end (summative) to test programs and
policies
• Works well in linear, simple and complicated
social situations where conditions are stable and
standardized in many different locales
• The evaluator is an independent, objective
assessor
07/13/12 Christopher Wilson & Associates 4

5. Two Other Evaluation Needs
• One focuses on supporting the group’s social learning in the face of
complexity and uncertainty
– This evaluation assesses many things: whether the initial plan continues
to fit with the current assessment of needs and circumstances; whether
there are unforeseen factors involved; how an innovation might be
traditionally evaluated; to acquire more information from
experimentation
– This is particularly prevalent in areas of ‘wicked’ problems where there
is a need to learn both ‘ends’ and ‘means’ at the same time. Feedback
is obtained to ascertain whether the right thing is indeed being done.
Partners can then collectively reflect on this feedback to determine the
ongoing fitness between problem understanding and response.
• The other need of evaluation promotes the ongoing contingent
cooperation of partners
– This assessment is conducted to validate the commitments, trust, and
participation of an initiative’s partners to each other, especially when the
evaluation tools are used as a means of supporting and empowering the
ownership of stakeholders to be part of defining and delivering a
solution. It helps satisfy the caveat “I will if you will.”
• These will not be met by traditional evaluation approaches
07/13/12 Christopher Wilson & Associates 5

6. Developmental Evaluation -
when to use it
• Adaptation of an existing programs and ongoing
development to an evolving context
• Application of principles and best practice
transferred from one context to a new one
• Exploring impacts of small innovations and
policies in preparation for larger interventions
using formative or summative evaluation
• Providing feedback on major systems change
• Rapid response to crisis or sudden change
Source: Michael Quinn-Patton, Developmental Evaluation, Guilford Press, New York, NY, 2011
07/13/12 Christopher Wilson & Associates 6

7. DE Assumptions
• The connection between ends and means is not well
understood. There is no known solution.
– This is being learned via the intervention
• The situation being evaluated will continue to change &
evolve
– The situation continues to respond to a number of factors not all
of which are under the control of the evaluators
• Key variables may not be known nor is their potential
influence on outcomes
– Intervention is a means of learning what key variables might be
• Outcomes are not directly measurable, predictable
– Interveners are likely to be surprised by their experience
• The system under consideration may respond in
unpredictable ways to small local changes
• Evaluation is not a tool for validation but a tool of
‘learning while doing’ through experimentation,
prototyping and serious play
07/13/12 Christopher Wilson & Associates 7

8. DE Approach
• Map system dynamics, partner assumptions,
and interdependencies
• Develop mechanisms to provide quick
feedback on group decisions and assumptions-
in-use
• DE must be flexible to changes in collective
understanding and to changes in the nature of
the intervention and be able to adjust the
evaluation accordingly
• Emphasis on evaluation to support the
judgement of participants
• Attention to unanticipated results
07/13/12 Christopher Wilson & Associates 8

9. DE Evaluator Role
• Evaluator seen as co-creator with participants, providing
them with the information they require to make sense of
what’s going on
• The evaluator is part of a team whose members
collaborate to conceptualize, design and test new
approaches in a long-term, on-going process of
continuous improvement, adaptation & intentional change.
• The developmental evaluator helps partners to ‘pay
attention to the ‘right’ things’
• The evaluator's primary function is to foster various group
conversations contributing evaluative questions, data and
logic, and thereby facilitate social learning, contingent
cooperation and data-based decision-making.
• Evaluation is a stewardship function: encouraging reality
testing; cross learning; joint ownership; and shared
commitment
07/13/12 Christopher Wilson & Associates 9

10. Typical Results from DE
• What are the principles, factors,
relationships that people need to pay
attention to?
• How could these principles, factors,
relationships be applied in a local context?
• How can real time feedback be structured?
• How does that feedback foster collective
learning?
• How does feedback contribute to effective,
ongoing collaboration and teamwork?
07/13/12 Christopher Wilson & Associates 10

11. “Theory of Change” Orientation
• Instead of linear “logic models”, DE makes use of
“theory of change” models”
– The presumed building blocks and pathways of change that
are required to bring about a given long-term goal.
• Evaluation conducted to support meaning making,
reframing, belonging, trust, commitment and the
conceptualization of a “theory of change”
• Evaluation conducted to promote conversations that
challenge the mental models that contribute to the
status quo and to possible transformation
• Theories of change are reality tested and evaluation is
conducted to support shared learning
• Evaluation conducted to support the process of
collaboration
07/13/12 Christopher Wilson & Associates 11

12. For further information please contact:
Chris Wilson
Christopher Wilson & Associates
Tel: (613) 355-6505
chris@christopherwilson.ca
www.christopherwilson.ca
07/13/12 Christopher Wilson & Associates 12