from Jennifer von Draanen

1. +Doing Developmental
Evaluation at the
System Level:
Lessons learned from
an addiction system
change initiative
Tuesday, June 11th, 2013
Jenna van Draanen, Kaileah McKellar, Sanjeev Sridharan

2. +
Agenda
 An overview of the DTFP Evaluation
 DTFP Evaluation Methods
 DTFP Evaluation Results
 Developmental Evaluation at the System Level
 Lessons learned
 Lingering questions for the field of evaluation

3. +
Drug Treatment Funding Program
 Addiction treatment system change
 Health Canada-funded project, delivered through the Ontario
Ministry of Health
 11 system change projects (3 streams)
 Multiple project leads and organizations
 5 year timeline compressed to 18 months

4. +
DTFP
Funding
Selection of
Projects &
Organization
s
Project Implementation
Testing &
validatio
n
Development
of Tools and
resources
Development of
Inter-organizational
Networks
System-level evaluation
& Evaluation capacity
building
Uptake
&Spread
System Change
and practice
improvement
• Increased access
and use of
evidence
• Enhanced
Collaborative
Problem-Solving
Capacity
• Enhanced
Evaluative Culture
Planning for
sustaining
networks
Collaboration & Co-Development
Knowledge Exchange
Sphere of Direct
Control
Collaborative
Activities
Evaluation
Activities
4
Project-level
activities
Sphere of Direct
Influence
Sphere of Indirect
Influence

5. +
Developmental Evaluation of DTFP
 Incomplete knowledge
 1) Spread processes within the system need to be developed
 2) Relationships between different stakeholder groups need to be
strengthened.

6. +
Developmental Evaluation of DTFP
 Purpose
 To bring evaluative thinking to DTFP and to help DTFP to evolve
and create change
 Multiple and flexible methods
 Key-informant interviews, case studies, surveys, capacity-building
workshops, demand mapping, observation, videography,
document review, realist synthesis, and timeline of impact
exercises
 Heavy engagement
 What needs to be developed?

7. +
Results
 Enhancements seen in evaluation and performance
measurement, evidence-informed practice, knowledge
exchange projects
 At the project level
 Case studies giving insight into the system performance

8. +
Results Themes
 Define the system
 Pay attention to dynamics
 Lack of ownership
 Tools themselves will not create change
 Planning for spread
8

9. +
Developmental Evaluation Challenges
 Multiple simultaneous impacts
 Projects having limited sphere of direct control
 Timeline of impact
 Providing feedback in an environment of uncertainty

10. +
Lessons Learned
 Focus on learning
 Developing
relationships
 Rapid feedback
 Redundancy in a
good thing

11. +
Lingering questions

12. +
Thank you
Questions? Comments?
Jenna van Draanen
vandraanenj@smh.ca
www.torontoevaluation.ca