1. Improving the Quality of Services
Through Measurement and Feedback
Supported by the National Institute of Mental Health grant (RO1 MH068589) and
a grant from the Leon Lowenstein Foundation .
Leonard Bickman, Ph.D., Susan Douglas Kelley, Ph.D., and Susan Casey, MSW
www.CFSystemsOnline.com
www.TheNationalCouncil.org

2. Outline
• The challenge and opportunity
• The theory of change
• Contextualized Feedback
Systems and its components
• Preliminary results of evaluation
• Benefits
• Plans
1

3. The Challenge:
A Quality Chasm Exists for Mental
Health Services
2

4. “The greatest obstacle
to discovery is not
ignorance - it is the
illusion of knowledge”
Daniel Boorstin, 1984 Librarian of Congress
3

5. Little Research Support for the
Effectiveness of Services in
Community Settings
& Little Change in a Decade
• “The findings to date offer little support for the
effectiveness of conventional clinical intervention for
young people.” (Weisz & Jensen, 1999, p. 133)
• Some common elements of effective practice have
been observed in treatment as usual but not in the
depth and breadth needed for effective treatment.
(Garland, Bickman, & Chorpita, 2010)
• But lack of evidence doesn’t necessarily mean lack
of effectiveness 4

6. Little Accuracy About Client
Progress & Focus of Treatment
• No correlation between standardized measures and
clinicians’ perceptions of progress (Love et al., 2007)
• Difficulty predicting and detecting worsening of
symptoms and functioning over the course of treatment
(Hannan et al., 2005; Hatfield et al., 2009)
– Yet, 89% of clinicians said they would know based on their
clinical judgment
• Little agreement on target problems by parents and
therapists, and children and therapists (Hawley, KM & Weisz,
JR. (2003).
• Chance Agreement on Level of Care Assignment
(Bickman, L., Karver, M., & Schut, L. J. A. (1997).
5

7. Typical Community Based Care has
Significant limitations
• “The findings to date offer little support for the
effectiveness of conventional clinical intervention for
young people.” (Weisz & Jensen, 1999, p. 133)
• Some common elements of effective practice have
been observed in treatment as usual but not in the
depth and breadth needed for effective treatment.
(Garland, Bickman, & Chorpita, 2010)
• But lack of evidence doesn’t necessarily mean lack
of effectiveness
6

8. Clinical Intuition and Regulations
are Insufficient to Produce Good
Outcomes
• Fifty years of research does not support
relying only on clinical judgment for effective
practices
• Reliance on presumed standards of quality
such as licensing and accreditation may also
impede development of effective services
• Unmonitored “single shot” training on EBTs
are not effective
7

9. Systematic and
Accurate Feedback Can
Improve Outcomes
8

10. “Thus, use of patient monitoring and
feedback in routine practice is
imperative.
The focus on the individual rather than
the average patient empowers the
patient and encourages dialogue about
progress, the direction of treatment and
achievement of treatment goals”.
Newnham & Page. (2010). Bridging the gap between best evidence
and best practice in mental health. Clinical Psychology Review, 30,
127–142 (p.138).
9

11. Strong Support for Feedback
Concept
• Measurement and feedback are the core of
all management and learning theories.
• Thousands of studies outside of mental
health show that improvement is minimal
without measuring performance and
providing feedback.
• Direct feedback occupations show
improvement with experience. However,
clinician experience alone is not a good
predictor of client outcomes.
10

12. Providence Service Corporation delivers home and
community based social services to government
beneficiaries and privatized social service projects.
Providence Service
Corporation

13. .
Providence Overview
PRSC Service
Offering
Home and
Community Management
Based Services
Services
Foster Care
Services
12

14. • Providence Service Corporation was aware of the
need for feedback and was committed to
implementing a quality enhancement initiative
• In 2004 we began a collaboration with Vanderbilt
University to combine their efforts to create an
evidence-based feedback system
13

15. Providence Core Values
• Community-based and Multi-systemic
Services
• Identifying and Building on Strengths
• Local Viability that is Nationally
Supported
• Respectful Organizational Culture
• Best Practice Models
• Cultural Diversity
Providence Service
Corporation

16. Quality Enhancement
Initiative
• 12 states, 38 sites, 287 clinicians
• 836 youth 11-18 years of age and their
caregivers
• 10,431 client weeks of Providence provided
services
15

17. CFS - A Concurrent, Systematic
Monitoring and Formative Feedback
Practice Improvement Tool
Practice Without Feedback
Does Not Lead To Improvement
16

18. CFS is a Practice Improvement
Strategy
• is part of treatment – provides an opportunity for brief
reflection at the close of a session
• is primarily designed to support clinicians – provides
guidance for those moments when little seems to be working
• supports supervisors – helps supervisors identify areas
where clinicians need extra guidance to ensure they feel
confident
• is flexible – optional custom measures, schedules, reports
can be tailored for office workflow and QI initiatives
• can be used for administrative and reporting functions –
provides tools for leadership to become or remain successful
at meeting the needs of their funders and clients
17

19. CFS Theory of Change is on
Two Levels
• The individual or psychological level
• The group or organizational level
18

20. Key Domains of CFS
• System Development
• Measurement
• Feedback
• Training in Common Factors
• Comprehensive Support
19

21. Dashboard-driven System
Dashboards are designed for clients and
Easy to navigate
Highlights common actions by user
caregivers, clinicians, supervisors,
administrative assistants, program
directors, etc.
20

22. Sample Clinical Alerts
New = most recent alerts;
not yet viewed
Link to the feedback
report containing the
Pending = oldalert; Archive alerts
alert;
not yet viewed
21

23. Sample Client Login
After logging in, client
is presented with
questionnaire to be
completed.
22

24. Barry’s Sample Case – Clinician Judgment is
Necessary - What Do You Notice?
23

25. What do you see here?
24

26. Weeks
1 2 3 4 5 6 7
25

27. Measurement: The Peabody Treatment
Progress Battery (PTPB)
• Domains include processes and outcomes
• Includes strength-based measures
• Brief – each takes 15 seconds to 2 minutes
• Reliable
• Show convergent and divergent validity
• Information from youth, clinician, and caregiver
• Easy to score and interpret
• Sensitive to change
• Available in English and Spanish
• Free for anyone to use – now in second edition
• http:peabody.vanderbilt.edu/ptpb
26

28. CFS Measures Map onto the Relevant
Questions
• How is the youth doing overall?
– Severity of symptoms and functioning (y, cg, cl)
– Life satisfaction (y)
– Hope (y)
• How is treatment going with the youth?
– Therapeutic alliance (y, cl)
– Motivation for treatment (y)
– Counseling impact (y)
– Service satisfaction (y)
• What are important caregiver issues?
– Caregiver strain
– Life satisfaction
– Therapeutic alliance
– Motivation for treatment
– Service satisfaction
27

29. Organizational Measures
• Services, staffing, any data already
collected; experience with other initiatives,
etc.
• Culture, climate, adaptability, leadership,
organizational learning, & initial
perceptions of CFS
• Current attitudes toward CFS,
barriers/supports, value of reports, self-
efficacy, & goal commitment
28

30. CFS Consultation and
Training Model
• CFS is a tool for transforming your agency
into a learning organization
• Ongoing support to
– Promote sustainability through local ownership
– Contextualize CFS operations and feedback to
your agency’s needs and resources
– Apply feedback to inform clinical sessions,
supervision, program planning, professional
development, and more
– Empower practice improvement leadership at
all levels of your agency
29

31. Analyses of Current Project
• Randomized experiment in 33 sites -
Analyses ongoing
• Randomly assigned sites to feedback and no
feedback conditions
• Started with most direct question: Does
feedback appear to affect clinician behavior?
• The answer to this question does not depend
upon the effectiveness of the treatment
provided
30

32. CFS Feedback Influences
Clinician Behavior in Sessions
• Cases selected where clinician had not previously reported
a problem with youth symptoms and functioning in a
specific domain (e.g., behavioral issues, emotional
problems, friend/peer issues)
• When a client or caregiver alert was present for that
domain
– Viewing clinical feedback reports associated with addressing that
domain in a future session
– More feedback viewed more times addressed
– Not explained by more reporting of problems by clients or
caregivers
31

33. Feedback Reduces Symptoms
and Improves Functioning
• Used HLM random effects with new clients,
controlled for nesting within sites, and number of
sessions.
• Youth respondent (ages 11-18): 340 clients
• Caregiver respondent: 308 clients
• Clinician respondent: 294 clients
• There were no significant differences at baseline
between the two groups for any respondent
• Results showed Feedback Group improved more
than the non-Feedback Group according to all
three respondents.
32

34. Modeled Values of
Youths’ SFSS p = 0.025
80
75
No Feedback
70
Youths’ SFSS in CBCL Units
65
60
55
50
45
40
35 Feedback
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Months in CFS
33

35. Modeled Values of
Caregivers’ SFSS p = 0.05
80
No Feedback
75
Caregivers’ SFSS in CBCL Units
70
65
60
55
50
45 Feedback
40
35
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Months in CFS
34

36. Modeled Values of
Clinicians’ SFSS p = .0005
80
No Feedback
75
Clinicians’ SFSS in CBCL Units
70
65
60
55
50
45 Feedback
40
35
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Months in CFS
35

37. What Can CFS Do
For Clients and Caregivers?
• Provide a way to raise issues they may not feel
comfortable addressing aloud or in the presence of
caregiver or youth
• Reassure them that the clinician is paying attention to
their needs and is contextualizing services to their
concerns
• Demonstrate that effective services are a priority
• CFS is a tool that takes the consumer’s voice beyond
the session and into all aspects of care
36

38. What Can CFS Do
For Clinicians?
• Better identify thoughts, events, and feelings of clients and
caregivers
• Identify successes and problem areas to focus sessions
• Provide consistent and systematic feedback of clinically
relevant information
• Inform treatment planning and goal setting
• Focus sessions on clients’ issues to show that you are
attending to their concerns
• Check on how well treatment is working
• CFS is a tool that enhances the clinician’s ability to tailor
treatment as it progresses
37

39. What Can CFS Do
For Clinical Supervisors?
• Provide clinical data on each case’s progress and
areas that seem to be improving or declining
• Provide a tangible framework from which to guide
clinical supervision that is independent of the
clinician
• Provide the needed information for a continuous
quality improvement effort that facilitates
accountability
• Most of all…CFS is a tool that provides the
supervisor with needed resources to promote
evidence-based practice
38

40. What Can CFS Do
For an Agency?
• Serve as evidence to funding agencies that
quality of services and effectiveness are a priority
• Manage clinical services on their impact as well
as their cost
• Provide data on the effectiveness of services, how
clients are improving, the typical problems being
encountered, and where needs are not being met
• CFS is a tool that supports overall practice
improvement
39

41. What CFS Can Do for the
Funder?
• Determine who is being served and their
progress
• Provide a Quality Assurance mechanism
• Compare services and agencies on cost
effectiveness to optimize value
• Reduce costs of some regulations and
documentation activities such as
excessively detailed treatment planning
and progress notes
40

42. Plans
• Continued partnership between Vanderbilt and Providence in
training, development, and research
• Continue software development
• Use with adults as well as children and youths
• Continue development of new measures
• Proposal pending to test CFS with different types of
feedback with youth with substance abuse problems
• Integrate CFS with other mental health EBTs (new NIMH
grant for Functional Family Therapy combined with CFS)
• Develop web-based “treatment” intervention with CFS
(Australia Project)
• Use CFS in primary care settings to bridge medical and
mental health fields
41

43. Web Site: www.CFSystemsOnline.com
42