This presentation was delivered on Monday, May 16, 2011, as part of the free monthly webinar series from Friends for Youth's Mentoring Institute. One of the most important aspects of running a successful mentoring program is the evaluation of both participant outcomes and the quality of your service delivery. But, despite its importance, program evaluation in a mentoring context is often misunderstood, misapplied, or even feared. This webinar will attempt to take some of the mystery and insecurity out of mentoring program evaluation and goals, featuring a live demo of the Oregon Mentors Evaluation Instrument Toolkit, a collection of downloadable evaluation instruments, surveys, scales, and questionnaires designed to provide youth mentoring programs with increased access to reliable evaluation tools. This comprehensive site was developed by two organizations and our webinar features key staff from each: Mike Garringer, Resource Advisor with the National Mentoring Center, and Celeste Janssen, Program Director with Oregon Mentors. Designed for program coordinators and other staff who may be new to evaluation concepts and resources or for those who would like a refresher, the presentation will address aligning evaluation with the true goals and activities of your program, explore new tools that can help you collect meaningful data, and review some common pitfalls that can complicate mentoring program evaluation. This presentation was featured as a pre-conference course at Friends for Youth’s January 2011 annual mentoring conference. Presented as a webinar, you will have the opportunity to learn more about free or inexpensive tools available to your program in case you missed this session. This webinar was also offered by the Mentoring Partnership of Minnesota’s Quality in Action Webinar Series on Wednesday, May 4, 2011.

1. Mentoring
Program
Evaluation and
Transforming lives through
the power of mentoring Goals
Friends for Youth’s
Mentoring Institute
May 2011 Webinar
In collaboration with the Mentoring
Partnership of Minnesota

2. Webinar Logistics:
Adding Comments
• All attendees muted for best sound
• Type questions and comments in the question box;
responses will either be direct to you or shared with all
attendees
• “Raise your hand” to be unmuted at end to ask question
live during webinar
• Works best for telephone or headset-to-computer
connections
• Please monitor background noise

3. Panelists
Sarah Kremer, ATR-BC Michael Garringer
Program Director Resource Advisor
Friends for Youth’s April Riordan Education Northwest/ Celeste Janssen
Mentoring Institute Director of Training & National Mentoring Center Program Director
Partnerships Oregon Mentors
Mentoring Partnership
of Minnesota

4. • Slides posted to
SlideShare
• Recording of webinar will
be posted to
http://
www.friendsforyouth.org/
Webinars.html
• Link and brief survey
included in follow-up
email

5. Recommended Best Practices
• Mission Statement and Long Range Plan
• Recruitment Plan for Mentors and Mentees
• Customized Orientations for Mentors and Mentees
• Eligibility Screening/Application Process for Mentors and Mentees
• Training Curriculum for Mentors and Mentees
• Matching Strategy
• Monitoring Process
• Support, Recognition, and Retention Component
• Relationship Closure Steps
• Evaluation Process
California Governor’s Mentoring Partnership, 2002

6. Friends for Youth’s Impact
• Nearly 1,800 matches
• 88% overall success rate of mentors and mentees
reaching one year together
• Program Coordinator contacts
– 1x/week for first 3 months + 2x/weeks for remainder
(Senior Friends)
– 1x/month (Junior Friends + parents/caregivers)
• Junior Friend Evaluation outcomes
– Developed by Cindy Sipe, Ph.D.
– Evaluation by William Lapp, Ph.D.
– Over 10 years of data on matches

7. 5 Outcome Areas
• Increase in involvement in school
(attendance, truancy, attitude, grades,
suspensions)
• Reduce risk behaviors (drug/alcohol
use, police involvement, probation)
• Increase positive behaviors
(community service, new activities)
• Increase opportunities to influence
future potential (office visits, talking
about college & career with adult)
• Improve self-concept (trusting
relationships, self-assuredness, self-
reliance, handle unexpected
problems)

8. Friends for Youth’s Impact
After 12-month period, youth made significant changes
• Improved school attendance
– 81% reported not skipping school at all
– 75% reported avoiding suspensions
• Improved attitude toward school
– 100% showed positive change in at least 1 aspect
– 69% showed positive change in 2 or more aspects
• Reduction in risk behaviors
– reduced self-reported incidents of police involvement (13% vs.
28%), stealing (4% vs. 23%), hurting others (12% vs. 32%)
– alcohol & drug use decreased in time & showed delay in starting
use
Lapp 2009 (2007-2009 results)

9. Friends for Youth’s Impact
After 12-month period, youth made significant changes
• Increase in positive behaviors
– 87% participated in community service project vs. 35%
baseline
– significant increase in new activities
• Improved self esteem
– 95% reported increase in at least 1 aspect of self-esteem
– 63%reported increase in all 3 aspects
• Increase in number of trusting relationships with
family, peers, & other adults
– nearly 90% reported positive overall change
Lapp 2009 (2007-2009 results)

10. Friends for Youth
Transforming lives through the power of mentoring .
Youth-in-Need
“My mentor Friends for Youth participants are:
Lisa doesn’t * mostly middle-school age Societal Cost
scream at me * mainly low income and two-thirds
are below the poverty level California spends
or hit me. She
helps me with * often from single parent homes $234,000 per year for
stu , and she teaches me things. * sometimes have a parent who is each youth in a Juvenile
I’ve stopped hurting people. I’ve incarcerated Justice Facility.
stopped beating up people.” * all at-risk of not reaching their full
potential and may be susceptible to:
Using
Mentee Pedro For each youth who
Substance Abuse avoids dropping out of
The very foundation of mentoring Violence & Gang Involvement school for a life of drugs
is that if a caring, concerned adult is Academic Failure and crime, the country
involved in the life of a young person, These youth have little adult support, saves between $2.6 to
the youth will be more likely to and now schools are cutting back $5.3 million.
Evaluation
become a successful adult. after-school programs.
Measurable Impact of Our Program
Friends for Youth puts
!"#$%&'(&$)#*'+",-%.)'/012"13%#' signi cant emphasis on
program evaluation and
Results
!"#$"%&'(")*+,#-."+"%&)/0"#)1)2"'#)3%)&4")5#3"%67)8-#)2-9&4)!#-(#'+)
its impact on our youth.
(""#
Ten years of statistical
anaylsis con rm the
'"# positive impact of our
program.
&"#
%"# “In conclusion, the Friends
for Youth program is
$"# functioning at an extremely
high degree of e ciency
!"# and has shown a positive
# # # # # C# # F# B;# 0# # F# improvement...as well
,+
3 6;0 20 +3 02 ,3 ,52 *,; -/ ,1 28 235
How0 Mentoring 52Helps;F#.26*
01
23
,5#
9:
+
*2
#>
67
-
?6
+@
7A #)
@/
-#
.;
27
B
#)
5A
#I
=)
A +3
0M
B NO9
0;2
20
,B,
as helping to address
negative in uences that
./ 8 + -# ;7
3 +, /3
, 2I B2C2 2B7 2#.2 02
#L
-# -2 16 +, @0 )* #H +B ;#L
could otherwise cause
+, 57 =48 )* 8 F#7 70 27
)* #) ,3 #./ +8 G6 0#E 7623 62 43
56 problems in [mentees]
02 ;7 +,
- #E /0 43
5
27 )* D+ 05 6#? immediate lives and
56 7,3 D, K2
43 < J2 possibly into adulthood.”
William Lapp, Ph.D.
“My mentor Mike came to conferences and spoke to my
teachers about my potential and about how smart and
promising I am. It was nice to have someone tell them that
I am a smart kid.” Mentee Diego

11. Overview
• Introduction & Best Practices
– Elements of Effective Practice
– Evaluation Plan & Logic Model
• Tips for Selecting Instruments
– Oregon Mentors Evaluation Toolkit
• Important Rules for Administering
Instruments
• Additional Resources

12. Measuring Effectiveness
How well Youth
How well Quality of Mentee’s
your behavior
you run match a;tudes
volunteers (outcomes
your rela7on‐ on certain
mentor the you wish
program ships topics
youth to achieve)

13. Tips for Evaluating Mentoring Programs
Let’s look at the mechanisms of mentoring…

14. Missed Opportunities in Mentoring
Program Evaluation
1. Service delivery
Did they get the mentoring
promised?
2. Participant satisfaction
3. Mentoring relationship
Did they form the bond needed
to make any of this work?
4. Intermediary outcomes are OK!

15. What is a Logic Model?
• Systematic picture of how you believe your program works
• “…helps you visualize and understand how human and
financial investments can contribute to achieving your
intended program goals.”

16. Sample Vacation Logic Model

17. Sample YouthBuild Mentoring Logic Model

18. Tips for Selecting Good Instruments
• Try to use instruments that have been tested for
reliability and validity
• Make sure they fit your logic model, especially in
terms of proximal and distal outcomes
• Note the complexity of the tool:
– Age range
– Reading level
– Concepts
• Can you provide a comparison group of some kind?

19. Quantitative Tools
• Indicator: Example
– a variable with the purpose
to measure change in a
phenomena or process
• Valid:
– Does the tool measure Pretest Posttest
what it claims to measure?
• Reliable:
– Does the tool give
consistent results?
• Pre and Post Test
Pretest Posttest

20. Live Demo - Oregon Mentors
Evaluation Instrument Toolkit
• Browse for specific instruments (19
“domains”)
• Simple format for each instrument “record”
• Tips for using
• Links to other online repositories

21. Toolkit Domains of Interest
Live Demo of Toolkit & Instruments
Comprehensive Surveys: Personal Attitudes and Beliefs:
• Youth Outcome Survey • Antisocial attitude scale
• Youth Risk Behavior Surveillance • Decision making skills scale
System • Goal Setting Skills Scale
• Prosocial Involvement,
Connectedness:
Opportunities and Rewards
• Hemingway Scale of Adolescent
Connectedness
Mentoring relationship quality:
School and Academics: • Match Characteristics
• Education Expectations and Questionnaire (MCQ)
Aspirations Scale • Youth Mentoring Survey (YMS)
• Motivated Strategies for Learning • Youth Survey: Measuring the
Questionnaire (MSLQ) Quality of Mentor-Youth
Relationships

22. Tips for Administering Instruments
1. Plan for problems 6. Don’t provide guidance
2. Don’t hand-pick which 7. Administer the
questions to use instruments in a
consistent, uniform way
3. Don’t reword questions
8. Keep the answers
anonymous/ sort out any
4. Get parent permission coding before
administration
5. Provide a private space
to administer 9. Plan for data storage
and safety

23. Tips for Selecting External Evaluator
1. Consider his or her philosophical assumptions about
evaluation and how appropriate they are to addressing the
questions you want answered.
2. Invite finalists to meet project staff and others with whom
they will be working closely to see who best fits with
individual styles and your organizational culture.
3. An important part of an evaluator’s job is to assist in
building the skills, knowledge, and abilities of other staff
and stakeholders.
4. Think of evaluation as everyone’s responsibility. Be careful
not to delegate all evaluation decision making to your
evaluator.
See handout: Hiring an Evaluator Checklist

24. Tips for Working with External Evaluator
1. Don’t try and fake serious evaluation
2. Set aside appropriate funds (5-10 %)
3. Try and get a grad student from local college
or university - departments of education, child
psychology, social work, health, etc.
4. Remember that you are still in charge of your
evaluation!

25. Resources
http://oregonmentors.org/library/evaluationtools/

26. Resources
http://educationnorthwest.org/nmc
http://mentoringforums.educationnorthwest.org/

27. Resources
• W.K. Kellogg Foundation Evaluation Handbook
http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-
Foundation-Evaluation-Handbook.aspx
• W.K. Kellogg Foundation Logic Model Development Guide
http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-
Foundation-Logic-Model-Development-Guide.aspx
• Stand by Me: The Risks and Rewards
of Mentoring Today's Youth
by Jean E. Rhodes

28. Handouts
 Using Standardized Instruments
 Comparing Program Outcomes to National Standards
 Blank Logic Model Template
 Sample YouthBuild Mentoring Logic Model
 Using Your Logic Model to Focus the Evaluation
 Checklist for Selecting an Evaluator
 Sample Contact & Information Release Form
 Sample Evaluation Press Release
 Evaluating Mentoring Programs – P/PV Brief
 Additional Reading and Resources for Evaluating Your
Mentoring Program
 Glossary of Measurement and Evaluation Terms

29. Questions?

30. Thank you!
• Slides posted to
SlideShare
• Recording of webinar will
be posted to
http://
www.friendsforyouth.org/
Webinars.html
• Link and brief survey
included in follow-up
email

31. Next Webinar
• Tuesday, June 14 Helping Young
People Find Their Spark: A New
Gateway into Mentor‐Mentee
Rela?onships with April Riordan
Mentoring Partnership of Minnesota
• Wednesday, June 1: Eat Well,
Stay Ac7ve, Have Fun: A new
resource from the Harvard School
of Public Health
Register online:
http://www.mpmn.org/Training/MPMTRAINING/WebinarSeries.aspx

32. www.mentoringinstitute.org
650-559-0200
http://www.facebook.com/pages/Friends-for-Youth/
105093182858863
http://twitter.com/friendsforyouth
http://www.friendsforyouth.blogspot.com/
http://www.youtube.com/user/FriendsforYouthOrg