Presentation give to annual COE Policy Seminar-Washington DC March 2011

1. Annual Seminar with Department of Education
Using Empowerment
Evaluation and Data to
Strengthen Your Talent Search
Program
Maggie Cahalan and Stephaine Miller
Pell Institute, COE
March 9 2011

2. Presentation Topics
1. Overview of Empowerment Evaluation Concepts
and Steps with example from COE’s new I-3
project testing new models for Talent Search
(Maggie)
2. Using Data for Evaluation Program Improvement
and Student Success (Stepahanie)
3. Breakout Session (Speed Dating Process to move
through ideas about steps in empowerment
evaluation of your program)—(Everyone--35
minutes)
4. Group empowerment evaluation of our session
(what you liked and did not like and how we could
improve (Everyone--10 minutes)

3. Personal Journey in Thinking
About Evaluation
Contractor Project Director
National Evaluation of Student Support
Services
National Evaluation of Talent Search
TRIO performance reporting support
contracts
Department of Education
National Evaluation of Upward Bound study
final technical monitor
Came to see need for taking a more
participatory approach involved and began
Designing Next Generation of GEAR UP

4. Next Generation of Gear Up
Evaluations
Grew out of challenges (problems with
meaningful comparison group, validity
issues, length of time to obtain results, black box
up or down studies not always useful for program
improvement, lack of involvement of stakeholders
meant work not viewed as very useful to them)
New Approach (18 month design work—sessions
with grantees, expert papers, systemic review of
past evaluation methods and challenges, issued
call for entries open to all grantees to design
evaluations of what they viewed as promising
practices, funded 44 grantees proposals and
provided technical assistance, and funded 12 to
implement the project led evaluations of rigorous
designs)

5. Definition
Empowerment evaluation is the
use of evaluation
concepts, techniques and
findings to foster improvement
and self determination. It
employs both qualitative and
quantitative methodologies.
Empowerment evaluation: Knowledge
and Tools for Self-Assessment and
Accountability
Fetterman, Kaftaraian, and Wandersman
1996

6. Empowerment Evaluation
Aims to increase the probability of program
success by:
1 Providing program stakeholders with tools for
planning and implementation, and self evaluation
of their program, and
2 Mainstreaming evaluation as part of the planning
and management of the program organization
(Wandersman, et. Al, 2005, p. 25)

7. Empowerment Evaluation
Widely used
Joint Committee for Education Program
Standards have included concepts
Amercian Evaluation Association and AERA
workshops
Public Health Field
International Development work
Education school systems

8. Contrasts
Traditional Evaluation Empowerment Evaluation
External Internal
Expert Coach or Critical friend
Self-determination &
Dependency capacity building
Independent Collaboration
judgment
Makes use of Data
Developed when data Revolution—
internet, web, real time
not available to all— interactive sharing of
elite with resources knowledge—all
to collect and skill to publish—face
analyze book, blog, twitter

9. Internal or External
Evaluation
The dilemma of whether to
use external or internal
evaluation is as false as that
between qualitative and
quantitative methods. The
solution is always to use the
best of both, not just one or
the other (Michael Scriven)

10. Learn by Doing
Organizations and their staff evaluate their
own strategies
May hire an evaluator to work with them
Rather than presenting a report
card, empowerment evaluators coach staff
through an evaluation of their own strategies
by providing them with the knowledge, skills
and resources they need to conduct
evaluation

11. 10 Key Principals
(Wandersman et. Al 2005)
1. Community Ownership—primary responsibility with
organization and not outside evaluator
2. Inclusion—involves representation of key stakeholders
3. Democratic participation—highly collaborative—
opportunity to voices questions—every stakeholders
voice is heard
4. Community knowledge-promotes growth of
knowledge in community—stakeholders are considered
to be in the best position to understand the issues and
generate solutions to problems
5. Evidence Based strategies-promotes use of
strategies with high quality evidence—research evidence
of effectiveness—evidence strategies contextualized to fit
community

12. 10 Key Principals (cont)
6. Accountability-provides data that can be used to determine whether a
strategy has achieved its goals—negative results are not punished;rather
are use to inform change in a strategy or the selection of a new strategy
for the purpose of producing better outcomes
7. Improvement—Helps organizations improve strategies so that they
are more likely to achieve stated goals—process and outcome evaluation
(Rossi 1999)
8. Organizational learning—fosters a culture of learning—view
positive and negative feeedback as valuable information and believe that
all strategies can be improved
9. Social justice—increase capacity to reduce disparities that affect
marginalized by persecution, discrimination, prejudice and intolerance
10. Capacity building—builds capacity of organizations to conduct their
own evaluations, understand results and use them to continuously
improve organization

13. 10 Steps (Chinman, Imm &
Wandersman, 2004)
1. Conduct needs and 6. Finalizing plan
resource assessment
7. Evaluate process
2. Develop goals and
outcomes sought 8. Evaluate outcomes
3. Selecting evidence 9. Implementing
based strategies continuous quality
improvement
4. Assessing strategies
fit 10.Sustaining efforts
5. Building capacity

14. Fetterman (2001)
Define mission
Taking stock-
Planning for future

15. Example---COE Investing in
Innovation (I-3) project
ED wishes to build capacity at all levels to use data and build in
internal and external evaluations as way to improve
programming on an on-going basis
October COE won I-3—Investment in Innovation--Using Data to
Inform College Access—Power Knowledge
Build on and help validate GE/COE college access work in 2
cities (Erie and Louisville)
Enhanced whole school college access services—college coaches
in school
Intensive service Learning Communities
COE Building large data warehouse Power Knowledge Tool for use
by all
Contribute to innovative adaptive models for established
Federal College Access programs such as Talent Search, GEAR
UP, Upward Bound

16. COE (I-3) project Testing
and Validating ---Model-
--Increased access to, skill, and use of data
by all stakeholders
(students, parents, school, college
coaches, community) will lead to
increased high school and college
participation and success among
students in high schools
Internal Empowerment Evaluation and
External evaluators both using the DATA
TOOL

17. DICAP--Builds and Enhances on
GE-COE Model
Multiple Power Knowledge Learning
Communities
Using data among all stakeholders—key
District and school staff
College access service providers and
planners--College Coaches
Students
Parents
Community
Internal and External Evaluators

18. Goals of UN Education for
Sustainable Development
What if every person benefited from an education promoting
development that is environmentally sound, socially
equitable, culturally sensitive and economically just?
What if learning was about knowledge and also about
doing, being, interacting with others and changing the world?
What if formal learning was enjoyable, hands-on and relevant to life
outside school while addressing the problems of our world?
What if every person benefited from genuine learning opportunities
throughout life, in the workplace, and within the community?
What if education systems prepared learners to enter the workforce
as well as handle a crisis, be resilient, become responsible
citizens, adapt to change, recognize and solve local problems with
global roots, meet other cultures with respect, and create a peaceful
and sustainable society?

19. Partnership—Collaborative-
Empowerment Model
All stakeholders full partners—those most
impacted –strongest roles
Stakeholders
Help set goals
Identify assets, challenges, describe current
baseline situation and data
Take responsibility for identifying and
implementing steps (actions) to meet goals
Take responsibility for analysis, reflection on
results and become learning community
Transparency among all stakeholders

20. Partnership—Collaborative-
Empowerment Model
All stakeholders full partners—those most
impacted –strongest roles
Stakeholders
Help set goals
Identify assets, challenges, describe current
baseline situation and data
Take responsibility for identifying and
implementing steps (actions) to meet goals
Take responsibility for analysis, reflection on
results and become learning community
Transparency among all stakeholders

21. Smaller Intensive Learning
Community
Power knowledge Learning Community
Leadership Group
Summer Bridge
On-going school year work
Project to participate in building and using the
data tool
Project to communicate to whole school—lead
to goals
Project to evaluate and assess efforts and to
improve the program –internal empowerment
evaluation

22. Theory of Participatory
Action/Change
Learning
Reflection and Community
Prepares various
Participatory Communicate/di reporting's/feedba
goal setting sseminate ck for
findings stakeholders
Identify Inputs Impact
(assets, baselin assessment and
Power Data Tool
es, challenges recommendatio
and barriers) ns for future
Identify indicators
Identify and monitor
fidelity of Assessment of
Steps/actions intermediate
implementation/ch indicators/outcomes
needed to allenges/counterfa
meet goals ctuals

23. Logic of Evaluation
Goal or purpose
Baseline –where before intervention
Program or intervention
Measurement (including criteria)
Process
Outcomes and impacts

24. Speed Dating Stations Idea
Generating
1. Goal and Strategy setting
1. Setting Goals/Mission—what do want to change or
improve—how much?
2. Evidence based strategy for improvement—what
concrete steps do you need to take that you think might
work? Now? Next year? Next five years?
2. Taking Stock
1. Baseline—where are you now—take stock of assets and
barriers?
2. How can you assess or measure if what you are trying is
working-----choose an evaluation design and indicators
3. Provide for gathering credible evidence---Data
3. Plan for Future—Learning Organization
1. Use data to analyze and reflect on what you find
2. Improve organizational capacity and the particular
strategy evaluated

25. Resources and contact
information
Contact---margaret.cahalan@pellinstitute.org
Empowerment evaluation resources
http://eevaluation.blogspot.com/
www.cdc.gov/violenceprevention/pdf/evaluati
on_improvement-a.pdf
http://wwwstatic.kern.org/gems/region4/David
FettermanPresentation.pdf