The document outlines a framework for systematically evaluating informal education and outreach programs, which often have diverse activities and diffuse audiences that make rigorous evaluation challenging. It recommends sharing best practices across agencies to improve coordination and leverage the potential for real impact through economy of scale. A variety of qualitative and quantitative data collection methods are proposed to assess programs aimed at different age groups.
1. Top moments in UNAWE programmes
Co
un
try
CountryTeachers Ch
ild
re
n
How can we summarise
and communicate impact
in systematic ways?
Grace Kimble and Dr Cecilia Scorza
2. Framework for evaluation
Evaluation materials:
To do
1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
2. Survey for teachers- give after training
3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
4. Activity for younger pupils: before/after
5. Observation sheet for early years (4-7)
Optional: long term impact interviews
4. “"the systematic application of social
research procedures for assessing the
conceptualization, design,
implementation, and utility of social
intervention programs."
http://ideas.stsci.edu/Evaluation.shtml
5.
6. Front end Formative Summative
Process
Impact Outcome
7. Quantitative
Large sample sizes, closed questions
Example- Rosa’s likert scales
Qualitative
Small sample sizes, in depth, allow for open
ended responses
Mark and Libby’s questionnaires
9. Descriptive statistics
Correlational analysis
Statistical significance
It is necessary for data to be comparable in order for analysis
10. A way to collect data that is consistent and
systematic across different types of programme
11. Clarify Goals and Objectives- March 2012
Create a Model of Your Program
Choose Methodology
Choose Data Collection Method(s)
Analyse Data
Report and review
Disseminate
12. 1.1 The Program is addressed specially to disadvantaged children in different
social environments.
1.2 The program intends to stimulate via images, playful activities, experiments,
models and artistic activities the children´s awareness of astronomical objects
and the place of the Earth as part of the Universe in a way of broadening the children
´s view of the world. In doing so try to use throughout daily-life materials in the
activities, experiments and models such that children can reproduce them at home.
1.3 Awaken the awareness of the uniqueness of the Earth and of the need for
environmental preservation.
1.4 Motivate children to appreciate other cultures via stories and twinning
activities with children living in other countries, making them aware of the place in
which other children live, their common cultural features and differences.
1.5 Throughout stimulate children´s curiosity giving them the occasion and time to
formulate their own questions related to the Universe.
1.6 Develop EUNAWE-contents to be linked to daily school curricula via training
of the teachers.
13. Clarify Goals and Objectives- March
Create a Model of Your Program
Choose Methodology
Choose Data Collection Method(s)
Analyse Data
Report and review
Disseminate
14. Key questions
What are our assumptions about
a)What learning is?
b)How participants might change as a result of
programme interventions?
16. 1. The variety in types of programs is expansive. Informal education and outreach activities
can take place in schools, museums, the community, the media, and various other locations
where people gather information and experience the world. Almost all ACC agencies had some
type of program that was designed to generate awareness and engage the public in the agency’s
work. Further, the types of activities varied considerably across programs.
2. The nature of these programs makes it difficult to conduct rigorous evaluation because,
among other reasons:
(1) the audience for these programs is diffuse and difficult to identify;
(2) the multiple factors affecting and affected by these activities cannot be isolated for assessment;
(3) the modest scale of these efforts does not warrant a costly assessment approach.
There are examples of pre- and post- quasi-experimental evaluations of these programs, but it is
extremely challenging to carry out rigorous studies to identify causality in these programs.
3. Third, despite all of these complexities many programs share the same or similar goals. Though
the programs are varied, the Informal Education and Outreach group agreed on the importance
of better interagency coordination and information sharing. Sharing best practices across
agencies could offer significant benefits as these programs often bear a closer relationship to
outreach and informal education programs in other agencies than they do to K–12 or
postsecondary education programs in the same agency
(U.S. Department of Education. [DoEd], 2007, p. 26.).
17. “Furthermore, due to budget and time
constraints,
the evaluation of initiatives is often
limited. The interconnections between initiatives
are very rare which effectively eliminates
the possibilities for scaling-up and for the
dissemination
of the new ideas: the dynamics of
“economy of scale” and the huge potential for
real impact are simply not being exploited”.
http://ec.europa.eu/research/science-society/document_library/pdf_06/report-rocard-on-s
18.
19. a cy
Leg Awareness
of the
Universe
(knowledge)
EUNAWE
fostering…
Scientific Intercultural
skills attitudes
v at ion
Moti
“DOMAINS OF LEARNING”
20.
21. y
g ac Awareness
Le
of the
Universe
(knowledge)
EUNAWE
fostering…
Scientific Intercultural
skills attitudes
ion
Motivat
22.
23.
24. Clarify Goals and Objectives
Create a Model of Your Program
Choose Methodology
Choose Data Collection Method(s)
Analyse Data
Report and review
Disseminate
30. Mixed methods are required owing to variety
in programmes and ages
Qualitative evidence needs to be gathered:
-observation templates for children aged up to 7
-before and after activities for children 6-10
(Personal Meaning Mapping, John Falk)
-drawings for ages 6-8, surveys ages 8-10
31. Framework for evaluation
Evaluation materials:
To do
1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
2. Survey for teachers- give after training
3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
4. Activity for younger pupils: before/after
5. Observation sheet for early years (4-7)
Optional: long term impact interviews
42. Task 1: Submit a case study about your programmes and
development: http://goo.gl/yJQY5
These will form the basis for identifying examples for long term impact
Task 2: Modify the pupil survey- translate. Check teacher
survey in paper and online: http://goo.gl/XejyP
Task 3: Complete the checklist spreadsheet and return
Extra: add to observation framework
43.
44.
45.
46.
47.
48. Clarify Goals and Objectives- March
Create a Model of Your Program
Choose Methodology
Choose Data Collection Method(s)
Analyse Data
Report and review
Disseminate
49. Clarify Goals and Objectives- March
Create a Model of Your Program
Choose Methodology
Choose Data Collection Method(s) –Dec 2012
identify opportunities for evaluation
check materials suitable for group
Seek support
Analyse Data- GK: Jan- Feb 2013
Report and review- Mar 2013
Disseminate- Oct 2013
50. Framework for evaluation
Evaluation materials:
To do
1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
2. Survey for teachers- give after training
3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
4. Activity for younger pupils: before/after
5. Observation sheet for early years (4-7)
Optional: long term impact interviews
51. Materials returned to GK by end of Dec 2012
Draft report by end of Feb 2013
Identify issues in data
Modification and addressing gaps Mar 2013
Final report