Presentation at the OE Global Conference held in Banff, Canada in April 2015. This presentation introduces the development of the evaluation strategy, using Utilization Focused Evaluation (UFE), for the ROER4D Project (http://roer4d.org/). You can access the abstract here: http://conference.oeconsortium.org/2015/presentation/developing-an-evaluation-strategy-to-gain-insights-into-a-multi-national-project-roer4d/

1. Sarah Goodier
ROER4D Evaluation Advisor
OE Global Conference
22 April 2015
Developing an evaluation strategy to gain insights
into the ROER4D multi-national project
4/24/20151

2. EVALUATING ROER4D

3. in the Global South
 In what ways, and under what
circumstances can the adoption of
OER address the increasing
demand for accessible, relevant,
high-quality and affordable
education and what is its impact in
the Global South?
Research on OER for Development

4. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Objectives

5. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Key Evaluation Areas
EVALUATION

6. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Key Evaluation Areas
EVALUATION

7. ROER4D Evaluation and Communication
supported by DECI-2
DECI-2
An IDRC funded research project to build and mentor Communication and Evaluation for IDRC
flagship projects
http://evaluationandcommunicationinpractice.net/

8. Ricardo Ramirez
Dal Brodhead
Wendy Quarry
(IDRC DECI-2 Project)
Julius Nyangaga
(Kenya)
Charles Dhewa
(Zimbabwe)
Sarah Goodier
(Evaluation)
Sukaina Walji
(Communication)
(South Africa)
ROER4D Evaluation and Communication supported by DECI-2

9. UTILIZATION FOCUSED
EVALUATION (UFE)

10. UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. Situational analysis
5. Identification of primary intended uses
6. Focusing the evaluation
7. Evaluation design
8. Simulation of use
9. Data collection
10. Data analysis
11. Facilitation of use
12. Meta evaluation
Steps in the
Utilization
Focused
Evaluation
(UFE)
process

11. UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. Situational analysis
5. Identification of primary intended uses
6. Focusing the evaluation
7. Evaluation design
8. Simulation of use
9. Data collection
10. Data analysis
11. Facilitation of use
12. Meta evaluation
Progress
towards
fine-tuning
evaluation
Steps are
iterative,
not linear
In progress

12. UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. Situational analysis
5. Identification of primary intended uses
6. Focusing the evaluation
7. Evaluation design
8. Simulation of use
9. Data collection
10. Data analysis
11. Facilitation of use
12. Meta evaluation
Progress
towards
fine-tuning
evaluation
In progress
Different
ROER4D
objectives are
at different
stages in the
UFE process

13. DEVELOPING AN
EVALUATION STRATEGY

14. ROER4D Process of developing evaluation strategy
1. Understand what is needed in terms of the scope of evaluating the
ROER4D project – the evaluation work is iterative by nature.
2. In collaboration with the ROER4D network hub team, formulate an
evaluation plan, including what to evaluate and how.
> The experience of the evaluation process and the effect this has is a
key component of the evaluation.
3. Get feedback from DECI-2 around the evaluation work and incorporate
this into the process.

15. ROER4D Process of developing evaluation strategy

16. ROER4D Process of developing evaluation strategy

17. ROER4D Process of developing evaluation strategy
4. Connect with members of the ROER4D, where needed (surveys,
interviews, etc.)
5. Assess the findings.
6. Share products of the evaluation work (e.g. slides around process and
results, reports, etc.) timeously to allow the findings and
recommendations to effect change.
7. Be aware of all components of the evaluation work and collaborate/
share information where possible and where needed.

18.  Benefits of being “open”
 The team dynamic in the network hub
is very open and inclusive. This
bolsters the evaluation work as the
team is very willing to share, discuss
and reflect on their processes and
work. There is a strong desire to
improve the project where possible.
(Transparency, Adaptability)
 The open and dynamic nature of the
project can also result in
opportunities with regard to the
evaluation work. As the project
evolves, new activities and outcomes
related to the priority objectives will
provide new windows of insight into
the project. (Adaptability)
 Benefit from the other components of
the evaluation work being
undertaken. (Reciprocity)
ROER4D Evaluating research in “the open”
 Challenges to being “open”
Evaluation work, by it’s nature, cannot
always be conducted in the open. In
working in an open project, where
outputs can be shared with a wide
audience, specific care needs to be
employed in the production of the
evaluation outputs and making explicit
what can and can’t be shared openly
and at what stage. (Vulnerability,
Liability)
The geographical distance between the
hub and many of the sub-projects can
also be an issue. Any evaluation work
involving sub-project feedback needs to
take into account differences in time-
zones and language as well as what is
possible using the communication
technology available. (Connectivity)

19. FINDINGS
Snapshots of communications
findings and recommendations

20. KEQs
 4.1 How well is the ROER4D communication
strategy working?
◦ 4.1.1 How has ROER4D’s social media presence
helped build the project brand?
◦ 4.1.2 To what extent has the ROER4D project
been established as a significant OER research
project in a) the Global South b) globally(and how
has the social media presence helped with this)
◦ 4.1.3 Which of ROER4D’s communication
channels/approaches has been the most effective
for communicating about the project’s key
objectives and to whom

21. Tweets per week
 Average: 7.75 tweets per week
Data downloaded from Twitter Analytics; analysis conducted in Excel: grouped
individual tweets into tweets per week

22. Tweets per week
 Average: 7.75 tweets per week
GO-GN and OCWC,
Slovenia (end April
2014)
GO-GN and
OpenEd, USA
(end November
2014)
OER Asia
Symposium,
Penang (end
June 2014)
UCT T&L Conf.
(20 Oct 2014)

23. Activity around tweets (per week)

24. Activity around tweets (per week)
GO-GN and OCWC,
Slovenia (end April
2014)
GO-GN and OpenEd,
USA (end November
2014)
OER Asia
Symposium,
Penang (end
June 2014)

25. Activity around tweets (per week)
GO-GN and OCWC,
Slovenia (end April
2014)
GO-GN and OpenEd,
USA (end November
2014)
OER Asia
Symposium,
Penang (end
June 2014)
Recommendation:
Continue tweeting around events as this draws higher levels of
engagement

26. “ROER4D” search: replies and mentions
(at 2 March)
Going to track over time
Data fetched from Twitter’s API via NodeXL; Frucherman-Reingo algorithm graph
constructed in NodeXL

27. “ROER4D” search: replies and mentions
(at 2 March)
Going to track over time
Data fetched from Twitter’s API via NodeXL; graph constructed in NodeXL
Recommendation:
Pick up conversations with key people identified from these
snapshots

28. UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. Situational analysis
5. Identification of primary intended uses
6. Focusing the evaluation
7. Evaluation design
8. Simulation of use
9. Data collection
10. Data analysis
11. Facilitation of use
12. Meta evaluation
In progress
Focus on:
• Data collection
and analysis
• Facilitation of
use by the
ROER4D team
Progress in the
communications
evaluation

29. Next steps
 More data collection and analysis (ongoing for Twitter as well as other social
media platforms)
 Feedback from the facilitation of use into refinements in the identification of
primary intended uses and users to help focus the evaluation further
 Constant dialogue with the ROER4D team and PIUs

30. Links and license
This work is licensed under a Creative Commons Attribution 4.0 International License.
Website: http://roer4d.org
Contact author: Sarah Goodier
sarah.goodier@uct.ac.za
@SarahGoodier
Excluding images,
screenshots and
logos and/or
unless otherwise
indicated on
content