2. In this session we will be learning about one of the 3
methods of monitoring evaluation outlined by Owen (2007).
To begin, we will be looking at “Monitoring” and “Evaluating”
as they go hand in hand to create an assessment.
A program is monitored for it’s use of resources to obtain
clearly specified results/outcomes. The data that is
collected from monitoring is used to assess a program. The
assessment is then used to guide and make optimal
management decisions.
Introduction
3. Monitoring vs Evaluation
This comparison chart extracted from the handbook
for development practitioners tells us about the
different roles of both monitoring and evaluation
Notice how monitoring involves routine(Continuous)
collection of data on specific indicators, linking
activities to resources and the role of management
and stakeholders to interpret this data into
knowledge and information to make decisions
Evaluation however is periodic, relies on
information provided from constant monitoring of
the processes/activities, helps the evaluator
analyze the shortcomings of the program,
accomplishment of objectives and provide
recommendation on improvement in order to fine
tune program (Kusek & Rist, 2004)
4. Systems Analysis
According to Owen (2006) a systems analysis uses a centralized
evaluation capacity to benchmark the performance of several
branches of an implemented program using the same operational
criteria
Designed for implementation on several sites using a central
framework for consistency
A system specifies how a program accomplishes its defined
goals across several sites to compare efficiency of outcomes
Usually involves a central management information system (MIS)
to collect data in a systematic way
Uses a logic framework to distinguish between an activity that
requires recourses and an outcome which defines progress
5. Should be performed during delivery of the program with the program plan
already in place
Only effective when program facilitators have little to no say in the
implementation of the program
When use systems analysis?
Why use systems analysis?
Increased accountability in the public sector
Increase in the need of performance/result based management and
evaluation
Why and when use Systems Analysis
6. Is the program reaching the target population?
Is it being implemented in the ways specified?
Is it effective?
How much does it cost?
What are the costs relative to its effectiveness?
Key evaluation questions according to Owen:
(Owen, 2007)
Key evaluation questions
Evaluate how every site of a program is implemented,
managed and administered
7. Management Information Systems (MIS)
“Management information system, or MIS, broadly refers to a
computer-based system that provides managers with the tools to
organize, evaluate and efficiently manage departments within an
organization.” (Webopedia, 2014)
In other words it a system for collecting, organizing and analyzing
data from different sites from a company or in this case a program
According to Owen (2007) a MIS is essentially the backbone for a
systems analysis
8. Pros and Cons
+
Systems analysis provides a systematic
approach therefore organizations/programs
can be benchmarked due to the similar
framework
Streamlining the monitoring process through
MIS opens new doors to communication
between different sectors of a program while
easing already existent channels
-
Tradeoffs between determining performance
Indicators that define appropriateness,
efficiency and effectiveness
Since evaluation happens periodically the
data collected from monitoring can only be
utilized later rather than sooner
Implementing a system for monitoring such
as a MIS can be costly
9. Project Overview
This National Monitoring and Evaluation System for Rural Water Supply and Sanitation Sector would
be used by Government of India for keeping track of the performance of the different States in
implementing the reforms agenda. This would be done through a comprehensive mechanism
involving: Computer based MIS, Periodic Reviews, Community Monitoring, and Benchmarking &
Performance Index Cards. By using progress (financial, physical and time progress) and process
indicators the performance of all the states, districts and block can be monitored so as to get an
overview of the inputs, outputs and outcomes of the entire water and sanitation sector. We developed
a web based Monitoring and Evaluation System along with a system for benchmarking, generating
performance index cards and community monitoring through which the Government of India to
monitor the progress on the implementation of reform agenda for the entire Rural Water and
Sanitation Sector in the country. The team piloted and operationalized this computerized system in
eight districts in Uttar Pradesh, Andhra Pradesh, Tamil Nadu and Maharashtra.
http://tscpl.com/?portfolio=monitoring-and-evaluation-system-for-rural-water-supply-sector-reform-program-and-total-
sanitation-campaign#prettyPhoto
Case Study
10. http://tscpl.com/?portfolio=monitoring-and-evaluation-system-for-rural-water-supply-sector-reform-program-and-total-
sanitation-campaign#prettyPhoto
Connecting with the Case Study
Monitoring and evaluation occurs on the implementation of a reform
across different states in India
The use of MIS to collect and organize data into information and
knowledge
The use of specific performance indicators measuring the overall
success of the state as well as specific districts and blocks within
states
The systems analysis is being implemented across the country to
monitor the progress of implementation of reform agenda
Linked to the accountability of the public sector as well as
performance management
11. Implementing a MIS can be costly and a rather time consuming process, what
other potential barriers can rise from the use and implementation of
Management Information Systems in a systems analysis?
Question 1:
Discussion Questions
“Better to be approximately correct than precisely wrong.” (Anonymous).
Defining performance indicators can be a difficult task as an evaluator, What
steps can you take as a professional evaluator to conceptualize indicators that
measure relative efficiency and effectiveness of a program at different sites?
Question 2:
12. Kusek, J. Z., & Rist, R. C. (2004). Ten steps to a results-
based monitoring and evaluation system a handbook for
development practitioners. Washington, D.C.: World Bank
Owen, J. M. (2007). Program evaluation: forms and
approaches (3rd ed.). New York: Guilford Press.
Goeller, B. F. (1988). A framework for evaluating success in
systems analysis. Santa Monica, CA: Rand.
References
