Presented by Scott McKeown, Janine Barden O’Fallon and Jennifer Chapman at the MEASURE Evaluation End-of-Phase-III Event.

1. Introduction to Session on
Advances in Outcome Monitoring
Scott McKeown, MPH DrPH
Jack Hazerjian, MPH
MEASURE Evaluation
End-of-Phase-III Event, May 22, 2014

2. The basic tracking of variables that have been
adapted as measures or “indicators” of the desired
program outcomes. Outcome monitoring does not
infer causality; changes in outcome could be
attributable to multiple factors, not just the
program.
Definition: Outcome Monitoring
Global HIV M&E Information
– TWG UNAIDS MERG

3. Organized2006meetingforappraisal
ofmethodologiestomonitorprogram
outcomesas
acomplementtotheroutine
monitoringofprogramoutputs
Phase II: Review by Field Experts of
Outcome Monitoring Methodologies
 Lot Quality Assurance Sample (LQAS) Survey
 30 x “n” Cluster Sampling
 Cluster Sampling with Stratification
 Rider/Omnibus Survey (concurrently conducted with larger
household survey)
 Longitudinal Demographic and Community Surveillance

4. Consensus on Use of Sampling Methodologies
to Measure Health Program Outcomes
Sampling Methodologies Health Program Study Areas
Lot Quality Assurance
Sampling
* Immunization rates, MCH service access, HIV risk
factors
* Operations research, quality management
30 x “n” Cluster Sampling * Immunization, MCH,FP and HIV/AIDS rates
Other Cluster Sampling
Methods
* Widely used for multiple indicators
Rider/Omnibus Surveys
* FP use, prevalence and contraceptive security
*Willingness to pay for FP
Longitudinal Community
Surveillance
* Demographic & health surveillance
* Evaluation studies
* Health worker training in M&E

5.  Continue use of outcomemonitoring methodologies,as
indicated by health programcontext
 Develop/refineindicators and toolsfor outcome
monitoring
Phase III: Outcome Monitoring Priorities
o Where good indicators exist – develop improved data
collection tools (e.g., for maternal and child health programs)
o Where program areas are new – develop indicators and data
collection tools (e.g., for orphans and vulnerable children
programs)

6. Lot Quality Assurance
Sampling
 Kenya:
o Child health outcomes study
(2009, 2010, 2011-2012)
o Malaria health behavior
outcomes study (2010)
 Liberia:
o Maternal & child health, water &
sanitation outcomes study (2011,
2012, 2013)
Phase III Field Work in
Outcome Monitoring (1)

7. Cluster Sampling
 Zambia:
o Sexual behavior study (2009)
Stratified Cluster Sampling
 Nigeria:
o Household survey on women’s
reproductive health and children’s
health and primary education (2009)
Phase III Field Work in
Outcome Monitoring (2)
Wave 1 May 2009
Wave 2 July 2009
Kano State
% urban and rural respondents rejecting common
misconceptions about HIV transmission

8. Phase III Field Work in
Outcome Monitoring (3)
Rider/Omnibus Survey
 Angola:
o Gender-based violence sub-study within
PLACE Survey (2010-2011)
Longitudinal Community
Surveillance
 Uganda:
o Study of effect of mobile phone text
messaging on uptake of family planning
services (2012)
Eric Lafforgue (2010)
Health Child, Uganda (2013)

9.  Design of appropriate indicators and survey
instruments remains within work scope as new
health program interventions are introduced
 Collaboration with partners in design and
implementation of outcome monitoring studies
provides a valuable, learning-by-doing
opportunity in capacity building
Parting Thoughts
Outcome Monitoring Under MEASURE Evaluation

10. Rapid Assessments for Monitoring
MNCH Knowledge and Behavior
Experiences with LQAS
Janine Barden-O’Fallon, PhD
MEASURE Evaluation
End-of-Phase-III Event, May 22, 2014

11. Why Use Rapid Assessments?
Benefits
 Savings in cost, human resource commitment,
and time
 Can be conducted as often as needed
 Information is optimal for program management
Considerations
 Low statistical power
 Not tools for measuring incremental change

12. The Rapid Household Survey Handbook
How To Obtain Reliable Data on Health at the Local Level
Two-stage cluster
sampling (30 x “n”) and
Lot Quality Assurance
Sampling (LQAS)
Two Methodologies

13. Key Features
of LQAS
 Goal is to improve
project performance
 Designed to assist
project management
 Flags issues needing
attention
 Provides local level
information
 Uses a Decision Rule

14. Key Features Of LQAS Cont’d
Decision Rule = thepoint at which a sub-project area
(or “Lot”) satisfactorily meets the target
Example
LQAS with 5 lots
Sample size = 19 per lot (Total=95)
Target for indicator = 80%
Decision Rule = 13

15. LQAS for OM of Child Health
in Kenya
 Pilot in 2 provinces (2009)
 Rollout to 6 additional provinces
(2010)
 Return to 2 provinces (2011/12)
 Reflected integrated programming
 Focused on strengthening capacity
at the district and program
management level; “learning by
doing”

16. Example Results: Kenya 2009
Indicator: Percentage of mothers of children aged 0-23 months who
boil or chlorinate their drinking water to make it safe
Western
Province
Coverage: 44.8%
Decision Rule: 6 CI: 33.2-56.4
Districts Sample
Size
Correct Responses Meets decision
rule?
Budalangi 19 10 Yes
Teso North 19 9 Yes
Teso South 19 4 No
Busia 19 10 Yes
Samia 19 8 Yes

17.  Further train CHWs
 Focus on water and sanitation initiatives:
 Tablets
 Educate community members
 Water tanks
 VIP latrines
 Hand washing
Use Of Results:
Teso South District, Kenya

18. Example Results: Kenya 2012
Indicator: Percentage of mothers of children aged 0-23 months who
boil or chlorinate their drinking water to make it safe
Western
Province
Coverage: 55.9%
Decision Rule: 9 CI: 44.2-67.6
Districts Sample
Size
Correct Responses Meets decision
rule?
Bunyala 19 11 Yes
Teso North 19 11 Yes
Teso South 19 11 Yes
Busia 19 11 Yes
Samia 19 8 No

19. LQAS for OM of Child Health
in Liberia
 Pilotin 4counties(2011)
 Rolloutto7counties(2012)
 Capacitybuilding exercises (2013/14)
 Reflected integrated programming
 Focusedonstrengthening capacityatthe
countyandprogrammanagement level;
“learning bydoing”

20. LQAS Capacity Building: Liberia
 Data use workshops(2013)
 MOH-led in 2 counties(2013/14)
 CB practicum for MOHSW
central and county levels(2014)
 Interviewerscomprisedofcounty healthteams(2012)
 CB practicum for MOHSWcentral and county levels(2013)

21. Example Results: Liberia 2013
Indicator: Percentage of mothers of children aged 0-23 months
who received second dose of IPT for malaria during pregnancy
Lofa
County
Target: 80% Coverage: 61.2%
Decision Rule: 13 CI: 51.5-70.9
Health District Sample
Size
Correct Responses Meets decision
rule?
A 19 12 No
B 19 13 Yes
C 19 12 No
D 19 6 No
E 19 11 No
F 19 12 No

22. Rapid Assessments for OM and HIS
Basic requirements
 Good planning
 Understanding type of
information
 Intention to use
information
 Human and financial
resources

23. Evaluating OVC Outcomes
Jenifer Chapman, PhD
MEASURE Evaluation
End-of-Phase-III Event, May 22, 2014
Global Indicators and Tools for
Assessing Child Well-being

24. The Problem
 High investment
 BUT “what works” in
improving household
well-being?
 Challenge: lack of
standardized measures
and tools

25. A Proposed Solution
Standardized questionnaires for a
survey of children and their adult
caregivers

26. The Purpose
 Standardize population-level data
beyond what is available from routine
surveys
 Produce actionable data to inform
programs
 Enable comparative assessments

27. Focus on PEPFAR OVC Programs
 Indicators needtoreflect&be
amenabletochangebyPEPFAR
programintervention
 HH intervention by home
visitors
 Low direct funding per target,
focus on linkages
 Often inadequate services in
vicinity

28. Who Are These
Tools For?
 Local and
international
research
institutions
 USAID Forward

29. Guiding Principles
1. Focus on measuring program outcomes
2. Data collection by trained data collectors,
not service providers
3. Documented protocol is required
4. Protocol with tools needs to undergo ethical
approval
5. Tools require pilot testing in new settings

30. Developing the Tools
 Two-phase process
 Stakeholder-driven,
multi-agency
Result:
 3 tools
 Pilot tested

31. Core questions Optional
 Household schedule* (10)
 Changes in household
composition (4)
 Demographic information* (7)
 Work* (3)
 Access to money (3)
 Shelter (1)
 Household food security (6)
 General health (2)
 Caregiver support (4)
 Parental self-efficacy (1)
 Basic HIV/AIDS knowledge* (7)
 HIV testing* (3)
 Attitudes to condom education (1)
 Household access to services (1)
 Household Economic Status
 Progress out of Poverty Index
or similar (country specific)
 Dietary Diversity (1)
 Perceptions and experience of
abuse, exploitation and
violence
 Gender roles and decision
making power* (9)
 HIV/AIDS attitudes* (4)
*DHS, bold=core indicator
Caregiver Questionnaire

32. Core questions Optional
 Confirm demographics (5)
 General health & disability (4)
 Birth certificate (2)
 Vaccinations (11)
 Fever (<5 years)* (1)
 Diarrhea (<5 years)* (1)
 Shelter (1)
 Experience of neglect (2)
 Slept under mosquito net* (1)
 HIV testing experience* (2)
 School attendance*, progression/
repeats, drop-outs, missed school
days (5+ years) (9)
 Work for wages (2)
 Early childhood stimulation (2)
 Food consumption (2+ years) (8)
 Child access to services (1)
 Weight*, Height*, MUAC
 Fever: extended* (4)
 Diarrhea: extended* (3)
 Dietary diversity (1)
Child Questionnaire (Ages 0-9)

33. Child Questionnaire (Ages 10-17)
Core questions Optional
 Confirm demographics* (5)
 Identity of caregiver (1)
 Daily log (6)
 School attendance*, progress/ repeats,
drop-outs (9)
 Chores (3)
 Work (7)
 Food consumption (8)
 Alcohol consumption (3)
 Birth certificate (2)
 General health & disability (3)
 General support (4)
 Basic HIV/AIDS knowledge* (7)
 HIV testing* (3)
 Child access to services (1)
 Weight, Height, MUAC
 Dietary diversity (1)
 Perceptions/
experience of abuse,
exploitation, violence
 Child development
knowledge (6)
 HIV/AIDS attitudes
and beliefs (4)
 Sexual behavior
(13-17 yrs) (5)

34. The Toolkit – Where It All
Comes Together
 Tools & Manual
 Template protocol with
consent/assent forms
 Data analysis guide
 Data collector training
manual and materials
 French translations

35. So What?
 We know what we’re reaching for
 No more reinventing the wheel
 We’re accountable – MER reporting

36. Where can I find
out more?
Go to our website:
http://www.measureevaluation.or
g/our-work/ovc
Keep in touch on ChildStatusNet:
http://childstatus.net/
Email:
Jenifer Chapman:
jchapman@futuresgroup.com

37. www.measureevaluation.org/eop