IHME presents the mixed-methods evaluation plan for the Salud Mesoamerica Initiative evaluation project.

1. Salud Mesoamerica Initiative
Mixed-Methods Evaluation Plan
March 21, 2023
Ali H. Mokdad, Ph.D.
Principal Investigator, Salud Mesoamérica Initiative
Chief Strategy Officer, Population Health
Professor, Health Metrics Sciences

2. Outline
• Introduction
• Quantitative measurement and results
• Qualitative measurement
• Overall evaluation
• Discussion and recommendations
2

3. Framing
• IHME has been involved from early on
• IHME has done a qualitative evaluation focusing on Chiapas
• Other qualitative evaluations of SMI have occurred
• We have limited resources for the overall evaluation
• Our approach is not about the Initiative but we view it as an
organizational approach with layer. SMI is one part.
• Covid19 and it is impact. Absorb, Adapt, and Transform
3

4. Introduction
• SMI is an ambitious and innovative
project:
o Financing based on results
o Regional perspective
o Local and SMI support
• Indicators and goals defined by the
Ministries of Health and the IDB
• IHME serves as an independent evaluator
• The collection of baseline and follow-up
information at 1st, 2nd, and 3rd operation
has been completed in all countries
4

5. COVID-19 pandemic, 2020-present
• COVID-19 pandemic affected health systems around the world
o Human resources were diverted to infectious disease surveillance,
treatment, and vaccination
o Public health orders and service closures kept people from seeking
care for emergent and chronic conditions early in the pandemic
o Fear of infection has continued to deter patients from seeking
needed care
o Economic impacts also affect care-seeking behavior
5

6. Pediatric vaccination, 2020
Source: Santoli et al, Effects of the COVID-19 Pandemic on Routine Pediatric Vaccine Ordering and Administration —
United States, 2020. MMWR
6

7. Global Progress and Projections for Vaccines
Source: 2022 Goalkeepers Report, Gates Foundation
7

8. Latin America and Caribbean
Source: 2022 Goalkeepers Report, Gates Foundation
8

9. Baseline
HH & HF
surveys
2011-13 2017-18 2022
1st
operation
Target
verification
HF surveys
Target verif.
HH & HF
surveys
1
2
9
SMI Measurement Design & Timeline
Target verif.
HH & HF
surveys
3
Target verif.
HH & HF
surveys
3
2nd
operation
3rd
operation COVID-19
2014 2020
Qualitative
midterm
evaluation
Final
evaluation

10. Third operation time periods
• For additional context related to the pandemic and the delay of
the third operation target verification, additional medical
records were collected from directly before the evaluation
performance period (referred to in subsequent tables as ‘pre-
evaluation’ period)
Period Start date End date
Pre-evaluation January 1, 2019 June 30, 2020
3rd operation
evaluation period*
July 1, 2020 June 30, 2022
*Some indicators are evaluated for only a subinterval of the above evaluation time
period based on specifications of the indicator definition;
Start and end dates of both periods were 15 days later in Belize.
10

11. Outline
• Introduction
• Quantitative measurement and results
• Qualitative measurement
• Overall evaluation
• Discussion and recommendations
11

12. Census
• Full household listing for selected
segments
• Strict protocol for visits
• Response rates monitored
• Unique IDs assigned for reference
during household survey
• Manual checking to map if
discrepancies found in data
• All household members accounted
for along with data on age, sex,
relationship to head of household,
languages spoken, and other key
demographics information
12
Honduras, May 2017

13. Household Survey
• Household characteristics
• Expenditure and health expenditure
• Impacts of COVID-19
• Health service utilization (women and
children)
• Family planning, reproductive history
• Child health, breastfeeding, immunization
• Physical measurements
o Weight, height, hemoglobin
• Belize measurement included LQAS
survey rather than full census and
household survey
Honduras, May 2017

14. Health Facility Survey
• Questionnaire administered to facility manager
o Facility operations
o COVID-19 vaccination
o Pandemic closures
• Physical observation
o Equipment and inputs
o Pharmaceuticals
o Review of registries to detect stock outs
o Posters and health promotion materials
• Medical record review
o Record quota according to characteristics of
health unit
─ Antenatal and postnatal care
─ Deliveries
─ Maternal and neonatal complications
─ Child growth and development, deworming
─ Diarrhea

15. 3rd Operation Results – Sample Sizes
Country
Health
Facilities
Medical
Records
Census Household Women Children
Belize 20 1,085 - - 482 182
Nicaragua 42 1,800 5,646 1,414 1,708 1,338
El Salvador 36 2,074 4,442 1,145 1,355 796
Honduras 37 1,429 5,929 1,420 1,620 1,174
SMI 3rd Operation: Sample sizes
Sample sizes limited to intervention areas only
15

16. Belize Performance Indicator Results
Verification
source
Number of
indicators
Number of indicator
targets met
Number of indicator
targets not met
TOTAL 9 5 4
Health Facility 6 4 2
LQAS 3 1 2
Verification
Source
Indicator
#
Indicator
Indicator
value (%)
Indicator
confidence
interval
Target
(%)
Target
status
Health Facility 3000 Preconception care 0 (-) 20.0 Not met
Health Facility 3030 Antenatal care with quality 9.5 (5.1 - 17.0) 36.9 Not met
Health Facility 4050 Postpartum care with quality 49.2 (36.4 - 62.1) 46.6 Met
Health Facility 4070 Management of neonatal complications 56.2 (45.0 - 66.9) 44.3 Met
Health Facility 4080 Management of obstetric complications 56.1 (45.0 - 66.6) 49.6 Met
Health Facility 7500 Use of data for decision-making 50 N/A 40.0 Met
LQAS 5020 Complete vaccination for age 52.8 (41 - 64.2) 22.6 Met
LQAS 5060 Diarrhea treatment with ORS and zinc at home 0 (-) 45.3 Not met
LQAS 6000 Cervical cancer screening 53.2 (46.5 - 59.7) 75.5 Not met
16

17. 3rd Operation Results – Belize Health Facilities
1
Rapid Plasma Reagin (RPR) not captured as syphilis test at 1st
operation or baseline; HIV not captured at baseline; Gestational age eligibility for uterine height
and fetal checkups only available for first ANC visit at baseline.
2
Heart rate not measured as alternative to pulse at 1st
operation or baseline; unable to exclude births that were referred at 1st
operation or baseline.

18. 3rd Operation Results – Belize LQAS
1
3rd Operation results adjusted to reflect recollected data on HPV screening in households.

19. Nicaragua Performance Indicator Results
19
Verification
Source
Indicator
#
Indicator
Indicator
value (%)
Indicator
confidence
interval
Target
(%)
Target
Status
Health Facility 3040 First antenatal care within 12 weeks 63.4 (57.3 - 69) 58.7 Met
Health Facility 4103 Routine newborn care with quality 40.3 (33.3 - 47.8) 58.7 Not Met
Health Facility 4070 Management of neonatal complications 59.5 (52.2 - 66.4) 61.3 Met
Health Facility 4080 Management of obstetric complications 44.5 (38 - 51.2) 69.9 Not Met
Health Facility 6005 Cervical cancer screening with quality 76.9 (70.7 - 82.2) 60 Met
Health Facility 7500 Use of data for decision-making 93.3 (58.4 - 99.3) 66 Met
Household 1060 Children 5-23 months with hemoglobin <110g/L 51.4 (42.3 - 60.5) 43.5 Met
Household 4030 Women receiving skilled post-partum care within 10 days 86.8 (81.7 - 90.6) 90.6 Met
Household 5020 Complete vaccination for age 52.5 (45.0 - 59.9) 56.7 Met
Household 5060 Diarrhea treatment with ORS and zinc (0-59 months) 10.7 (7.2 - 15.6) 16.4 Not Met
Verification
source
Number of indicators
Number of indicator
targets met
Number of indicator
targets not met
TOTAL 10 7 3
Health Facility 6 4 2
Household 4 3 1

20. 3rd Operation Results – Nicaragua Health
Facilities
20
Indicator Description Time period N n % CI Target
Baseline 106 42 39.6 (30.3 - 49.6)
1st Operation 371 111 29.9 (25.5 - 34.8)
2nd Operation 389 201 51.7 (46.7 - 56.6)
Pre-evaluation 151 96 63.6 (55.5 - 70.9)
3rd Operation 262 166 63.4 (57.3 - 69) 58.7
Baseline 69 5 7.2 (3 - 16.6)
1st Operation 184 125 67.9 (60.8 - 74.3)
2nd Operation 279 122 43.7 (38 - 49.6)
Pre-evaluation 122 49 40.2 (31.7 - 49.2)
3rd Operation 176 71 40.3 (33.3 - 47.8) 58.7
Baseline 190 77 40.5 (33.7 - 47.7)
2nd Operation 283 131 46.3 (40.5 - 52.2)
Pre-evaluation 133 80 60.2 (51.5 - 68.2)
3rd Operation 185 110 59.5 (52.2 - 66.4) 61.3
Baseline 204 75 36.8 (30.4 - 43.7)
2nd Operation 301 135 44.9 (39.3 - 50.5)
Pre-evaluation 131 58 44.3 (35.9 - 53)
3rd Operation 218 97 44.5 (38 - 51.2) 69.9
Baseline
1st Operation
2nd Operation
3rd Operation 208 160 76.9 (70.7 - 82.2) 60
Baseline
1st Operation
2nd Operation
3rd Operation 15 14 93.3 (58.4 - 99.3) 66
7500 Use of data for decision-making
Not measured at baseline
Not measured at 1st operation
Not measured at 2nd operation
6005 Cervical cancer screening with quality
Not measured at baseline
Not measured at 1st operation
Not measured at 2nd operation
4070 Management of neonatal complications
4080 Management of obstetric complications
3040 First antenatal care within 12 weeks
4103 Routine newborn care with quality

21. 3rd Operation Results – Nicaragua
Households
21
Indicator Description Time period N n % CI Target
Baseline 435 236 53.9 (48.0 - 59.7)
2nd Operation 487 251 51.3 (44.8 - 57.7)
3rd Operation 350 182 51.4 (42.3 - 60.5) 43.5
Baseline 197 3 1.4 (0.5 - 4.3)
2nd Operation 246 16 6.5 (3.9 - 10.7)
3rd Operation 184 18 10.7 (7.2 - 15.6) 16.4
Baseline 1398 706 48 (41.8 - 54.3)
2nd Operation 1802 836 46.7 (42.1 - 51.3)
3rd Operation 1331 746 52.5 (45.0 - 59.9) 56.7
Baseline 657 409 60.1 (54.2 - 65.7)
2nd Operation 874 714 82.3 (77.6 - 86.7)
3rd Operation 572 501 86.8 (81.7 - 90.6) 90.6
Children 6-23mo with hemoglobin <110g/L
Diarrhea treatment with ORS and zinc (0-59mo)
Complete vaccination for age
Women receiving skilled PPC within 10 days
1060
5060
5020
4030

22. El Salvador Performance Indicator Results
22

23. 3rd Operation Results – El Salvador Health
Facilities
* Injection and implant postpartum contraceptives not captured at first operation; 'progestin-only' not specified for OCP at first operation.
** Referral not captured at first operation so the subsequent exclusion cannot be applied as it is at second and third operation. At first operation, uterine
height and fetal checkups are only evaluated at first visit, if eligible based on gestational age. Risk factor management not captured at first operation. RPR
not captured as VDRL alternative at first operation.
*** Blood abnormalities postpartum check not captured at first operation.
23

24. 3rd Operation Results – El Salvador
Households
24

25. Honduras Performance Indicator Results
25

26. 3rd Operation Results – Honduras Health
Facilities
Indicator Description Time period N n % CI Target
Baseline 125 16 12.8 (7.9 - 20)
1st Operation 228 98 43 (36.7 - 49.5)
2nd Operation 272 223 82 (76.9 - 86.1)
Pre-evaluation 82 50 61 (49.8 - 71.1)
3rd Operation 177 113 63.8 (56.4 - 70.6) 90
Baseline 120 26 21.7 (15.1 - 30.1)
1st Operation 119 71 59.7 (50.5 - 68.2)
2nd Operation 150 112 74.7 (67 - 81)
Pre-evaluation 78 53 67.9 (56.6 - 77.5)
3rd Operation 146 115 78.8 (71.3 - 84.7) 82.5
Baseline 70 7 10 (4.8 - 19.8)
2nd Operation 84 35 41.7 (31.4 - 52.7)
Pre-evaluation 33 12 36.4 (21.3 - 54.7)
3rd Operation 104 71 68.3 (58.6 - 76.6) 61.7
Baseline 100 36 36 (27.1 - 46)
2nd Operation 105 66 62.9 (53.1 - 71.7)
Pre-evaluation 36 22 61.1 (43.7 - 76.1)
3rd Operation 113 75 66.4 (57 - 74.6) 82.9
Baseline
1st Operation
2nd Operation
3rd Operation 251 144 57.4 (51.1 - 63.4) 70
3050 Antenatal care with quality*
4103 Routine newborn care with quality**
4070 Management of neonatal complications
4080 Management of obstetric complications
6005 Cervical cancer screening with quality
Not measured at baseline
Not measured at 1st operation
Not measured at 2nd operation
* Notes on baseline and first operation results: Referral not captured, so the subsequent exception cannot be applied as it is at second and third operation.
Uterine height and fetal checkups are only evaluated at first visit, if eligible based on gestational age. RPR not captured as VDRL alternative for syphilis test.
** Temperature check not captured at baseline or first operation.
26

27. 3rd Operation Results – Honduras
Households
27
Indicator Description Time period N n % CI Target
Baseline 437 153 35 (29.3 - 41.2)
2nd Operation 458 287 62 (56.0 - 67.7)
3rd Operation 260 94 37 (29.0 - 45.9) 52
Baseline 77 0 0 (-)
2nd Operation 49 4 8.3 (3.0 - 21.0)
3rd Operation 75 8 9.6 (4.1 - 21.0) 25
Baseline 666 465 68.9 (60.8 - 75.9)
2nd Operation 709 611 85.2 (79.1 - 89.8)
3rd Operation 606 532 86.9 (83.1 - 90.0) 91.7
Baseline 597 289 46.8 (40.3 - 53.5)
2nd Operation 624 206 33.3 (25.7 - 41.8)
3rd Operation 529 176 34.3 (29.0 - 40.0) 41.3
Baseline 969 685 69.3 (62.6-75.3)
2nd Operation 1198 902 75.4 (69.9 - 80.2)
3rd Operation 934 748 79.8 (75.0 - 83.9) 80.4
4010 Institutional delivery
4100 Postpartum care for neonates (72 hours)
2010 Use of modern family planning methods
1060 Children (6-23mo) with hemoglobin <110g/L
5060 Diarrhea treatment with ORS and zinc (0-59 mo)

28. 3rd Operation Results - Overview
28
Country
Number of
Indicators
Number of
Indicator Targets
Met
Percentage of
Indicator Targets
Met
Belize 9 5 56%
Nicaragua 10 7 70%
El Salvador 10 4 40%
Honduras 10 4 40%
SMI 3rd Operation Performance Indicator Results

29. Performance challenges
• The COVID-19 pandemic tested the sustainability of the
systems put in place through SMI
o Demand for health services was impacted as people shied away
from care
o Supplies were impacted due to global shortage
o Economic impacts should not be underestimated
29

30. Findings: Progress despite COVID-19
• Strong performance on quality of care indicators in several countries
o Neonatal and obstetric complications management
o Cancer screening
• Even some coverage indicators improved
o Complete vaccination for age (Belize, Nicaragua)
o Use of modern contraceptive method (Honduras)
• Indicators that rely on patients seeking care had barriers to success
o Antenatal, postpartum, postnatal checkups
o Diarrhea treatment with zinc
• In Nicaragua, SMI improvements may be sufficiently integrated into
the health system that performance improved even during the
external stress to the health system
• There is still room for improvement in recordkeeping and
management of health records
30

31. Conclusions – Verification of targets
• The measurement happened before health systems had
recovered
• Yet, over half of the 3rd operation performance targets
were met, when considering all countries together
o Initiative programs were on track before COVID-19
o Countries remained engaged during and after the pandemic
o Public healthcare systems overcame substantial challenges to
display this level of progress
o Continued improvement under these conditions provides evidence
in favor of the sustainability of the changes brought by SMI
31

32. Outline
• Introduction
• Quantitative measurement and results
• Qualitative measurement
• Overall evaluation
• Discussion and recommendations
32

33. Need for mixed-methods final evaluation
• Unexpected findings to explain
• Crucial design topics not explored
• Inability to assess certain topics quantitatively
• Basically, the “How” and “Why” questions
33

34. Qualitative evaluation questions
• SMI influential components
o Lessons from its unique model
• SMI contribution in the performance of health systems
o Mechanisms for change in quality and coverage
• SMI vs. other financing or intervention models
o SMI design
• Sustainability of SMI
o Adaptation, continuation, integration of changes due to SMI
• SMI and COVID-19
o Pandemic preparedness, response, recovery
34

35. Design
Collection and
analysis of quant
data
Collection and
analysis of qual
data
Comparison
and integration
of data
Interpretation
• Qualitative data analyzed
alongside quantitative data
from ongoing evaluation
and target verification
• Convergent design
• Merged data analysis
35

36. Timeline
36
Activity January February March April May
Planning &
design
Data collection
Data analysis
Consolidate
results
Writing reports

37. Sample Summary
22
Central-
level
informants
88
Country-
level
informants
110
Total
participants
37

38. Key informant interviews
38
Study Audience # of KII
Funder representatives + knowledgeable others 8
IDB/SMI representatives 14
Ministry of Health (MoH) representatives 24
Health care providers: SMI 44
Health care providers: non-SMI 12
Traditional birth attendants/Community Health
Workers
8
Total 110

39. Sample
• Purposive sampling: information rich cases to provide depth of
information
─ Funders, IDB representatives, MOH representatives
• Probability sampling: representative units, linked with
quantitative data collection
─ Health care providers, community health workers, midwives
39

40. Analysis plan
• Integrative methodology
o Deductive: a-priori codes based on SMIPE results and
theorical frameworks
o Inductive: rigorous open coding process
• Multidisciplinary team
o Independently coding a sub-set of transcripts, followed by a
constant comparison method to end with an integrated code
structure
o Iterative approach to identify major themes, comparing
across countries and audiences
40

41. Frameworks
• Theoretical orientation for the topic guide design and
initial deductive codebook
• Program evaluation: OECD evaluation criteria
• Health systems performance and evaluation
─ The Health Systems Strengthening Evaluation Collaborative
─ WHO building blocks (for reference and comparison with SMIPE
results)
• Healthcare access: Levesque model
• Healthcare quality: AHRQ
• Sustainability: multiple source – literature review
41

42. Program evaluation
(OECD, Evaluation Criteria, 2022)
42

43. Health System Analysis
(Health System Strengthening Evaluation Collaborative, 2022)
43

44. Access to healthcare
(Levesque, JF., Harris, M.F. & Russell, G. 2013)
44

45. Six Domains of Healthcare Quality
• Safe: Avoiding harm to patients from the care that is intended to help them.
• Effective: Providing services based on scientific knowledge to all who could benefit and
refraining from providing services to those not likely to benefit (avoiding underuse and
misuse, respectively).
• Patient-centered: Providing care that is respectful of and responsive to individual
patient preferences, needs, and values and ensuring that patient values guide all clinical
decisions.
• Timely: Reducing waits and sometimes harmful delays for both those who receive and
those who give care.
• Efficient: Avoiding waste, including waste of equipment, supplies, ideas, and energy.
• Equitable: Providing care that does not vary in quality because of personal characteristics
such as gender, ethnicity, geographic location, and socioeconomic status.
(Agency for Healthcare Research and Quality. 2022)
45

46. What is SMI? How does it work?
• What is the vision of SMI?
• What are the key elements of SMI?
• How does it compare with other health and development
programs?
46

47. What was the role of technical assistance in
SMI?
• How did technical assistance influence the results of SMI?
• What was the role of IDB in the Initiative?
• Were any lessons or elements of technical assistance
transferable or useful in response to the pandemic?
47

48. How did SMI affect conversations around
policy?
• Did SMI promote changes to norms, policies, and protocols for
MNCH? Which ones?
• What topics were under discussion in the public agenda when
the Initiative started?
• Did the Initiative introduce new themes or perspectives? Which
ones?
48

49. How did the regional element of SMI affect
implementation and outcomes?
• What are the advantages and disadvantages of a regional
model like SMI’s?
• Are there any interventions or programs designed first in one
country and then implemented in another?
• What differences existed between countries and what
adjustments were necessary for SMI?
49

50. How did the financing model affect
accountability and participation in SMI?
• What makes the financing model of SMI distinct from other
investments in health?
• How did this model contribute to the results?
• Would these funds have had a greater impact on health if they
were invested differently? How else could they have been
invested?
50

51. What were barriers and facilitators for
implementation?
• To what extent where SMI activities implemented according to
plan?
• What were barriers to implementation? What were facilitators?
51

52. What was the contribution of SMI in the
health system?
• What were SMI’s key interventions for maternal and child
health?
• How have SMI interventions been integrated into health
systems in each country?
• What was the role of IDB in supporting health systems and
health service provision improvements?
52

53. How do participants perceive the results of
SMI?
• Were the expected results achieved?
• What aspects of the Initiative most contributed to achieving –
or not achieving – the expected results?
• Are there longer-term health benefits for participating
communities? What are they?
53

54. What does sustainability for SMI look like?
• What interventions will continue and how will they be financed?
• How has behavior, policy, or system change due to SMI been
sustained at different levels of the health system?
• What could have been done differently to ensure sustainability
for gains from SMI?
54

55. Has SMI mediated the effects of COVID-19
and economic crises in the region?
• What have been the main challenges for maternal and child
health during the pandemic?
• How have economic problems and inflation affected maternal
and child health services?
• What tools or lessons from the Initiative have been useful in
response to the pandemic? In response to economic crises?
55

56. Root cause analysis: sustainability of changes
due to SMI
56
Shortage in
human
resource
quantity and
specialists
Improved
quality in
human
resources
Stronger
facilities in
most SMI
areas
Use of
evidence in
decision
making
Between-
country
competition
environment
Continued
monitoring
and
evaluation
SMI as a
regional
model
SMI as a
results-based
aid model
SMI as a policy
dialogue
model
Availability
of supplies
Better health
services
Improved
processes
Technical
assistance Availability
of evidence
New and modified
policies
Need to meet
indicators
Start of a reform
of health
systems
Birth of a
culture of
accountability
Increased demand
from SMI and non-
SMI areas
Longer implementation and
planning time allowing for
changing habits in health
system
Times for measurements,
negotiations, and preparation
for following operations not
well accounted for
Application of newly-
introduced
approaches from SMI
to non-SMI areas
Positive environment for
sustainability of changes
introduced by SMI
Root cause
Obstacle
Context
Consequence
Success

57. Outline
• Introduction
• Quantitative measurement and results
• Qualitative measurement
• Overall evaluation
• Discussion and recommendations
57

58. Evaluation questions
58
1. What was the impact of SMI on maternal, neonatal and/or children under five
mortality?
2. What was the magnitude of change on maternal, neonatal and child health
outcomes in SMI target areas, and to what extent can changes be attributed
to SMI?
3. How did SMI influence changes in the coverage, quality of care and effective
coverage indicators and in health systems performance?
4. What was the contribution of SMI in the performance of health systems in the
region? What are the prospects for sustainability of SMI interventions and
results?
5. What components of SMI influenced whether outcomes were achieved or not
according to stakeholders?
6. How does the SMI model compare to other financing or intervention models?
7. What was the role of the IDB as a change agent supporting health systems
and health service provision improvements?

59. Evaluation questions
59
1. What was the effect of COVID-19 on coverage and quality of MNCH services
in the poorest regions? (including aspects of system resilience and
performance)
2. How did SMI mediate the effects of COVID-19 on coverage and quality of
MNCH services in the poorest regions? (including aspects of system
resilience and performance)
3. What was the contribution of SMI to COVID-19 pandemic preparedness,
response, and recovery? (at local/community/HF, country, and regional levels)

60. Theory of Change review & revision
60
• Where are the main pathways to impact?
• What are the hypothesized mechanisms?
• Which links can we actually test with our data?
• Which links are of the most interest?
• Which links are the most enigmatic?
• Where must we rely on the qualitative measurement the most?
• How to take into account the impact of COVID-19 on right-
hand-side outcomes?

61. Mixed-methods synthesis: Key themes
• Which components of SMI worked, and how?
• Sustainability
• Scalability, replicability, and translation to other settings
• Impact of COVID-19
61

62. Outline
• Introduction
• Quantitative measurement and results
• Qualitative measurement
• Overall evaluation
• Discussion and recommendations
62

63. Recommendations for future analysis
• Expected vs. Observed trend analyses
• Compare SMI vs. non-SMI areas
• Analyze what has happened since 2018 in countries that stopped
participating
• How does SMI model (of financing/of intervention) compare to others? à
Case study
• Organizational analysis
• Health systems frameworks (WHO model and others)
63

64. Ali H. Mokdad, Ph.D.
Principal Investigator, Salud Mesoamérica Initiative
Chief Strategy Officer, Population Health
Professor, Health Metrics Sciences
mokdaa@uw.edu
Thank you!

65. Draft evaluation framework
65
From memo update to tech form 5

66. Program Theory, RFP (SMI 2019)
66

67. Program Theory model
Data from both quantitative and qualitative measurements will inform the review and assessment of the SMI Theory of
Change, last revised in 2019 (as provided in Terms of Reference). We will review the pathways charted in the 2019
version in light of incoming quantitative and qualitative data, and identify amendments to the theory jointly with the IDB
SMI M&E Unit. In this way, we will refine the framework to be used for the final mixed-methods evaluation. Once the
newly revised Theory of Change accurately represents the hypothesized pathways of impact of SMI, we will test each
link in each causal chain using mixed analytical methods. We will map indicator results to realized service delivery
interventions as well as to related inputs, policies, and processes known to have changed over time, to identify
mechanisms of impact, to illustrate how a problem-driven approach has been assimilated into routine organizational
behaviors to improve quality, and to demonstrate the contributions of tangible and intangible resources and external
and internal context. Through this program theory evaluation model, the final SMI mixed-methods evaluation will assess
the processes behind each component of the Theory of Change to elucidate the roles of accountability, leadership,
reputational risk, and change management in achieving results on systems change (an intermediate outcome) and
population effects.
Some of the components of the Theory of Change are quantifiable, and have been estimated in previous rounds of
measurement. For example, inputs such as commodities, personnel availability, and available services are measured
through the health facility survey. Other important inputs, such as financing and policy, can be explored through review
of supplemental documents like budgets and standards of care. However, the quantitative measurement does not
encompass input themes such as governance, leadership, or community, nor intangible aspects of inputs like
productivity, motivation, or cultural competence of human resources. Interviews and focus groups with key informants
will provide qualitative measures of these intangible inputs, of mechanisms of impact such as management practices,
organizational culture, and service delivery processes, and of trademark elements of SMI including the RBF model,
regular results monitoring, quality improvement programs, and coaching. The primary outcomes in the Program Theory,
quality and coverage of healthcare, are similarly estimated quantitatively using household and health facility surveys,
and contextualized through qualitative findings. These quantitative data will also allow us to identify high-performing
units at local or subnational levels for further examination of which factors predict success through quantitative and
qualitative analysis.
67

68. Evaluation questions related to COVID-19
68
6a. What was the effect of COVID-19 on coverage and quality of MNCH services in the poorest
regions? (including aspects of system resilience and performance)
• What was the observed change over time (trend before and during the pandemic) in indicator Y in country X in SMI
areas?
• What was the observed change over time (trend before and during the pandemic) in indicator Y in country X in
comparison areas (where available)?
• What is the projection of performance for indicator Y – absent COVID-19 – in country X and in its non-intervened
areas?
• When adjusting for observable time-variant differences, to what extent can we attribute change over time to the
COVID-19 pandemic (for indicator Y in country X)?
• What qualitative evidence (anecdotes or references) suggests an effect of the COVID-19 pandemic on:
• indicator Y in country X?
• system resilience in country X?
• system performance in country X?
• regional system resilience or performance?
Hypothesized mechanisms of impact on MNCH coverage and quality indicators:
Illness/personal health effects
Deferred careseeking for other conditions
Economic hardship for families (income loss, inflation)
Secondary economic effects like migration, security
6b. How did SMI mediate the effects of COVID-19 on coverage and quality of MNCH services in
the poorest regions? (including aspects of system resilience and performance)
• When adjusting for observable differences between intervention and comparison areas, to what extent can we
attribute difference in outcomes during the COVID-19 pandemic to SMI (for MNCH quality and coverage indicator Y
in country X)?
• What is the projection of performance for indicator Y – absent COVID-19 – in SMI areas of country X?
• What qualitative evidence (anecdotes or references) suggests a (protective) effect of SMI on outcomes during the
COVID-19 pandemic (for MNCH quality and coverage indicator Y in country X)?

69. Eval questions related to COVID-19: cont.
6c. What was the contribution of SMI to COVID-19 pandemic preparedness, response, and
recovery? (at local/community/HF, country, and regional levels)
• What qualitative evidence (anecdotes or references) suggests a (protective) effect of SMI on:
• pandemic preparedness at the
• local/community/health facility level in country X?
• national level in country X?
• regional level?
• pandemic response at the
• local/community/health facility level in country X?
• national level in country X?
• regional level?
• pandemic recovery at the
• local/community/health facility level in country X?
• national level in country X?
• regional level?
• Do pandemic preparedness outcomes* differ between:
• SMI and non-intervened areas in country X?
• Countries that left after the second operation of SMI and countries that continued to the third operation?
• Do pandemic response outcomes* differ between:
• SMI and non-intervened areas in country X?
• Countries that left after the second operation of SMI and countries that continued to the third operation?
• Do pandemic recovery outcomes* differ between:
• SMI and non-intervened areas in country X?
• Countries that left after the second operation of SMI and countries that continued to the third operation?
*These outcomes and data sources have not been defined in the theory of change nor evaluation framework
Example pandemic response outcome: hospital saturation
Example pandemic recovery outcome: vaccination coverage over time
69

70. Eval question explored with VR/GBD data
70
1. What was the impact of SMI on maternal, neonatal and/or children under five
mortality?
• What was the trajectory of maternal mortality in country X preceding and during the time frame of SMI?
• How did the trajectory differ in SMI area vs. non-intervened areas?
• How did the trajectory differ before and during the COVID-19 pandemic?
• Can any differences be linked or attributed to SMI?
• What quantitative evidence suggests an effect of SMI on intermediate outcomes (e.g., obstetric
complications, maternal death during or immediately after delivery)?
• What qualitative evidence (anecdotes or references) suggests an effect of SMI on intermediate
outcomes?*
• What was the trajectory of neonatal mortality in country X preceding and during the time frame of SMI?
• How did the trajectory differ in SMI area vs. non-intervened areas?
• How did the trajectory differ before and during the COVID-19 pandemic?
• Can any differences be linked or attributed to SMI?
• What quantitative evidence suggests an effect of SMI on intermediate outcomes (e.g., neonatal
complications, neonatal death during or immediately after delivery)?
• What qualitative evidence (anecdotes or references) suggests an effect of SMI on intermediate
outcomes?*
• What was the trajectory of under-5 mortality in country X preceding and during the time frame of SMI?
• How did the trajectory differ in SMI area vs. non-intervened areas?
• How did the trajectory differ before and during the COVID-19 pandemic?
• Can any differences be linked or attributed to SMI?
• What quantitative evidence suggests an effect of SMI on intermediate outcomes?*
• What qualitative evidence suggests an effect of SMI on intermediate outcomes?*
This investigation requires receipt of VR/admin data for the relevant countries
* We expect there may not be sufficient evidence to address these questions
Note: this evaluation question does not apply for Costa Rica per IHME’s interpretation

71. Eval questions with quant and qual data
2. What was the magnitude of change on maternal, neonatal and child health
outcomes in SMI target areas, and to what extent can changes be attributed to SMI?
• What was the observed change over time in indicator Y in country X in SMI areas?
• What was the observed change over time in indicator Y in country X in comparison areas (where
available)?
• When adjusting for observable differences between intervention and comparison areas, to what extent can
we attribute change over time to SMI (for indicator Y in country X)?
• What qualitative evidence (anecdotes or references) suggests an effect of SMI on indicator Y in country X?
Note the integration of question 2 with question 6b for countries participating in the third operation.
3. How did SMI influence changes in the coverage, quality of care and effective
coverage indicators and in health systems performance?
• Same components as above for the coverage and quality indicators measured quantitatively
• What qualitative evidence (anecdotes or references) suggests an effect of SMI on health systems
performance?
• Through which mechanisms does qualitative evidence suggest that SMI affected indicator Y?
• Through which mechanisms does qualitative evidence suggest that SMI affected systems performance?
Countries with 3 rounds comparison data: Honduras, Nicaragua
Countries with 2 rounds comparison data: Guatemala, Mexico
Country with 1 round comparison data: El Salvador
Countries without comparison data: Costa Rica (reproductive health outcomes only), Belize, Panama
71

72. Eval questions to be explored qualitatively-1
4. What was the contribution of SMI in the performance of health systems in the
region [note this question has overlap with question (3)]? What are the prospects for
sustainability of SMI interventions and results?
• What qualitative evidence (anecdotes or references) suggests an effect of SMI on health systems
performance?
• What evidence suggests that SMI interventions influenced performance during the system stress of the
COVID-19 pandemic?
• What qualitative evidence suggests SMI interventions have been operationalized/integrated to health
systems?
• What evidence suggests SMI interventions will continue to be funded through national spending or other
sources?
• Which SMI interventions have been operationalized and/or funded in countries that left SMI after the
second operation?
• What is the performance of MNCH outcomes (indicator Y in country X) in these cases?*
* We expect data may not be available to address these questions
5. What components of SMI influenced whether outcomes were achieved or not
according to stakeholders?
• How did component V influence (or not) outcome W according to stakeholder Z?
* We need to define the relevant list of components V
• regional model
• policy dialogue model
• RBF model
• technical assistance/coaching/mentoring
• implementation science
• responsiveness to context
• focus on local/community/facility level
• use of evidence
• external evaluation
72

73. Eval questions to be explored qualitatively-2
73
7. How does the SMI model compare to other financing or intervention models?
• How do unique aspects of the SMI model compare to other RBF project examples (before and during the
COVID-19 pandemic)?*
• technical assistance/coaching/mentoring,
• implementation science,
• responsiveness to context, focus on local/community/facility level, ”decolonization”,
• use of evidence,
• external evaluation
• How do unique aspects of the RBF model (public-private partnership/blending financing) compare to other
health intervention examples (before and during the COVID-19 pandemic)?*
• From the perspective of country X Ministry of Health, how did this funding mechanism differ from a
loan?
• How did SMI change IDB’s “business as usual”?**
8. What was the role of the IDB as a change agent supporting health systems and
health service provision improvements?**
*Is it the first component, the second component, or both that are of interest?
**It would be helpful to have more input from IDB about how this question would be explored

74. Key informant profiles
• Respondents will be selected from across the 8 countries
represented in SMI, with emphasis on the 4 countries that
participated in the 3rd Operation
o Representation from intervention and comparison areas
Study Audience N/A Honduras Nicaragua El Salvador Belize Chiapas Guatemala
Costa
Rica
Panama
Funder representatives 8
IDB/SMI coordinating unit 6 1 1 1 1 1 1 1 1
Ministry of Health (MoH) 4 4 4 4 2 2 2 2
Health care providers:
SMI
8 8 8 8 3 3 3 3
Health care providers:
non-SMI
3 3 3 3 0 0 0 0
TBAs/CHWs 2 2 2 2 0 0 0 0
74

75. Key informant profiles
• Ministry of Health respondents:
o The 4 individuals most knowledgeable about SMI implementation
per country
o Ideally a mix of currently-involved respondents and respondents
with knowledge of SMI’s history (at least the 2nd Operation)
o Will depend on the organization of the health system and SMI
activities in each location
75

76. Key informant profiles
• Healthcare providers:
o Medical doctors, nurses, and medical directors (a mix of all 3)
o At least 5 years of clinical experience, but preferably 10 years
o At least 1 year in current position
o SMI respondents: At least 5 years working in SMI areas
o Non-SMI respondents: Should not have worked in SMI areas
o Ability to participate in an online or telephone interview
• Traditional birth attendants/community health workers:
o Traditional birth attendant or community health worker
o At least 5 years practicing in SMI areas, but preferably 10 years
o Ability to participate in an online or telephone interview
76