A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. The Science of Delivery:
Impact Evaluation Results and
Lessons from RBF
Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo
Learning from Implementation

2. The Why and How of
Top Ten Lessons
• Review of ongoing PBF operations suggested
some consistent challenges & useful lessons
• Focusing on the most important lessons will
facilitate learning by other teams
• Selection of the Top Ten was done by TTLs
and RBF experts involved in a portfolio
review of eight ongoing RBF schemes
2

3. Lesson #1
“Show me the money!”
- Make timely payments
• Over 70% of projects have had issues with
making payments on time
3

4. Outpatient visits in Nigeria declined when
payments were interrupted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Adamawa
Nasarawa
Ondo
Payments
interrupted
4

5. Lesson 1: “Show me the Money”
Make Timely Payments
Recommendations
Track time required for payments to reach facilities
Establish explicit standards for maximum allowable
delay (2 or 3 months)
Provide resources up-front to health facilities: (i)
gains their confidence; (ii) indicates PBF has
started; and (iii) provides investment funds
5

6. Lesson #2
“Keep moving the goalposts!”
Continuous Quality Improvement
(CQI) implies Changing the
Quality Indicators
• Many facilities make rapid improvements in
quality and then plateau
6

7. 0
20
40
60
80
100
Mwaro
Muramvya
Kirundo
Cibitoke
Buja-Rural
Kayanza
Ngozi
Makamba
Rutana
Bubanza
Bururi
Gitega
Karuzi
Muyinga
Ruyigi
Cankuzo
Buja-Mairie
Qualityscores by province
Average quality score plateaued after 3
quarters in Burundi
7

8. Lesson 2: “Keep moving the goalposts!”
Continuous Quality Improvement (CQI)
implies Changing Quality Indicators
Recommendations
Quantified quality checklists should emphasize
process over structural indicators
Revise checklist every 1–2 years to emphasize
continuous improvement
Include: 1) observation of patient-provider
interaction; 2) records reviews (need standard
records); 3) Vignettes; 4) Exit interviews, etc.
Invest in training supervisors
8

9. Lesson #3
“To be discerning, We keep
on Learning”
• RBF as a tool allows for experimentation,
learning and course-correction
9

10. What did Zimbabwe do?
Quantitative analysis
Qualitative analysis
 Design demand and supply-side RBF (urban pilot)
 Targeted urban maternal voucher (demand-side)
 Pay-for-quality RBF
 Reviewed pricing
 Improved quality checklists
10

11. Operational research studies should be budgeted
Carry out process evaluations and use results for
implementation dialogue and decision-making.
Participate in RBF portfolio reviews – look for new
ideas
Lesson 3: To be discerning - Keep on
Learning
11

12. Lesson #4
“Money is NOT the root of all
evil, Lack of Money is”
– Worry about Financial
Sustainability
• PBF should not be made to substitute
deficiencies in national health financing
12

13. PBF sustainability requires wider health
system reforms
 Burundi PBF is
combined with FHC
 Government pays almost
half–1.4% of budget
allocated
 Common pool for
development partners– two
decreased contribution
 Cumulative deficit of
almost US$ 8M
 Rwanda PBF health
reforms were part of
larger reforms
 CBHI reform
 HR reforms
13

14. Where is the PBF money going?
14

15. 35%
37%
39%
40%
45%
11%
10%
10%
9%
5%
16%
16%
12%
13%
10%
29%
31%
31%
33%
37%
8%
7%
7%
5%
3%
Q2 2012
Q3 2012
Q4 2012
Q1 2013
Q2 2013
Intervention HFs Intervention DMO Hospital
Control 1 HFs Control 1 DMO
Zambia: Aggregate consumption of PBF funds from Q2, 2012 – Q2, 2013
Where is the PBF money going?
15

16. Recommendations
Track overall PBF expendituresand understand where
most PBF funds are going
It is easier to increase tariffs than decrease them. This
suggests starting with relatively lowertariffs and
increasing them
Insist that Government(s) live within an envelope
budget.
Be careful about increasing expenditure untied to
improved performance.
Lesson 4: “Money is NOT the root of all
evil, Lack of Money is” – Worry about
Financial Sustainability
16

17. Lesson #5
“Math-Phobes of the World
Unite!”
– Use your data
• Data is vital but under-utilized, despite a lot
of effort invested into collecting, verifying
and putting payment data on the web
17

18. Internet applications with public front-
end displaying performance & financial
information
18
Burundi
Benin
Nigeria

19. How Zambia uses its data
• Quarterly analysis
• Extensive analysis by expert
o Diff-in-diff analysis of trends using HMIS to compare
across the 3 study arms
RBF vs. Additional financing
Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783
p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024
RBF vs. Control
Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761
p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011
Attendance
outpatient
total (calc)
Immunised
fully <1 year
new
Antenatal
1st visit
before 20
weeks
IPT 3rd
dose to
pregnant
woman
Postnatal
care within
6 days
Attendance
Family
Planning
total (Calc)
Delivery by
skilled
personnel
19

20. Lesson 5: Math-Phobes of the World
Unite! – Use your data
Recommendations
 Create reliable systems for data collection and
compilation
 Ensure consistent flow of PBF data with minimal
reporting burden on health workers
 Make it somebody’s job to analyze data
 Deploy software to help with data analysis
 Crowd-source analysis!
 Attempt to triangulate all available data sources to
determine trends in utilization and outcomes
20

21. Lesson #6
“Understand your
Customers”
-Demand side issues may be
under-appreciated
• There is need to understand root-causes of
the problem RBF is trying to solve.
21

22. Even with large improvements in utilization
since PBF was introduced, absolute
coverage levels remain low in Nigeria
Barriers to service utilization in two Nigerian LGAs:
 Transportation Challenges
 Variable & unpredictable fees for Services and Drugs
 Social and cultural Barriers
Proposed solution: A voucher scheme to improve service
uptake
 Continuous cycle of learning and responding to help
households overcome service utilization barriers and
improve health outcomes.
22

23. Lesson 6: Understand your customers –
Demand-side issues may be under-
appreciated
Recommendations
Consider demand-sideissues if coverage levels remain
low in spite of PBF (even if PBF has made a big
difference).
Understand barriers to access – cultural, social and
financial barriers through household surveys; focus
groups, key informant interviews, etc.,
Options for addressing demand side include: (i) closer
work with community structures; (ii) CCTs; (iii)
vouchers; (iv) BCC through facility staff or NGOs
23

24. Lesson #7
“Away with Flat-Liners”
- Identify poorly performing
regions & facilities. Do
something about them!
• Detailed understanding of predictors of
success and failure is required
24

25. There are clearly facilities that are NOT making
progress i.e. “flat-liners”
Non-determinants
• Number of staff
• Remoteness
• Qualification of in-
charge
• Business planning
-
20
40
60
80
100
120
140 Positive deviants
• Community
engagement
• Management
capacity of in-
charge
Institutional Deliveryin Adamawa health facilities,
normalized by 100,000 population
25

26. Lesson 7: Away with “Flat-Liners”
Recommendations
Use existing data to identify facilities that have not
improved or are not performing well
Carry out quantitative and qualitative studies (e.g.
Nigeria and Zimbabwe)
Consider the kind of technical support needed for
the poorly performing facilities and re-examine
what is currently available.
26

27. Lesson #8
“You can run but you can’t
hide”
– Worry about the politicaI
threats to PBF
• Even successful RBF schemes face political
pressures that threaten their sustainability
27

28. Zambia RBF Pilot
 RBF PIU not mainstreamed into MOH structure
 Institutional changes in MOH
 Key champions moved
28
Watch for winners and losers

29. Recommendations
Political economy expertise should be engaged to
study factors important for scale-up and
sustainability of RBF, to figure out what can be
done differently.
Institutionalizing roles such as purchasing and
project implementation within government
structures may help improve buy in
29
Lesson 8: Worry about the political
threats to PBF

30. Lesson #9
“Even the Best Laid Plans
can go wrong”
– Analyze your Project Design
to Ensure Assumptions
Remain Valid
• Understand key-determinants of the PBF program
and the linkages between them to produce results
30

31. Benin
 Bureaucracy makes it
difficult for facilities to
spend PBF incentives
 May reduce health worker
motivation to perform
 User-fees easier to spend
 High fees may decrease
healthcare utilization
31
Challenges with health facility autonomy
in Benin and nested PBF in Afghanistan
made implementation different
Afghanistan
 Unique environment of
performance agreement
with NGO
 Ongoing security concerns
 PBF overlaid on PPA

32. Recommendations
Periodically review the progression of elements in your
theory of change, not just what you are trying to
achieve, but how you get there
Useful analyses for every project include:
 What is additional percentage of PBF to health workers'
take-home pay? Are they motivated by this amount?
 Balance of supervision versus autonomy at the health
facility level
 Resources available at the front-line for health facilities
32
Lesson 9: “Even the Best Laid Plans can
go wrong” – analyze project design to
ensure assumptions remain valid

33. Lesson #10
“KISS – Keep Impact Studies
Straightforward”
- Avoid Impact Evaluation
Questions that are TOO Subtle
• It is very difficult to assess differential
impacts when there is minimal variation
between experimental groups
33

34. 0
600
1200
1800
2400
1* 2 3 4 1 2
2012 2013
Children immunized
Control
Treatment
The RBF intervention and additional
financing group in Benin are looking the
same
 No clear difference is
emerging
 Another control group
with no added finances
has been included for
counterfactual
34
0
2000
4000
6000
8000
1* 2 3 4 1 2
2012 2013
Assisted delivery
Control
Treatment

35. Experimental & Control groups in Nigeria
– try hard to be explicit on differences !!!!
35

36. Recommendations
Assess in advance if experimental variations are
likely to make a difference in outcomes.
Work with the IE team from the start
Ensure counterparts fully understand research
questions, and are convinced of utility.
36
Lesson 10: “KISS – Keep Impact Studies
Straightforward” Avoid Impact Evaluation
Questions that are TOO Subtle

37. A ship in the harbor is safe
– but that is not
what ships are built for.-John A Shedd (1928)
Implement, learn and share
lessons!