Matthieu antony

1. Design and implementation
of operational components
of Result-Based Financing
schemes
The case of the verification in Benin
Matthieu Antony
Agence Européenne pour le Développement et la Santé
European Agency for Development and Health
mantony@aedes.be
Dar es Salaam / 25.11.2015

2. Introduction
• Few evaluations have been published and many Randomized Controlled Trials
(RCTs) are currently ongoing to assess the effects of RBF
- Basinga and al, 2011; Falisse and al, 2014; Bonfrer and al, 2014; Huillery and al, 2015
• Quantitative methods are often silent on the paths and processes through which
results are achieved, and on wider health system effects
- Witter and al, 2013
• Our study aims at analyzing how implementation challenges can modify the
original design, affect RBF’s theory of change and therefore the scheme’s potential
for results
– We focus on the ‘verification of results’ in Benin
– We also draw some lessons on the design and implementation of key operational
components of RBF (such as verification), and on verification itself

3. Introduction
Role of verification in RBF’s Theory of Change
Verification
- Calculate rewards  pay bonus in a transparent, timely and regular manner
- Detect fraud & signal a threat of sanctions to providers
- Channel the “voice” of communities and patients, and improve providers’ accountability
- Improve governance & stewardship through DHMTs’ involvement
- Provide reliable data  data analysis  feedback and “coaching”
* Should be financially viable
Performance
(quantity + quality of outputs)
Payment

4. Context
The RBF project pilot
• Started in October 2011 with WB
funds – in 8 districts (“zones de
santé”)
• Population covered: 2,377,559
• Focus on health facilities’
productivity and quality of health
services
• On-going RCT
• Scale up to 21 districts with the
support of GF and GAVI (April
2015)

5. Context
Key roles in the scheme’s management
World Bank Funding
A Project Coordination Unit at MoH in charge of
signing contracts, purchasing services and
payment transfers.
An implementation agency in charge of technical
assistance, coaching and verification

6. Context
Design of the RBF scheme
• Contracts signed, and verification performed, with facilities in
both control and intervention arm (n = 188)
• Quantity indicators
– 8 indicators at community level
– 28 indicators at health centre level
– 14 indicators at hospital level
• Quality checklist
– 400 items
• “Results validation meeting” at central level on quarterly basis

7. Context
Design of the verification of results
Quantity verification
Monthly basis
Technical quality evaluation
Quarterly basis
Community verification
Quarterly basis
• Counting health services produced
from facilities’ registries
– performed by implem. agency
• Assessing quality against checklist
– carried out by DHMTs for health centres & by
peers for hospitals
– under the supervision of implem. agency
• Tracing patients in communities
• Patients’ satisfaction survey
– carried out by contracted Community Based
Organizations (CBOs)
– implem. agency in charge of sampling of
patients to track, CBOs supervision, reports’
validation

8. Research methods
Secondary data
Project documents
(reports, budgets) 2012-2015
TA activities daily timesheets
(n=20)
RBF specific data series
2012-2015 (n=188)
Focus Groups with CBOs (n=5)
Document review
Participant
observation

9. Findings
Results of verification
Indicators Timeframe Results
Difference between declared
and verified data on num. of
services provided
Duration of the project
Varies between 4% (new
users of family planning) and
51% (patient referred to
hospital)
Quality scores Third quarter of 2015 Between 4% and 96%
Num. of patients missing
from community tracing*
Third quarter of 2013 (first
community survey)
Between 12% and 47%
Patients’ satisfaction* - -
* Data collected, but not systematically analyzed

10. Findings
Implementation issues in verification processes
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities.
No analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification

11. Findings
Little time for data analysis, feedback and coaching
Proportion of time spent on different activities for imp. agency
staff in the field, based on timesheets of implem. agency staff
Quantitative
verification
46%
Community
verification
26%
Quality
evaluation
18%
Data
recording
10%
Verification
67%
Others
16%
Data analysis
and Meeting at
central level
9%
Training and
Coaching
8%

12. Findings
verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in
data reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification

13. Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification

14. Findings
Payment delays
Quarter Quarter End
Invoice
Transmission
Delay
due to verif.
Transfer
Delay
due to transfer
Total Delay
(month)
Q2 2012 Jun-12 Sep-12 3,5 Nov-12 1,5 5
Q3 2012 Sep-12 Dec-12 3,5 Apr-13 3 6,5
Q4 2012 Dec-12 Apr-13 4,0 Jul-13 3,5 7,5
Q1 2013 Mar-13 Jul-13 4,5 Nov-13 3,5 8
Q2 2013 Jun-13 Sep-13 3,5 Dec-13 3 6,5
Q3 2013 Sep-13 Dec-13 3,5 Feb-14 2 5,5
Q4 2013 Dec-13 Apr-14 4,0 May-14 0,5 4,5
Q1 2014 Mar-14 Jun-14 3,5 Jul-14 0 3,5
Q2 2014 Jun-14 Sep-14 3,5 Jan-15 3,5 7,0
Q3 2014 Sep-14 Dec-14 3,0 Feb-15 2 5

15. Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for
feedback and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity reported
are rarely applied. Rewards for higher patients’
satisfaction, or lower discrepancies in data reported are
also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and
stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of
communities and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification

16. Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the “voice” of communities
and patients
- Elite appropriation of CBOs
- No analysis of (no rewards/sanctions based on)
patients’ satisfaction survey
Financially viable Relatively costly, esp. community verification

17. Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification

18. Findings
Verification processes are costly
Funds to implem
agency for verification
activities (only)
Funds to CBOs
0.33
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Funds for RBF bonus to facilities Total funds for verification activities
1,595,644 USD
718,295 USD
0.12

19. Discussion & conclusions
• Methodologically
What are Impact Evaluation really testing, if the theory of change is
modified by implementation challenges?  it is essential to include an
analysis of the implementation processes in the RBF schemes’
evaluations
• Design
The design of the key elements of RBF schemes should be adapted to
the context and iteratively modified and improved during implementation
• Verification
Our analysis leads us to question whether the rational for three-pronged
(incl. community verif.) and thorough (i.e. non-random) verification is
valid under all conditions.

20. Acknowledgment
• Thanks to all the Technical Assistants of the Implementation Agency
who provided time, insight and expertise that greatly assisted the
research
• To Maria Paola Bertone and Olivier Barthes for assistance with
methodology, and all the AEDES team for comments that greatly
improved this presentation
• Many thanks also to Dr Akpamoli and his team and to Maud Juquois
and Ibrahim Magazi from the World Bank for their constant support
during the implementation of the project and with this research.