Soumya Balasubramanya Making Impact Evaluation Matter: Better Evidence for Effective Policies and Programs September 3-5, 2014 Manila, Philippines

1. Photo: Photo :DToavmid v Banra Cziaekre/InWbeMrgI he/IWMI
www.iwmi.org
Water for a food-secure world
Photo: Andrea Silverman/IWMI
Name
Date
Event
Location
Time matters:
‘short’ term vs. ‘medium’ term impacts
1
Soumya Balasubramanya
Making Impact Evaluation Matter: Better Evidence for Effective Policies and Programs
September 3-5, 2014
Manila, Philippines
Balasubramanya, S., Pfaff, A., Bennear, L., Tarozzi, A., Ahmed, K.M., van Geen, A. (2014). Evolution of households’ responses to the groundwater arsenic crisis in Bangladesh: information on environmental health risks can have increasing behavioral impact over time. Environment and Development Economics, 19(05), pp. 631-647.

2. www.iwmi.org
Water for a food-secure world
Summary
•
Impact of an arsenic information and testing campaign on adoption of risk-avertive behaviors over time.
•Followed households over two time periods and tracked behaviors and perceptions over 2003-05 and 2005-08.
•Impact of information on risk-avertive behaviors depends on ability to recall information
–Households that recalled: behaviors were persistent and double the # of households took up behaviors over time
–Households that didn't’ recall: observed perverse behaviors
2

3. www.iwmi.org
Water for a food-secure world
Background
•
GW is primary drinking source; accessed through private tubewells (shallow, < 30 m; 90% of wells are shallow )
•
GW contaminated with naturally occurring arsenic; unpredictable in shallow aquifer
•
WB- funded countrywide groundwater testing program b/w 1999-2003
•
Tested households’ tubewells for arsenic; painted spouts red/green
•
Encouraged households to share green wells
•
Village level awareness campaigns
•
Media campaigns
3

4. www.iwmi.org
Water for a food-secure world
What happens within two years of testing?
•
Schoenfeld, 2005
–
Census of all tubewells tested by WB in 2003; in 62 villages of Araihazar Upazila
–
Interviewed the wives of the well owners
–
Recorded color of well (observed by enumerator)
–
Elicited safety perception (asked respondent if well was safe or unsafe)
–
Elicited switching behavior b/w 2003-05
•
Switching behaviors from unsafe wells ~20%
4

5. www.iwmi.org
Water for a food-secure world
What happens over time?
•
Household values both convenience (well nearby) and safe drinking water (green well)
–
Switch if cost (switching) < benefit (switching)
•Can the costs and benefits change over time?
–‘Social learning’ by observing peers (Miguel and Kremer, 2004; Munshi and Myaux, 2006)
–Household ‘loses’ information
–Concern about health risk reduces (Karlan et al., 2010)
–Safe well owners reduce access (Fehr and Fischbacher, 2003; Hanchett et al., 2002)
5

6. www.iwmi.org
Water for a food-secure world
Two Questions
•
Is well-switching a persistent behavior change?
•
How does adoption of well-switching evolve over time?
6

7. www.iwmi.org
Water for a food-secure world
Methods and Data
•
Follow-up study on households whose switching behaviors had been previously studied in Schoenfeld, 2005
–
Randomly selected subset of households in 59 villages
–
Interviewed same person as in previous survey
–
1557 households/wells (1038 unsafe; 519 safe)
–
Use sampling weights
7

8. www.iwmi.org
Water for a food-secure world
Data 8

9. www.iwmi.org
Water for a food-secure world
Consistency in recall
9
•
78% recalled switching behavior of 2003-2005 consistently
•
Always consider households with consistent recall of 2003-2005 switching behavior
•By 2008, no color left on any well spout
•78% recalled well safety consistently
(well color = safety perception in 2005 = safety perception in 2008)

10. www.iwmi.org
Water for a food-secure world
Did impact of testing increase or erode over time?
10
•
Consider households with consistent recall of well tests
•If switched in 2003-2005, check if switches were persistent
–16% of households at unsafe wells switched during 2003-2005
•14% stayed put
•2% switched to a second well they thought safe
•If did not switch during 2003-2005, check if switched during 2005-2008
–17% of households at unsafe-tested wells had switched
–Switching over five years doubles that over two years

11. www.iwmi.org
Water for a food-secure world
11

12. www.iwmi.org
Water for a food-secure world
What about those that ‘forget’ information?
12
•
Consider households with inconsistent recall of well tests (remembered correctly in 2005, but not in 2008)
•In 2005
–Paint visible on all wells (1039 red/ 519 green)
–Almost all households recalled well safety correctly (97% unsafe/ 95% safe)
•In 2008
–Paint washed off
–77% unsafe tested households recalled safety accurately
–81% safe tested households recalled safety accurately

13. www.iwmi.org
Water for a food-secure world
Would it help to reinforce provision of arsenic information?
13
•Safe-tested hhs that changed safety perceptions after 2005: 26% switched in 2005-2008
•Unsafe-tested hhs that changed safety perceptions after 2005: 11% switched in 2005-2008

14. www.iwmi.org
Water for a food-secure world
14

15. www.iwmi.org
Water for a food-secure world
Discussion
•
Behavioral:
•
Recall of arsenic information changes over time
•
Recall is critical for behavior
•Evaluation:
•Short-term evaluations present a snap shot
•Impacts are heterogeneous
•Policy perspective:
•Cheap, responsible for most reduction in arsenic exposure (World Bank, 2005; Johnston et al., 2010; Ahmed et al., 2006)
•Supporting continued well testing: new well constantly installed; 1/3 wells untested
15

16. www.iwmi.org
Water for a food-secure world
Thank you!
s.balasubramanya@cgiar.org
16

17. www.iwmi.org
Water for a food-secure world
17

18. www.iwmi.org
Water for a food-secure world
18