Fundamentals of Program Impact Evaluation

1. Peter M. Lance, PhD MEASURE Evaluation University of North Carolina at Chapel Hill MARCH 31, 2016 Fundamentals of Program Impact Evaluation

2. Global, five-year, $180M cooperative agreement Strategic objective: To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time. Project overview

3. Improved country capacity to manage health information systems, resources, and staff Strengthened collection, analysis, and use of routine health data Methods, tools, and approaches improved and applied to address health information challenges and gaps Increased capacity for rigorous evaluation Phase IV Results Framework

4. Global footprint (more than 25 countries)

6. How Do We Know IfAProgram MadeADifference? ABrief Helicopter Tour of Methods for Estimating Program Impact

7. • The Program Impact Evaluation Challenge • Randomization • Selection on observables • Within estimators • Instrumental variables

14. Newton’s “Laws” of Motion 𝐹𝑜𝑟𝑐𝑒 = 𝑀𝑎𝑠𝑠 ∙ 𝐴𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛

17. Did the program make a difference?

18. Did the program cause a change in an outcome of interest Y ? (Causality)

19. Our outcome of Interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌 : 𝑌0 : 𝑌1 :

20. Our outcome of interest What happens if an individual does not participate in a program What happens if that individual does participate in a program Potential Outcomes 𝑌𝑖 : 𝑌𝑖 0 : 𝑌𝑖 1 :

23. What happens if the individual participates {Causal} Program Impact 𝑌𝑖 1 − 𝑌𝑖 0 = Program Impact What happens if the individual does not participate

28. 𝑃𝑖 = 1ifindividual 𝑖 participates 0if individual 𝑖 does not participate Program Participation

29. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome

30. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

31. 𝑌𝑖 = 1 ∙ 𝑌𝑖 1 + 1 − 1 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

32. 𝑌𝑖 = 𝑌𝑖 1 + 0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 1

33. 𝑌𝑖 = 𝑌𝑖 1 Observed Outcome 𝑃𝑖 = 1

34. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome

35. 𝑌𝑖 = 𝑃𝑖 ∙ 𝑌𝑖 1 + 1 − 𝑃𝑖 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0

36. 𝑌𝑖 = 0 ∙ 𝑌𝑖 1 + 1 − 0 ∙ 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0

37. 𝑌𝑖 = 𝑌𝑖 0 Observed Outcome 𝑃𝑖 = 0

38. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome

42. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome Fundamental Identification Problem of Program Impact Evaluation

43. 𝑌𝑖 1 , 𝑌𝑖 0 Observed Outcome Fundamental Identification Problem of Program Impact Evaluation

44. Individual Population

45. Individual Population Hi. They call me individual i

46. Individual Population ?!?

48. 𝑌𝑖 1 , 𝑌𝑖 0

49. 𝑌𝑖 1 , 𝑌𝑖 0

50. An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value

51. An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents An expected value is the true average of a random variable across a population Expected Value

52. An expected value is the true average of a random variable across a population 𝐸 𝑋 = sometruevalue Expected Value

53. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

54. 𝑬 𝒄 = 𝒄 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

55. 𝐸 𝑐 = 𝑐 𝑬 𝒄 ∙ 𝑾 = 𝒄 ∙ 𝑬 𝑾 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

56. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝑬 𝑾 + 𝒁 = 𝑬 𝑾 + 𝑬 𝒁 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

57. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝑬 𝑾 − 𝒁 = 𝑬 𝑾 − 𝑬 𝒁 𝐸 𝑎 ∙ 𝑊 ± 𝑏 ∙ 𝑍 = 𝑎 ∙ 𝐸 𝑊 ± 𝑏 ∙ 𝐸 𝑍 Expectations: Properties

58. 𝐸 𝑐 = 𝑐 𝐸 𝑐 ∙ 𝑊 = 𝑐 ∙ 𝐸 𝑊 𝐸 𝑊 + 𝑍 = 𝐸 𝑊 + 𝐸 𝑍 𝐸 𝑊 − 𝑍 = 𝐸 𝑊 − 𝐸 𝑍 𝑬 𝒂 ∙ 𝑾 ± 𝒃 ∙ 𝒁 = 𝒂 ∙ 𝑬 𝑾 ± 𝒃 ∙ 𝑬 𝒁 Expectations: Properties

59. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

60. 𝑬 𝑾 ∙ 𝒁 ≠ 𝑬 𝑾 ∙ 𝑬 𝒁 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

61. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝑬 𝑾 𝒁 ≠ 𝑬 𝑾 𝑬 𝒁 𝐸 𝑓 𝑊 ≠ 𝑓 𝐸 𝑊 Expectations: Properties

62. 𝐸 𝑊 ∙ 𝑍 ≠ 𝐸 𝑊 ∙ 𝐸 𝑍 𝐸 𝑊 𝑍 ≠ 𝐸 𝑊 𝐸 𝑍 𝑬 𝒇 𝑾 ≠ 𝒇 𝑬 𝑾 Expectations: Properties

63. 𝑌𝑖 1 − 𝑌𝑖 0 Average Treatment Effect (ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Hi there Individual Impact

64. 𝑌𝑖 1 − 𝑌𝑖 0

65. 𝐸 𝑌𝑖 1 − 𝑌𝑖 0

66. Average Treatment Effect (ATE) 𝐸 𝑌1 − 𝑌0 Average Effect of Treatment on the Treated (ATT) 𝐸 𝑌1 − 𝑌0|𝑃 = 1 Treatment Effects

69. Suppose that we have a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

70. Suppose that we have a sample of 𝑖 = 1,…, 𝑛 individuals…. …but for each individual 𝑖 we observe either 𝑌𝑖 1 or 𝑌𝑖 0 … …but not both So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

71. Remember, however, a key property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

72. Remember, however, a key property of expectations: 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 …but this means that in principle we could estimate E 𝑌1 and E 𝑌0 separately So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

73. For instance, suppose that in our sample we have: 𝑛 𝑃 participants(𝑃𝑖 = 1) and 𝑛 𝑁 non-participants(𝑃𝑖 = 0) (hence 𝑛 𝑃 + 𝑛 𝑁 = 𝑛) So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

74. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

75. Then an estimator of 𝐸 𝑌1 is 𝒀 𝟏 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

76. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝒏 𝑷 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

77. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

78. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝒏 𝑷 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

79. Then an estimator of 𝐸 𝑌1 is 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝒀𝒋 𝟏 𝑛 𝑃 calculated with the 𝑛 𝑃 participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

80. Similarly, an estimator of 𝐸 𝑌0 is 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 calculated with the 𝑛 𝑁 non-participants out of the sample of 𝑛 individuals So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

81. So then an estimate of 𝐸 𝑌1 − 𝑌0 = 𝐸 𝑌1 − 𝐸 𝑌0 is 𝑌1 − 𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 So how do we estimate 𝑬 𝒀 𝟏 − 𝒀 𝟎 ??

82. But is it a good estimate??

86. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝑌1 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 = 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁

91. So we have two samples of size 𝒏 By random chance, between the two samples we almost surely have 1. A different precise mix of individuals 2. A different number of participants (𝑛 𝑃) and non-participants (𝑛 𝑁) 3. Different estimates 𝑌1 and 𝑌0 of 𝐸 𝑌1 and 𝐸 𝑌0 : 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑌0 = 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 𝒀 𝟏 𝑬 𝒀 𝟏

102. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷

103. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷

109. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏

110. 𝒀 𝟏 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝒏 𝑷 = 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏

113. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

114. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

115. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

116. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿 2nd Rule: 𝑬 𝑿 + 𝒁 = 𝑬 𝑿 + 𝑬 𝒁 𝑬 𝒊=𝟏 𝒏 𝑷 𝑿𝒊 = 𝒊=𝟏 𝒏 𝑷 𝑬 𝑿𝒊

117. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿

118. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 1ST Rule: 𝑬 𝒄 ∙ 𝑿 = 𝒄 ∙ 𝑬 𝑿 𝑬 𝟏 𝒏 𝑷 ∙ 𝑿 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝑿

119. 𝑬 𝒀 𝟏 = 𝑬 𝒋=𝟏 𝒏 𝑷 𝒀𝒋 𝟏 𝒏 𝑷 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝑬 𝒋=𝟏 𝒏 𝑷 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝟏 𝒏 𝑷 ∙ 𝒏 𝑷 ∙ 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏

120. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

121. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

122. 𝑬 𝒀 𝟏 = 𝑬 𝒀𝒋 𝟏 𝑬 𝒀 𝟏 = 𝑬 𝒀 𝟏 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

123. 𝑬 𝒀𝒋 𝟏 = 𝑬 𝒀 𝟏

126. 𝑬 𝒀 𝟏

127. 𝑃 = 0 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝒀 𝟏

128. 𝑃 = 0 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 1 𝑃 = 1 𝑃 = 0 𝑃 = 0 𝑃 = 1 𝑃 = 0 𝒀 𝟏

129. 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝒀 𝟏

130. Z W “Z Causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

131. Z W “Z causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

132. Z W “Z causes W” 𝑬 𝑾|𝒁 ≠ 𝑬 𝑾

133. X Y1

134. X Y P

135. X Y P 0

136. X Y P

137. X Y P

138. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

139. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

140. 𝐸 𝑋|𝑃 = 1 ≠ 𝐸 𝑋 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 X Y1

141. X Y P

142. 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1 𝑃 = 1

144. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0

145. 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 1 ≠ 𝐸 𝑌0 𝐸 𝑌0 |𝑃 = 0 ≠ 𝐸 𝑌0 |𝑃 = 0 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 1 ≠ 𝐸 𝑌1 𝐸 𝑌1 |𝑃 = 0 ≠ 𝐸 𝑌1 |𝑃 = 0

146. The estimator 𝑌1 − 𝑌0 = 𝑗=1 𝑛 𝑃 𝑌𝑗 𝑛 𝑃 − 𝑘=1 𝑛 𝑁 𝑌𝑘 𝑛 𝑁 = 𝑘=1 𝑛 𝑁 𝑌𝑘 0 𝑛 𝑁 − 𝑗=1 𝑛 𝑃 𝑌𝑗 1 𝑛 𝑃 of 𝐸 𝑌1 − 𝑌0 would be biased if some individuals occurred only among participants or non-participants Or more often among one of the two groups

147. X Y P

148. X Y P

149. Sir Austin Bradford Hill

150. Strength: How strong is the relationship? Consistency: How consistently is link found? Specificity: How specific is the setting or disease? Temporality: Does the cause precede the effect? Gradient: Does more cause lead to more effect? Analogy: Do similar “causes” have similar effect? Coherence: Are field and laboratory findings similar? Experiment: Was variation in the cause random? Plausibility: Does theory agree? Bradford Hill Criteria

161. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏

162. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.

163. We are presented with data in the form of a sample: Causality: Our Approach 𝒀𝒊, 𝑷𝒊, 𝑿𝒊 , 𝒊 = 𝟏, . . , 𝒏 Assumptions Model E(Y1-Y0), E(Y1-Y0|P=1), Etc.

164. Conclusion

165. Links: The manual: http://www.measureevaluation.org/resources/publications/ms- 14-87-en The webinar introducing the manual: http://www.measureevaluation.org/resources/webinars/metho ds-for-program-impact-evaluation My email: pmlance@email.unc.edu

166. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 and implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. www.measureevaluation.org

Fundamentals of Program Impact Evaluation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Fundamentals of Program Impact Evaluation

Similar to Fundamentals of Program Impact Evaluation (20)

More from MEASURE Evaluation

More from MEASURE Evaluation (20)

Recently uploaded

Recently uploaded (20)

Fundamentals of Program Impact Evaluation