This webinar by Peter Lance considered impact evaluation estimation methods based on an identification strategy that assumes we can observe all factors that influence both program participation and the outcome of interest. It was the third webinar in a series of discussions on the popular MEASURE Evaluation manual, How Do We Know If a Program Made a Difference? A Guide to Statistical Methods for Program Impact Evaluation.
1. Peter M. Lance, PhD
MEASURE Evaluation
University of North Carolina at
Chapel Hill
September 13 and 15, 2016
Selection on Observables
2. Global, five-year, $180M cooperative agreement
Strategic objective:
To strengthen health information systems β the
capacity to gather, interpret, and use data β so
countries can make better decisions and sustain good
health outcomes over time.
Project overview
3. Improved country capacity to manage health
information systems, resources, and staff
Strengthened collection, analysis, and use of
routine health data
Methods, tools, and approaches improved and
applied to address health information challenges
and gaps
Increased capacity for rigorous evaluation
Phase IV Results Framework
35. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average Y
across sample
of participants
β
Average Y
across sample
of nonβparticipants
36. πΈ π1
β π0
|π = π₯β
= πΈ π1
|π = π₯β
β πΈ π0
|π = π₯β
Average Y
across sample
of participants
for whom
π = π₯β
β
Average Y
across sample
of participants
of nonβparticipants
π = π₯β
47. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
π€π β ππ
π π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle class
ππ=4.4 if individual i is rich
48. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
ππ
π π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle class
ππ=4.4 if individual i is rich
49. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
π€π β ππ
π=1
π π
π€π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle class
ππ=4.4 if individual i is rich
50. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
ππ β ππ
π=1
π π
π€π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle class
ππ=4.4 if individual i is rich
52. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
π€π β ππ
π=1
π π
π€π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle
class
ππ=4.5 if individual i is rich
53. πΈ π1
β π0
= πΈ π1
β πΈ π0
. 4 β
Average Y
across sample
of participants
who are poor
+ .5 β
Average Y
across sample
of nonβparticipants
who are middle class
+.1 β
Average Y
across sample
of participants
who are rich
54. πΈ π1
β π0
= πΈ π1
β πΈ π0
Average of Y
across a sample
of participants
=
π=1
π π
π€π β ππ
π=1
π π
π€π
ππ=.63 if individual i is poor
ππ=1.47 if individual i is middle
class
ππ=4.5 if individual i is rich
82. An expected value for a random variable is the
average value from a large number of repetitions
of the experiment that random variable represents
An expected value is the true average of a random
variable across a population
Expected value
83. An expected value for a random variable is the
average value from a large number of repetitions
of the experiment that random variable represents
An expected value is the true average of a random
variable across a population
Expected value
84. An expected value is the true average of a random
variable across a population
πΈ π = some true value
Expected value
203. πΈ π1 = π½1 + π½2 β πΎ1
The Actual
Causal Effect of
P on y
204. πΈ π1 = π½1 + π½2 β πΎ1
The actual
causal effect of
P on y
The actual causal effect of
the omitted variable X on
Y
205. πΈ π1 = π½1 + π½2 β πΎ1
The actual
causal effect of
P on y
ThβEffectβ of P on x:
π₯π= πΎ0 + πΎ1 β ππ + ππ
The actual causal effect of
the omitted variable X on
Y
206. True model:
π = π½0 + π½1 β π + π½2 β π₯ + π
We actually attempt to estimate:
π = π0 + π1 β π + π
Error term now contains:
π½2 β π₯
207. True model:
π = π½0 + π½1 β π + π½2 β π₯ + π
We actually attempt to estimate:
π = π0 + π1 β π + π
Error term now contains:
π½2 β π₯
208. True model:
π = π½0 + π½1 β π + π½2 β π₯ + π
We actually attempt to estimate:
π = π0 + π1 β π + π
Error term now contains:
π½2 β π₯
209. True model:
π = π½0 + π½1 β π + π½2 β π₯ + π
We actually attempt to estimate:
π = π0 + π1 β π + π
Error term now contains:
π½2 β π₯
211. πΈ π1 = π½1 + π½2 β πΎ1
The actual
causal effect of
P on y
ThβEffectβ of P on x:
π₯π= πΎ0 + πΎ1 β ππ + ππ
The actual causal effect of
the omitted variable X on
Y
212. πΈ π1 = π½1 + π½2 β πΎ1
The actual
causal effect of
P on y
ThβEffectβ of P on x:
π₯π= πΎ0 + πΎ1 β ππ + ππ
The actual causal effect of
the omitted variable X on
Y
224. πΈ π1 = π½1 + π½2 β πΎ11 + π½3 β πΎ21 + π½4 β πΎ31
The actual
causal effect of
P on y
225. πΈ π1 = π½1 + π½2 β πΎ11 + π½3 β πΎ21 + π½4 β πΎ31
The actual
causal effect of
P on y
ThβEffectβ of P on x1:
π₯1π= πΎ10 + πΎ11 β ππ + π1π
The
actual
causal
effect of
the
omitted
variable
X1 on Y
226. πΈ π1 = π½1 + π½2 β πΎ11 + π½3 β πΎ21 + π½4 β πΎ31
The actual
causal effect of
P on y
βEffectβ of P on x1:
π₯1π= πΎ10 + πΎ11 β ππ + π1π
The
actual
causal
effect of
the
omitted
variable
X1 on Y
The actual causal effect
of the omitted variable
X2 on Y
ThβEffectβ of P on x2:
π₯2π= πΎ20 + πΎ21 β ππ + π2π
227. πΈ π1 = π½1 + π½2 β πΎ11 + π½3 β πΎ21 + π½4 β πΎ31
The actual
causal effect of
P on y
βEffectβ of P on x1:
π₯1π= πΎ10 + πΎ11 β ππ + π1π
The
actual
causal
effect of
the
omitted
variable
X1 on Y
The actual causal effect
of the omitted variable
X2 on Y
βEffectβ of P on x2:
π₯2π= πΎ20 + πΎ21 β ππ + π2π
ThβEffectβ of P on x1:
π₯3π= πΎ30 + πΎ31 β ππ + π3π
The actual causal effect of
the omitted variable X3 on Y
228. πΈ π1 = π½1 + π½2 β πΎ11 + π½3 β πΎ21 + π½4 β πΎ31
The actual
causal effect of
P on y
βEffectβ of P on x1:
π₯1π= πΎ10 + πΎ11 β ππ + π1π
The
actual
causal
effect of
the
omitted
variable
X1 on Y
The actual causal effect
of the omitted variable
X2 on Y
βEffectβ of P on x2:
π₯2π= πΎ20 + πΎ21 β ππ + π2π
ThβEffectβ of P on x3:
π₯3π= πΎ30 + πΎ31 β ππ + π3π
The actual causal effect of
the omitted variable X3 on Y
303. The basics of the method
1. Pool your sample of participants and non-participants
and define various characteristics π.
2. Estimate the probability of program participation
conditional on
Pr π = 1|π
3. Compute the propensity score using the fitted binary
participation model.
ππ ππ = 1|ππ
4. Find a counterfactual outcome for each individual by
identifying some individual who did experience the
counterfactual conditions and had a similar propensity score
326. MEASURE Evaluation is funded by the U.S. Agency
for International Development (USAID) under terms
of Cooperative Agreement AID-OAA-L-14-00004 and
implemented by the Carolina Population Center, University
of North Carolina at Chapel Hill in partnership with ICF
International, John Snow, Inc., Management Sciences for
Health, Palladium Group, and Tulane University. The views
expressed in this presentation do not necessarily reflect
the views of USAID or the United States government.
www.measureevaluation.org