Ukes process tracing presentation
•Transferir como PPTX, PDF•
2 gostaram•5,548 visualizações
Itad's Melanie Punton's presentation on Process Tracing from the UKES Conference, May 2015.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (13)
Semelhante a Ukes process tracing presentation
Semelhante a Ukes process tracing presentation (20)
Último
Último (20)
Ukes process tracing presentation
- 2. Punton & Welle (2015) Straws-in-the- wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and- smoking-guns-what-can-process-tracing-offer-to-impact- evaluation Stedman-Bryce (2013) Health for All: Towards Free Universal Health Care in Ghana http://policy- practice.oxfam.org.uk/publications/effectiveness- review-health-for-all-towards-free-universal-health- care-in-ghana-306376 2
- 3. Process Tracing: A qualitative method for assessing causal inference within a single case Sits in the ‘evaluator’s toolbox’ alongside • Theory-based evaluation • Contribution analysis • Realist evaluation 3
- 4. Why process tracing? 0 Useful where pathways of change are uncertain 0Tells you WHY change happened 0 Rigorous approach through ex-post design without a control group 0Time intensive 0 Requires the outcome to be known 0 Uncharted territory 4
- 5. A brief history of process- tracing 0 A method stemming from the social and political sciences. Major contributors: Alexander George, Andrew Bennett, David Collier 0 Originally used to explain historical events, e.g. the cause of the Cuban Missile Crisis (Allison and Zelikow 1999) 0 A ‘contested method’ 0 Beach and Pederson’s book (2013): detailed theory and step-by-step guidance 0 Limited examples of its application in evaluation NB: See Punton & Welle (2015) for references 5
- 6. Counterfactual framework Generative framework Experimental (RCTs) Quasi-experimental Process tracing Realist evaluation Intervention A Outcome B???? Opening up the black box 6
- 8. Causal mechanisms Part3Part2Part1 Intervention A Entity 1 Entity 2 Entity 3 Activity 1 Activity 2 Activity 3 Outcome B 8
- 9. The Universal Health Care Campaign in Ghana 0 Collaborative advocacy effort by civil society to promote universal free access to health care 0 Core funding from Oxfam until 2013 0 Lobbying, mobilising, media and research 0 Effectiveness review in 2012-13, using elements of process tracing Stedman-Bryce (2013) Effectiveness Review: Health for All: Towards Free Universal Health Care in Ghana http://policy- practice.oxfam.org.uk/publications/effectiveness-review- health-for-all-towards-free-universal-health-care-in- ghana-306376 9
- 10. A (simple) example of a process tracing mechanism Universal Health Care campaign Free universal health care Civil society Conduct coordinated advocacy activities for free universal health care Part 1 Public Part 2 Become aware of the limitations of current health care financing Public Part 3 Demand free universal health care from government actors Govern- ment actors Part 4 Increasingly support free universal health care, based on desire for public support Govern- ment Part 5 Amend policies and processes to move towards free universal health care Activity Entity 10
- 11. Differences between a mechanism and a theory of change… 0 No leaps in logic between the parts 0 Every part should be necessary 0 Every part should be observable and empirically measurable 11
- 12. Collecting evidence • Examples of coordinated advocacy linked to campaign (e.g. events, platforms, meetings) • Interview data suggesting that the campaign increased knowledge about and interest in healthcare reform among civil society actors • Number of advocacy events held before the campaign started, compared to number held during campaign Hypothesis: civil society conducts coordinated advocacy for free healthcare as a result of the Campaign 12
- 13. Collecting evidence • Examples of coordinated advocacy taking place before the campaigns • Interviews suggesting that the campaign made little difference to activities Alternative hypothesis: The advocacy campaign made little difference to the coordination or activities of CSOs 13
- 14. “We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – The Hound of the Baskervilles 14
- 15. Assessing evidence: four tests • Analogous to a criminal trial • The evidence for each hypothesis is weighed…. • …in order to increase the researcher’s confidence in the hypothesis. • Bayesian probability logic followed to assess the strength of each part of the causal chain. • Evidence examined using concepts of necessity and sufficiency. 15
- 16. Assessing evidence: four tests Straw in the wind test Evidence: hotel receipts, suggestive text messages - Neither necessary nor sufficient Hypothesis: John shot Mary Hoop test Evidence: John lacks an alibi - Necessary, but not sufficient Smoking gun test Evidence: John was found holding a gun over Mary’s body - Sufficient, but not necessary Double decisive test Evidence: A tamper- proof CCTV camera recorded the murder - Necessary and sufficient 16
- 17. Assessing evidence: four tests Certainty of evidence Uniquenessofevidence Low (evidence is insufficient for h ) High (evidence is sufficient for h ) High (evidenceis necessaryforh) Low (evidenceisnot necessaryforh) Hoop tests Double- decisive Straw inthe wind tests Smoking gun tests 17
- 18. Assessing evidence: four tests Straw in the wind test Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care Evidence: 1. Campaign members were given the opportunity to discuss and debate a report on the complex National Health Insurance Scheme, in order to increase their understanding on the topic 2. Before the report was published, there was little coordinated advocacy for free universal healthcare Increase the plausibility of the hypothesis but do not prove it or disprove alternative hypotheses. BUT together they provide stronger evidence than they do alone 18
- 19. Assessing evidence: four tests Hoop test Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care Evidence: Until the campaign started, there was little notable coordinated advocacy on this issue. Not enough on its own to prove the hypothesis, BUT evidence must pass the test in order to keep the hypothesis under consideration 19
- 20. Smoking gun test Hypothesis: The National Health Insurance Authority in Ghana revised methodology for calculating membership because of pressure created by the campaign Evidence: Ghana delegation at international meeting stated that the campaign’s report ‘prompted us to revise our figures’ (despite initially publically dismissing the report) Evidence is sufficient to give high confidence that the hypothesis is true. BUT this type of evidence is hard to come by. Assessing evidence: four tests 20
- 21. The end goal… 0 Researcher can express level of confidence in each part of the mechanism 0 Mechanism is only as strong as its weakest part 0 If the researcher is confident in the whole mechanism… 0 …the researcher can express confidence that the mechanism holds in that particular case: that A caused B, and why. 21
- 22. Why process tracing? 0 Useful where pathways of change are uncertain 0 Rigorous approach through ex-post design without a control group 0Tells you WHY change happened 0Useful in a mixed methods design, or in combination with e.g. Contribution Analysis? 0 Time intensive 0 Requires the outcome to be known 0Uncharted territory 0 Risk of one weak link causing the whole mechanism to be rejected 22
- 23. Questions? melanie.punton@itad.com Punton & Welle (2015) Straws-in-the- wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and- smoking-guns-what-can-process-tracing-offer-to-impact- evaluation 23
Notas do Editor
- PT is a qualitative method in that it takes a qualitative approach to assessing inference – it collects, interprets and weighs up evidence and gives a qualitative rather than quantitative conclusion. BUT both qualitative and quant data can be used to reach these conclusions. PT is a case based approach. A ‘case’ might be an intervention, a project, a country, an organisation, a process or even an individual. It’s comparable to other theory based approaches to evaluation – including contribution analysis and realist evaluation
- So how does process tracing work? It works by opening up the ‘black box’ between an intervention, and an outcome. What is it about an intervention that causes it to lead to a specific outcome? This is the fundamental difference between process tracing and experimental and quasi-experimental evaluation designs: Methods based on counterfactual frameworks (such as RCTs) use control group to isolate the effect of an intervention on the treatment group. Causality inferred through comparison the outcomes of two or more (identical or very similar) cases, in which one received the intervention and the other did not. The black box of the intervention remains unopened In a generative causal framework, causality is inferred through explaining why an intervention lead to an outcome: generating evidence of what happens in the black box and how that connects the intervention to the outcome. This is the same framework underpinning realist evaluation. This is why process tracing doesn’t need a control group in order to rigorously attribute change.
- So how does process tracing open the ‘black box’? Inside the black box is the ‘causal mechanism.’ The concept of ‘causal mechanisms’ is central to process tracing. A causal mechanism explains what it is about intervention A that leads to outcome B - the ‘causal force or power’ that gives rise to the outcome. The concept of the causal mechanism is central to the generative causal framework that underpins process tracing. This concept also exists in other fields – for example in realist evaluation. The concept has quite a specific meaning in Beach and Pederson’s version of process tracing….
- A causal mechanism in process tracing looks like this. It elaborates all the steps between A (the intervention), and B (the outcome), opening up the black box and breaking down what is inside into the smallest feasible number of PARTS. Each part consists of ENTITIES conducting ACTIVITIEs. Each part gives rise to the next part in the mechanism.
- What does this actually look like in practice? I’ll now work through an example of what a causal mechanism might look like in practice, using the case study of the Universal Health Care Campaign in Ghana.
- This is an example of a causal mechanism based on the Universal Health Care Campaign. A caveat – it’s a simplified example adapted from the Ghana case, designed to show what a mechanism might look like. In actual fact Stedman Bryce’s paper does not consider the full mechanism, because the campaign was still ongoing – so it only tested part of the mechanism. The whole mechanism is a THEORY about how the intervention resulted in change. Constructing a mechanism has obvious parallels with developing a theory of change.
- Evidence can be qualitative and quantitative. Usual principles apply – reliability, validity, potential biases, the need for triangulation.
- For each part of the mechanism, the researcher should also collect evidence to test alternative hypotheses – what else might have happened, at each point in the chain?
- Evidence is necessary to prove a hypothesis if it must exist in order for us to keep the hypothesis under consideration Evidence is sufficient to prove a hypothesis if it is enough on its own to prove the hypothesis
- The researcher constructs these tests based on the evidence they expect to find. Following Bayesian logic, the researcher gives greater weight to evidence that she expects to be less probable, based on previous knowledge of the phenomenon: The more unlikely it is that you will find a piece of evidence given a specific hypothesis, the more the evidence will increase the probability of the hypothesis being true, if the evidence does in fact exist. More surprising evidence results in larger increases in our confidence in a hypothesis than less surprising evidence. Constructing tests is highly analytical and contextual – and there are risks of subjectivity. However, the tests provide a transparent and systematic approach to weighing up qualitative evidence.
- Three examples from the Universal Health Care Campaign to show how the tests can be applied in practice.
- If the evidence fails to support any one part in the mechanism, the whole mechanism must be rejected. There is insufficient evidence to show that A lead to B in the way specified in the mechanism (although A may have led to B in some other way).