This presentation was given at the International Family Planning conference in Kampala, Uganda in November 2009 by IRH Georgetown and the Extending Service Delivery (ESD) Project.

1. 1 Community and Social Change in ASRH Programs: Strategies for Measuring Change Uganda International Family Planning Conference, Kampala, November 2009 Facilitated by Cate Lane, Pathfinder International/ESD Project, Susan Igras, Institute for Reproductive Health/ Georgetown University

2. 2 Workshop Objectives By the end of the session, participants will be familiar with: Language (and concepts) to articulate more precisely the role of community involvement in improving ASRH Key issues about monitoring and evaluation of youth reproductive health and HIV prevention programs, and Issues relating specifically to evaluating community involvement programs for youth.

3. 3 Workshop Agenda Block 1 Introductions (15 min) Community involvement and ASRH and Monitoring and evaluation issues in ASRH and CI programs (30 min.) Block 2 Application and reflection 1: Case studies from Haiti (Robin Anthony Kouyate, ACCESS-FP) and Kenya (Pam Ondusu, Pathfinder International)(60 min.) Block 3 Application and reflection 2: Case study from northern Uganda (AbejaApunyo, Pathfinder International) and small group work (30 min.) Block 4 Final discussion and summary (30 min.)

4. 4 Importance of Community Involvement in ASRH Programs IAWG on Community Involvement in RH and HIV Prevention Programs

5. 5 Why is it Important to Involve Communities in ASRH Programs? Within your ASRH programs, do you set goals and objectives to define and measure success in involving communities (whether individuals, groups, or institutions)?

6. 6 Why is a Framework Needed? Challenges in articulating added value of community-driven interventions Good development practice or a unique program intervention? Process or outcome? Current definitions of success of ASRH programs often limited to service uptake and knowledge gains

7. 7 Why is a Framework Needed? Lack of written goals and objectives against which to measure successful community involvement in ASRH/RH programs Need more research and program evaluation to provide evidence of unique value of community participation Need more systematic knowledge of better program practices to guide resource allocations

8. 8 A First Step Develop a conceptual framework To guide program planners in thinking about and ‘objectifying’ realm of outcomes of community processes in ASRH programs To help evaluators in thinking about monitoring and evaluation of community involvement in ASRH

9. 9 IAWG Members Advocates for Youth CARE International (US) * CEDPA PathFinder International Save the Children Federation (US) UNFPA * Several independent consultants * Co-hosts of the IAWG

10. 10 Nepal Operations Research Study – Is Participation the Answer? Tested effectiveness of participatory approaches to define and address RH concerns of youth. CONTROL SITES Standard intervention set No participant input Didactic training approach Program and training limited Focus on RH info & services STUDY SITES Community-determined interventions Participant input in design and training Interactive program Participatory training approach Holistic programming

11. 11 Results from NepalSource: Mathur et al. YRH in Nepal: Is Paraticiaption the Answer, NY: EngenderHealth and ICRW, 2004 Participatory approaches yielded more positive results than traditional RH interventions - Improvement in KAP in both sites - no significant difference in KAP between sites - Substantial difference in broad contextual factors that influence YRH – norms, empowerment of youth and adults, development of stronger community institutions. Participatory approaches enable incremental progress that would not be achieved with a more ‘top-down’ approach.

13. Institutional structures,

15. 13 Role of Catalysts in Engaging Communities Provide spark to begin engaging community publicly around sensitive ASRH issues Internal sparks, eg, a women’s rights group External sparks, eg, a program supported by an NGO

16. 14 Community Involvement Processes Process by which community engages and acts collectively to address a condition or issue affecting the community. Basis is dialogue and public debate (communication for social change precept) Principles of involvement by external catalysts or ASRH programs: Taking a ‘back seat’ / build organizational / social capacity Supporting ASRH/RH activities defined by community Acknowledge and try to minimize power & resource differentials

17. 15 Results Across a Spectrum Structural Change • Well-functioning services • Effective inter-organizational networks • Inclusiveness of social networks • Org’al commitment to ASRH • Access to resources

18. 16 Looking Closer at Individual Change: Adolescents and Adults Individual Change ADULT GATEKEEPERS SKILLS: Improved parent-child communication on SRH-related issues IDEATION: Level of awareness of pregnancy risk of adolescents INTENTION: Intention to grant adolescents rights to SRH information and services BEHAVIOR: Improved family support for adolescent seeking SRH services

19. 17 Illustrative Outcome Indicators of ASRH Programs Structural Change ASSET BASE: Existence of laws and policies supportive of ASRH rights SYSTEMS’ EFFECTIVE FUNCTIONING: Health service have institutionalized ASRH-friendly services INTER-ORGANIZATIONAL CONNECTIONS: Existence of inter-organization collaboration on specific ASRH issues RESOURCE ACCESS: Increased resources for ASRH-related activities in an organizations’ health program

20. 18 Looking Closer at Social Changes Important to foster same level of interest, attention, technical support and resources to social change results. Not as easy as measuring individual or structural changes, But critically important to achieving broad-based and sustained individual changes.

21. 19 Measurement of Results: Illustrative Outcome Indicators of ASRH Programs

22. 20 Monitoring and Evaluation of ASRH/CI Programs

24. Institutional structures,

26. 22 Purpose of Monitoring and Evaluation (M&E): Identify factors (individual, community, programmatic) that influence health outcomes Strengthen the design and implementation of programs Improve use of existing resources, thereby increasing cost-effectiveness Ultimately, to better the human condition

27. 23 Programs Vary Widely in Scope and Content, but Have Similar Structure for M&E: Objectives (implicit or explicit) Intended outcomes Target population Mechanisms to deliver services A “theory of change” Implicit, or Explicit – a conceptual framework that provides rationale for program existence Very important for M&E

29. Acting on issues

30. AdvocatingIntermediate: Social changes Structural changes Individual changes Final: Infection rate Mortality Unintended preg.

31. 25 Quantitative & Qualitative Methodologies Quantitative data describe WHAT people do/think Useful for tracking trends and highlighting differences Qualitative data explore WHY people behave and think as they do Useful for understanding the context in which the trends/differences occur and interpreting quantitative data

32. 26 Why CI Programs Are Often Not Evaluated Programs are multi-sectorial Programs are targeting communities and not necessarily individuals Indicators difficult to measure Process – community engagement; collective action – no good indicators of quality/success Output – numbers served, with what activities; possibility for double/triple counting Outcome - Social changes, structural changes, individual changes – but these are affected by so many other things

33. 27 Evaluation Challenges of Most Youth Programs Including CI Programs Limited evidence of effectiveness Hard to separate contribution of multiple strategies Non-standardized definitions and indicators Climate of urgency around results but behavior change is long-term

34. 28 Challenges Evaluating Youth Programs Generally Measuring influences on behaviors that didn’t occur is difficult Measuring behaviors at a variety of developmental levels can be problematic Showing links with outcomes is not obvious Some changes may not be measurable for a long time; others are hard to measure Attributing changes to a particular strategy is difficult

35. 29 Other Challenges - Generally Programs and measurement often equate age with sexual experience Survey administration Interviewer administered vs. self-administered Literacy matters Location might matter Consent/Assent issues May depend on location of study (clinic, school, household)

36. 30 Additional Challenges Evaluating Youth Programs that are CI Participatory approaches (evaluation) may end up being part of the intervention Lack of tested/accepted indicators By nature, the programs are complicated with numerous levels of programming Consider strategies to measure program ‘participation’ and ‘reach’

37. 31 Don’t Give Up – Evaluation Data/Findings are Crucial Development of standard framework (IAWG on CI) and proposed indicators Need to validate indicators; provide depth on how to measure Consider a variety of data sources including KAP surveys (at community level) and qualitative data among community members (youth, adults)

38. 32 Don’t Give Up – Evaluation Data/Findings are Crucial, cont. Choose an evaluation design that is feasible relative to evaluation questions, program resources, and timeframe Consider complicated ethical issues because of focus on youth and communities Parental consent; youth assent Small communities – risk of deductive disclosure Get input from evaluation experts and M&E resources!

39. 33 Measuring Social Change in ASRH programs – Where are we?

41. Little work yet on social change indicators relating to ASRH specificallyMost indicators are qualitative Some work on quantitative indicators eg, individual psychological self-efficacy indices adapted for collective self-efficacy, Likert scales on social change, eg, perceptions of leadership and collective ownership of an issue

42. 35 Social Change Indicators & Data Sources Data sources Information collection is KI interviews, FGDs, structured observations (limited use of surveys) Need to define ‘target populations’ in which to assess changes – individuals (youth / adult), social groups/networks, institutions Preparing to expect the unexpected Use of qualitative methodologies such as Most Significant Change to capture unexpected change

43. 36 Community Evaluation of their Involvement Processes and Outcomes Important evaluation issue if respect community empowerment approach to addressing ASRH issues What happens inside the ‘conceptual box’ of community involvement? Use of qualitative methodologies such as Most Significant Change to capture changes observed by communities Use of participatory, qualitative methodologies to facilitate community reflections Guidelines from Communication for Social Change Consortium

44. 37 Useful resources: “Community Pathways to Improved Adolescent Sexual and Reproductive Health: A Conceptual Framework and Suggested Outcome Indicators” http://www.unfpa.org/publications/detail.cfm?ID=377&filterListType=5 “Communities Measure Change” (sourcebook on facilitating community measurement of social change processes) http://www.communicationforsocialchange.org/publications-resources.php?id=281 “Communication for Social Change: An integrated Model for Measuring the Process and its Outcomes” (presentation of the communication for social change model with suggested output and outcome indicators) http://www.communicationforsocialchange.org/publications-resources.php?id=107 “Using Narrative Methods to Link Program Evaluation and Organizational Development” (Evaluation Exchange discussion paper on application of Most Significant Change methodology) http://www.gse.harvard.edu/hfrp/eval/issue24/pp3.html

45. Case study questions: What community involvement approaches were used? How could the project strengthen its CI approaches? How could you measure changes in CI that lead to improved ASRH? (Think about indicators and data sources) 38