1. Assessing M&E Systems for Data Quality USAID Mini University George Washington University Washington, DC, October 5th, 2007

3. Data quality and the Program Cycle Data Quality Results Reporting Target Setting Improved Program & Resource Management

4. Data Quality Services: the REAL world In the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable. Data: INFORMATION SYSTEM An information system represents these activities by collecting the results that were produced and mapping them to a recording system. Linking your services and your data … but how well does it all work?

5. Aggregated Data from PEPFAR Countries Source: 3 rd Annual PEPFAR report Where did these OVC numbers come from? Do you think they are good numbers? Why or why not? Make a flow chart of the data from sites to PEPFAR

6. Data Quality REAL WORLD In the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable. INFORMATION SYSTEM An information system represents these activities by collecting the results that were produced and mapping them to a recording system. Data Quality: How well the information system represents the real world Data Quality Real World Information System 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality

8. Dimensions of Data Quality The data are protected from deliberate bias or manipulation for political or personal reasons. Integrity Clients are assured that their data will be maintained according to national and/or international standards for data. Confidentiality Data are up-to-date (current), and information is available on time. Timeliness The data have sufficient detail (e.g. collected by age, sex, etc.) Precision Completely inclusive: an information system represents the complete list of eligible names and not a fraction of the list. Completeness The data are measured and collected consistently (the same way with the same data collection instruments) over time. Reliability Valid data are considered accurate: They measure what they are intended to measure. Accuracy/ Validity

9. Information Flow Results Indicators Data sources Data collection/reporting systems

11. Program/project M&E Plan and List of Indicators What is the role of the central M&E Unit in the M&E system and data quality? What is the role of the intermediate aggregation level? What is the role of sites? Levels of the M&E System Data Requirements 1- Central/ national M&E Unit 2- Intermediate aggregation levels (e.g. district, region, state, sub-partner) 3- Service sites (health facility-based or community-based) Data Reporting Data Reporting

15. What is the Link between M&E Systems and Data Quality? 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Dimensions of Data Quality 1- Central/ national M&E Unit 2- Intermediate aggregation levels (e.g. district, region, state, sub-partner) 3- Service sites (health facility-based or community-based) Data Reporting Data Reporting Levels of the M&E System

17. Multiagency Tools Under draft

18. SYSTEMS APPROACH INDICATOR APPROACH AUDITING APPROACH (also for capacity building) Three Multi-agency complementary DQ Tools 1- M&E Systems Strengthening Tool What data management systems should be in place to ensure data-quality? 3- Data Quality Assurance Tool for Program Level Indicators What are the Data Quality challenges in collecting specific indicator data (e.g., for ARV, for People Trained, etc.) 2- Data Quality Assessment (DQA) Tool Are appropriate data management systems in place? Is reported data accurate and valid?

19. 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Functional Areas of an M&E System that Affect Data Quality Dimensions of Data Quality Data quality mechanisms and controls VII Data management processes VI Data collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality

22. DQA Protocol 1: Functional Areas of an M&E System that Affect Data Quality (from DQA Systems Assessment Protocol) Are there operational indicator definitions meeting relevant standards and are they systematically followed by all service points? 4 Indicator Definitions IV Has the Program/Project clearly documented (in writing) what is reported to who, and how and when reporting is required? 3 Data Reporting Requirements III Have the majority of key M&E and data-management staff received the required training? 2 Training II Are key M&E and data-management staff identified with clearly assigned responsibilities? 1 M&E Capabilities, Roles and Responsibilities I 11 Summary Questions 8 Functional Areas

23. DQA Protocol 1: Functional Areas of an M&E System that Affect Data Quality (from DQA Systems Assessment Protocol) Does the data collection and reporting system of the Program/Project link to the National Reporting System? 11 Links with National Reporting System VIII Are there clearly defined and followed procedures to periodically verify source data? 10 Are there clearly defined and followed procedures to identify and reconcile discrepancies in reports? 9 Are data quality challenges identified and are mechanisms in place for addressing them? 8 Data Quality Mechanisms and Controls VII Does clear documentation of collection, aggregation and manipulation steps exist? 7 Data Management Processes VI Are source documents kept and made available in accordance with a written policy? 6 Are there standard data-collection and reporting forms that are systematically used? 5 Data-collection and Reporting Forms and Tools V

24. Summary Question 1: Are key M&E and data-management staff identified with clearly assigned responsibilities?  The responsibility for recording the delivery of services on source documents is clearly assigned to the relevant staff (i.e., it is in their job description). 6   There are designated staff responsible for the review of aggregated numbers prior to submission to the next level (e.g. to the central M&E Unit). 5   There are designated staff responsible for reviewing the quality of data submitted by sub-reporting levels (e.g. regions, districts, service points). 4  The Program Manager(s) review(s) the aggregated numbers prior to the submission/release of reports from the M&E Unit. 3  There are staff dedicated to M&E and data management systems. 2  There is a documented organizational structure/chart that clearly identifies positions that have data management responsibilities. 1 I - M&E Capacities, Roles and Responsibilities Service Points Aggre-gation Levels M&E Unit Reporting System Level Component of the M&E System LIST OF ALL QUESTIONS List of Questions Related to the Data Management and Reporting System

25. DQA: Summary of M&E System Functional Areas Interpreting the findings: The larger the score, the stronger the component

26. Data verification: DQA Protocol 2 Perform “spot checks” to verify the actual delivery of services or commodities to the target populations. 5. Spot checks Perform “cross-checks” of the verified report totals with other data-sources (eg. inventory records, laboratory reports, etc.). 4. Cross-checks Trace and verify reported numbers: (1) Recount the reported numbers from available source documents; (2) Compare the verified numbers to the site reported number; (3) Identify reasons for any differences. 3. Trace and Verification Review availability and completeness of all indicator source documents for the selected reporting period. 2. Documentation Review Describe the connection between the delivery of services/commodities and the completion of the source document that records that service delivery. 1. Description Description Verification SERVICE DELIVERY POINT - 5 TYPES OF DATA VERIFICATIONS

27. DQ Dimensions, Levels of the M&E System and Functional Areas Dimensions of Data Quality Levels of the M&E System Functional Areas of an M&E System Needed to Ensure Quality Data quality mechanisms and controls VII Data management processes VI Data-collection and Reporting Forms and Tools V Links with the national reporting system VIII Indicator Definitions IV Data Reporting Requirements III Training II M&E Capabilities, Roles and Responsibilities I Functional Areas of an M&E System Needed to Ensure Quality 1. Accuracy 2. Reliability 3. Completeness 4. Precision 5. Timeliness 6. Integrity 7. Confidentiality Data Reporting 1- Central/ national M&E Unit 2- Intermediate aggregation levels 3- Service sites (health facility-based or community-based)

30. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GPO-A-00-03-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina in partnership with Constella Futures, John Snow, Inc., ORC Macro, and Tulane University. Visit us online at http://www.cpc.unc.edu/measure.