Organizing and Managing Program Evaluation
•Download as PPT, PDF•
1 like•669 views
![Presentation Outline ,[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to Organizing and Managing Program Evaluation
Similar to Organizing and Managing Program Evaluation (20)
More from Urban Community Research Center for Asia
More from Urban Community Research Center for Asia (14)
Organizing and Managing Program Evaluation
- 1. Organizing and Managing Program Evaluation
- 4. Budget and Program Overview: Fiscal Year 2000 Program Block Budget ($ billion) No. of Major Programs Percent of budget total Community development 7.8 12 24 Housing Assistance 23.3 11 72 Housing finance 0.8 25-30 2.5 Fair housing 0.4 2 1.4 total 32.2 60-65 100
- 7. 2. Organization of the Evaluation Function
- 10. 3. Defining Evaluation Topics
- 12. Federal Government Budget Cycle --Illustrated for the FY2001 Budget-- Date Action August 1998 OMB sends “management letter to agency January 1999 Minister issues instructions for the budget process, including guidance on priorities February-March Offices work on submission April – July Internal review process July-August Finalization September Submission to OMB October-November Formal hearings with OMB mid-November OMB “pass-back” mid-Nov to mid-December Appeals; final decisions mid-January 2000 Submission to Congress May-August / October 1 Congress acts ; new fiscal year
- 13. 4. Managing evaluation projects
- 19. Staff loading chart for analysis and results write-up for impact analysis using econometric methods (days) Task Senior staff Mid-level Junior staff Data analysis --specify models 12 8 -- --estimation 3 5 15 --sensitivity tests 4 3 9 Results write-up --outline development 3 1 -- --writing 10 15 2 --table preparation -- -- 8 TOTAL 32 32 34
- 30. Subject -- Using housing vouchers to help poor families leave inner-city neighborhoods for middle class areas -- Evidence from non rigorous studies that such moves had significant positive impacts on the families who moved
- 42. Subject-2 -- These loans are to businesses in CDBG target neighborhoods or primarily serving poor households --Evaluation Issue: how successful are these programs in generating economic development?
Editor's Notes
- Most of you have been to presentations about the technical aspects of conducting program evaluations. Such presentations focus on the methods for carrying out evaluations—sample sizes, control groups, statistical analysis receive prominent attention. This presentation is designed for those government managers who will be responsible for organizing and managing evaluations within their ministries. The material presented may also be of interest to others who want to understand broadly how a government evaluation office should operate.
- To make the later parts of the discussion concrete it is necessary to establish the context in which the evaluations are being defined and managed. I am using HUD because I know it well. So we open with a brief overview of the Ministry’s responsibilities and organization. Within this we focus on the location of the evaluation function.
- GNMA – credit enhancement to mortgage-backed bond sales by banks; government guarantee that investors will be paid. Community development: CDBG
- Define the fiscal year All figures are approximate. Some smoothing for year-to-year variation. Excludes tax benefits which are substantially larger ~$50 billion this year Total spending is about $120 per person. Total number of all programs : ~260 Staff: 1980 12,500 2000 8,000 and falling Outsourcing
- Flat organization chart. 12 A/Ms report to Minister. All A/Ms appointed by the president. D/M responsible for administration. IG – both fraud and more and more examination of inefficiency. Unclear lines sometimes. Does not report directly to the minister. President and Congress. Appointed by president and confirmed by the Congress. PDE – next slide
- Special position. In some sense the “right hand” of the minister. Centralization of the evaluation function is very valuable. Quality control, trained staff, objectivity. In conducting research and evaluation the office makes substantial use of outside contractors. PDE receives a separate appropriation (line item in ministry budget) for this.
- Program A/Ms – primary responsibility for program monitoring CFO – primary responsibility for the new performance measurement requirements under GPRA. Annual “Performance Accountability Report” PDE—consults all of the offices on these topics. Office of Research and Evaluation – next slide
- Office asks each A/M office for their ideas for evaluations needed in the next budget year. This is a formal request. The response from the program office forms the basis for an ongoing discussion on what will be included in the agenda. The reality is that the program offices often do not want programs evaluated—no good news from an evaluation. So PDE has the right to propose programs to be evaluated as part of the budget process—which is discussed more in a few minutes.
- Congressional mandates Explain authorizing and appropriations committees. Requests usually come from the authorizing committees. They take two forms: Request for a demonstration project with an accompanying evaluation (often result of lobbying by industry, e.g., homeownership counseling) Concern about the way the program is delivering services; these requests may be supported by advocacy NGOs who monitor the program or be stimulated by less systematic analysis (GAO). These requests are written into the law with a due date for the evaluation.
- Areas in yellow are those points at which requests for program evaluations can be made: OMB asks both for “in house” assessments and for large scale evaluations of programs about which it has concerns. These are strong requests not orders; usually a lot of discussion, especially about priorities. OMB staff can also veto evaluations proposed by the ministry. A/Ms – sometimes made requests; A/M PDE always has candidates. Review process – sometimes the need for an evaluation becomes evident and minister puts it on the list. Final agenda —depends on the available budget, the minister’s priorities, OMB’s requirements and Congressional mandated studies.
- Do your homework; map the program. Understand how each element works. Meetings with the program office will likely be essential. They will appreciate your effort to learn. If you meet with the program people about the evaluation before you really know the program, you are likely not to be taken seriously. Define “primary intended user” No surprises for your client
- Consider the case of a housing allowance program—define. Question that cannot be answered: on grounds of equal treatment of the population is targeting subsidies the correct thing to do? Example of Question to be addressed: what percentage of eligible households are receiving housing allowances Indicator: percent of eligible population receiving benefits at a particular point in time. Data sources Participants – program office Eligible households: secondary data or household survey Check the data sources on the program with the program office. Staff there should know which data series are not reliable.
- Not going to talk about alternative methods. Really no sufficient time for this.
- Labor: Spend a lot of energy on this estimate, as it is usually two-thirds or more of the total budget (including related overheads). More on see next slide Travel: important for studies involving a lot of field work or visits to other countries, etc. Data collection: after some experience, managers will develop their own estimates but can get rules-of-thumb from survey firms, e.g., cost per completed interview (in person, telephone, mail) of national random sample of 1,200 households with interviews lasting 30 minutes. Overheads: will learn quickly from reading proposals. At the beginning when you do not know how much it will cost, specify the resources available in man-months; can also set the sample size in the RFP and then change it to a more realistic level, if needed, after some initial analysis.
- Obviously, this is a small segment of the full staff loading chart. Once you have the total days, multiply the staff days in each category by the estimated per day salary level. Note that even though the total LOE per staff is about the same, the elapsed time would be quite a bit longer because the work has to be done sequentially.
- Define the difference between the two. Analysis plan – is an outline of the work to be done by an internal group. It should contain a clear statement on the methods to be used, the data sources, expected outcome and schedule—and the level of effort expected from various staff members. (Need to have the later well-understood at the outset.) The RFP will be less developed with respect to the methodology, giving some range for competition. On the other hand, it will contain all of the details about the agency’s contracting requirements.
- Be clear about… --sample sizes, etc. Important to give specific guidance so that you receive comparable bids. A first task in the contract can be for the contractor to review the sampling plans and make recommendations. Not too much on methodology . This is an important area of the competition, i.e., contractors can distinguish themselves with strong presentation. With no level of effort , will get estimates that vary widely because the contractors are guessing. UI’s experience bidding on the KZ financial sector reform project sponsored by USAID. Dissemination : the contractor should have the right to publish the results. But ministry can protect itself with a 60 or 90 period after the final report is accepted before the results can be shared.
- Participation by the client’s staff (program office) is important because: (1) they have more detailed knowledge of the program; (2) want them to endorse selection of the contractor. “Buy in”
- Understanding the issues : both program knowledge and knowledge of other research and evaluation on the topic Methodology : selection of appropriate evaluation approach, e.g., who to interview, how to interview (in-person, phone, mail; focus group); what data to analyze (file information, newly collected, etc); analytic methods. Management plan : responsibilities clearly allocated; who is control; who responsible for each major task; Too many persons or organizations involved—leads to loss of control and possibly duplication Level of resources assigned to different tasks is reasonable? Proposed staff : experience with doing similar tasks in the past, both analytic and managerial Organization’s past experience : reliable? PPRs should be requested and some checked by panel members Price
- AT THE BEGINNING: importance of making these deliverables and very carefully reviewing them. Time for outside consultants if you need them. Milestone reports : require reports at each major phase of the project so that you know if it is beginning to drift off of its intended direction. Draft final report : this is the time to request changes. If you ask later, the contractor can reasonably ask for additional funding.
- Contractors want future business and are very likely to respond to clear statements of dissatisfaction. Very important to be precise about problems. Sometimes the project manager will hide problems from senior officials at the company. That is why one should contact senior management at the firm if the project manager remains unresponsive to the requests for improvements.
- Evaluations will inevitably discover areas in which performance can be improved. That is why they are done. And appropriately, addressing these problems receives the majority of attention. BUT, it is important for inter-office relations, as well as making an objectively balanced presentation, to highlight the positive findings as well. Leading with the “good news” is often a good presentation strategy. The client is the organization that can make change. They have to understand the findings and be convinced that the findings are valid. It is at this point that the working relationships you have built with the other offices pays off.
- Pt. 2: No surprises! Positive attitude about information being publicly available.
- -- define vouchers -- define the nature of the earlier studies for Chicago (Geautreaux Case)
- Comment on Size/composition of Congressional staff Foundation participation
- Participants were all residents of public housing who applied to participate in a new program. So all were motivated to some degree to leave the “projects”
- Quality of schools measured by (examples) Percent receiving free lunches Percent white Pupil-student ratio Percentile rank on state exam Results were surprising. The treatment group should have been moving to better neighborhoods (as defined in terms of poverty rates) but that condition seemed not to have been sufficient to change the average school profile. This is different from earlier relocations. Results for employment are similar—measured by Currently employed Currently employed at job offering health insurance Currently employed full-time Annual earnings Current weekly earnings Employed over one year in main job
- Draft report was submitted to HUD in March 2003.
- Programs CDBG direct loan program, through banks S.108 guarantees loans made by private banks; guarantees are future funds to be received by the city through CDBG Credit enhancements are through: Section 108 loan guarantee program and the Brownfields Economic Development Initiative.
- Qualifying businesses Area standard. Business located in a CDBG neighborhood--80 percent of CDBG funds must be spent in areas where households have incomes less than 50 percent of the median for the urban area OR The business must primarily serve low-income families (50 percent of area median)
- Loan performance means default rates We do not further consider the issue of securitizing these loans.
- Standards to use in the evaluation were a significant issue. Jobs were those that the loan application claimed would be created as a consequence of receiving the loan. If anything, applicants had an incentive to overstate the number of jobs to give a greater justification for the subsidy. Results of lending programs working in other environments is of limited use but offers some guidance. If did as well as those in other environments, the program is doing very well. General business survival rates is a hard-to-interpret standard because of the very high rate of death of new firms. In U.S. in first two years about 20% of new firms fail each year.
- In reality banks are not making many loans to firms located in these neighborhoods and then only to the stronger firms. Indeed the reason the programs exist is to provide credit where it would not otherwise be available.
- Why was the contract so costly? Data collection in 51 high volume communities: Loan files from city agencies—976 third-party loans Information from borrowers on financial condition, employment, etc. over a several year period Supplementary data on businesses and loans made by banks Several hundred interviews with program directors and borrowers
- Explain difference between workplan and design report
- The study observed borrowers, on average, about 3 years after they had taken a loan. Failure rates for them were CDEBG loans 20% S.108 25 Compares with annual rates of 20% for all businesses in the early years down to about 7-8% by the ninth year. So overall look good.
- The guidance on underwriting guidelines would be based on extensive analysis of the probability of default on these types of loans that was conducted as part of the project.