SlideShare uma empresa Scribd logo
1 de 11
Baixar para ler offline
How to Hire


     The Right Evaluator
               For Your
Tobacco Prevention and Control Program




                   Jamey Wise, MS
                Director of Evaluation,
           Florida Tobacco Pilot Program

              Bureau of Epidemiology
           Florida Department of Health
                  February 2000
How to Hire the Right Evaluator

                      How to Hire the Right Evaluator
              for Your Tobacco Prevention and Control Program1

Introduction

    Program evaluation is an important aspect of every tobacco prevention and control program. The essential
components of successful programs can be identified and replicated through well-designed, systematic evalua-
tion. Evaluation also identifies less effective program elements that need to be modified. Ultimately, evaluation
results provide stakeholders − including the intended audience, community leaders and policy makers − with
evidence of the impacts of tobacco prevention and control activities, and help them make decisions about how
to improve these activities to achieve the desired outcomes.

     Many programs find hiring an external evaluator to be an effective means of obtaining a useful evaluation.
An external evaluator must be chosen carefully, however, because program personnel can easily become dis-
satisfied with an inappropriate or inadequate evaluation and may come to question the usefulness of the results
obtained.

    Well-planned and well-conducted evaluations are invaluable in determining where a tobacco prevention and
control program succeeds and how it can be improved. The following four guidelines can assist you in selecting
and working with an appropriate external evaluator.
     1. Define the evaluation.
     2. Select an evaluator.
     3. Work closely with the evaluator.
     4. Use the results.


Guideline 1: Define the evaluation.
     Defining the evaluation includes three primary steps:
     a) defining the focus of the evaluation;
     b) defining the evaluation activities; and
     c) developing a request for evaluation proposals.

     Evaluation Focus

    The focus of the evaluation is what you want to have evaluated. You need a clear understanding of what
you want the evaluation to accomplish. Do you want an assessment of how well you planned and implemented
your services? Do you want an assessment of the results of your services?

     Keep in mind that the evaluation should be decision-focused. At each step of the design process ask your-
self how the data collected will inform programmatic decisions. Evaluation can assist you in making many types
of decisions, including:
     1. information decisions: options for changes in collection and use of program performance information.
     2. management decisions: options for changes in program activities; and
     3. policy decisions: options for changes in program resources or objectives.




1
 This guide is based on an anonymously authored document, “Hiring the Right Evaluator for Your Program,” developed for the Corporation for
National Service, to which I contributed advice but did not author. I would also like to express my appreciation to the people who provided
helpful comments: Edward Trapido, Norman Weatherby, and Sharon Luebbers.


                                                                    1
How to Hire the Right Evaluator

    The focus in defining the scope of the evaluation is to identify the issues intended to be the focus of the
evaluation. Once the focus is established, then the key evaluation questions follow:
     ♦    How well are we managing our program?
     ♦    How well did we plan and implement our program?
     ♦    What are the outcomes achieved by our program?

    An evaluation can address any or all of these questions. These questions will eventually serve as the basis
for the evaluator to elaborate more detailed questions in an evaluation plan. Once you know what questions
you want answered, then you are ready to define the evaluation activities.

     Evaluation Activities

    Evaluation activities are the scope of work expected from the evaluator. These tasks should relate directly
to the conduct of the evaluation. The evaluator may be expected to:
     ♦    develop the evaluation plan;
     ♦    develop evaluation instruments;                               Using an Evaluation Committee

     ♦    select the sampling procedures and draw                           The question of what should be evaluated is often best an-
                                                                        swered by committee, and an Evaluation Committee can be a
          the evaluation sample;
                                                                        very valuable work group for all aspects of the evaluation.
     ♦    collect evaluation data;                                           An evaluation committee is useful for overseeing the entire
     ♦    analyze evaluation data;                                      evaluation process, from initial planning through implementa-
                                                                        tion and crafting of the final report. Persons representing the
     ♦    compose the evaluation report; and                            various aspects of your program, including Partnership and
     ♦    present the evaluation results.                               SWAT Members, supervisors and staff, partner agency staff,
                                                                        and other key people involved in programmatic planning best
     Requesting Evaluation Proposals                                    constitute this committee.
                                                                             The committee's size will depend upon the size of your
     Once the evaluation tasks have been                                program. A large program might support an evaluation com-
determined, you are ready to develop a request for                      mittee of five to seven persons, chaired by the program direc-
evaluation proposals (RFP)2. The RFP is designed                        tor. Smaller programs, of course, may require no more than
to solicit written evaluation proposals from which                      three people; some, only a single staff member. In other cases,
you can select the best candidate. The RFP must                         the evaluation committee might need to include the entire pro-
follow your agency’s purchasing guidelines, and                         gram staff.
should include a statement of the evaluation focus,                          The evaluation committee has primary responsibility for all
scope of work, and any other requirements. An                           aspects of the evaluation and receives regular reports from the
overall description of your tobacco prevention and                      external evaluator. Committee members must be actively in-
control program, an estimate of the available                           volved in the evaluation process to increase their understand-
                                                                        ing of it and to allow them to feel ownership of, and make use
evaluation funds, and all required meetings (e.g.
                                                                        of, evaluation results.
planning meeting, regular updates, interim report
presentation, final debrief) should also be included in               this document.

     The RFP should require evaluators to include several main sections in their evaluation proposal:
     1. Statement of the purpose of the evaluation. (This section ensures that the evaluator understands the
        intended focus of the evaluation being sought.)
     2. Statement of the program goals and objectives that are relevant to the focus of the evaluation. (This
        section ensures that the evaluator understands the program design.)
     3. Statement of proposed evaluation questions. These should be developed from the information provided
        under Guideline 2.
     4. Proposed methodology for addressing the evaluation questions.


2
  In Florida, the service procurement document may take the form of a less formal “invitation to negotiate” or more formal “request for proposals”
(or some other form). Check with your procurement office to determine which approach you should use. The steps outlined here work well under a
number of procurement methods.


                                                                        2
How to Hire the Right Evaluator

    5. Proposed timeline of evaluation activities.
    6. Proposed deliverables – typically a) evaluation instruments; b) data set; c) monthly reports; d) interim
       report (if needed); and e) final report.
    7. Proposed budget.

    Once the RFP has been completed, it should be distributed to a variety of local organizations. Suitable can-
didates with good credentials in program evaluation can usually be found in social science research organiza-
tions, institutions of higher education, and independent consultants. Try using your existing network to identify
persons who have conducted program evaluations for other organizations in your community or for other to-
bacco prevention and control programs.

        Helpful Hint: Under some circumstances it is not necessary that an RFP or other formal competitive
        process be used. Check with your budget or contract office for advice on the appropriate procedure for
        your program. If you do not conduct an RFP, the review and selection guidelines outlined her are still
        applicable – even for only one application.


Guideline 2: Select an evaluator.
    There are three basic steps in selecting the evaluator.
    a) reviewing and ranking the applications;
    b) interviewing the candidates; and
    c) negotiating and writing a contract.
An evaluator is not actually selected until a contract is executed. Steps a and b are used to assess and rank
candidates. During these phases of the selection process, the evaluators are also deciding whether to select
you. If you and the evaluator both agree that a good match is in the offing, then steps a and b become the
background work for step c, developing the contract. Each of these steps is described below.

    Reviewing and Ranking the Applicants

    Depending on your available resources and the number of available evaluators in your area, you will proba-
bly receive several applications or proposals. I recommend a two-step review process involving a review of the
proposals and then formal interviews with the top candidates. You would then use both the results of the pro-
posal review and of the interviews to hire the right evaluator for your program.

     The first step in selecting from among the applicants is to winnow down the proposals to the top three ap-
plicants through a thorough review of the proposals. The following review criteria will allow to you assess the
proposals based on the requirements established in the RFP. A sample evaluation review form is provided at
the end of this guide which includes these seven criteria and six additional criteria for interviewing the appli-
cants.

    1. How well does the evaluator understand the focus of the evaluation?

        If your intent is to assess how well services were planned and implemented, does the evaluator clearly
        state this focus in the proposal? If you want an assessment of the results or impact of the services,
        does the evaluator clearly state this focus in the proposal? Does the evaluator propose that the evalua-
        tion will be decision-focused − that is, that the evaluation findings will help you make decisions about
        your program, including the need for additional information, what activities might be changed, or new
        objectives that might be implemented?

        Importantly, the applicant's statement about the focus of the evaluation will be an indication of
        whether the evaluator understands the distinction between research and evaluation. Research is con-
        ducted primarily to expand our knowledge of a topic or program. Evaluation will help you decide what
        information to use and how to use it, how you might improve service activities, or how you might


                                                          3
How to Hire the Right Evaluator

    change your program's objectives. It is crucial that the evaluator understand that evaluation must help
    you make decisions about how to improve your program.

Program Evaluation or Research?
   Evaluation is closely related to, but distinguishable from social research. Evaluation utilizes many of the same
methods used in social research. According to Bill Trochim, evaluation takes place within a political and organ-
izational context and therefore it requires additional skills, including group process skills, management ability,
political savvy, and sensitivity to multiple stakeholders.
  A common definition of program evaluation is: the systematic assessment of the worth or merit of
some human service program in order to improve it. (Adapted from Michael Scriven)
   In contrast, a common definition of research is: any systematic activity designed to develop or con-
tribute to generalizeable knowledge. (Federal Regulations Regarding Human Subjects Protection)
Research emphasizes acquiring and assessing information. The primary purpose of research is to generate new
knowledge. Evaluation emphasizes assessing worth or merit in order to improve. The primary purpose of
evaluation is to determine the value of program strategies and improve the ones that can be improved.

2. How well does the evaluator understand the program goals and objectives?

    A good evaluation will be based on the goals and objectives you have established for your program.
    Does the evaluator communicate in the proposal a clear understanding of your program's goals and
    objectives?

    Does the evaluator understand your program's priorities? For example, if you emphasize retailer edu-
    cation over school-based instructional programs, the evaluation proposal should note these priorities.

3. How well do the proposed evaluation questions address the program goals and objectives?

    Evaluation questions are based on program goals and objectives. The most direct evaluation questions
    are simply objectives written as questions. For example, To what extent have comprehensive tobacco
    use prevention education programs been implemented in grades 9-12? (implementation assessment).
    Or, How have community leaders' attitudes toward tobacco changed during the past 12 months? (out-
    come assessment).

4. How appropriate is the proposed methodology for the evaluation questions?

    This can be a difficult issue to assess. Selecting evaluation methodologies can be a lot like selecting
    foods; not only are the needs of the people (program) important, but also their individual tastes. How-
    ever, the evaluator should clearly explain in the proposal why the methods being proposed are the
    most appropriate. In general, you will readily see the logic behind using surveys, records reviews,
    pre/post tests, or other approaches for answering questions about your program. Being intentionally
    redundant, whatever approach is used should yield information that you can use to make information,
    management, or policy decisions about your program.

    If you are not familiar or comfortable with assessing the proposed methodology, then I recommend
    that you enlist the services of an experienced evaluator to assist you with the review of proposals. An
    evaluator on the review team will serve two primary purposes: 1) assessing the appropriateness of the
    methodology; and 2) making recommendations for improving the evaluation methodology, if improve-
    ments are needed.

5. How reasonable is the timeline of evaluation activities?

    The reasonableness of timelines depends on several factors, including your program budget period,
    when you need the evaluation results, and the evaluation methods used. Don't rush the evaluation un-
    necessarily, but do allow yourself enough time after the evaluation is completed to use the results for
    your program planning.



                                                         4
How to Hire the Right Evaluator

   6. How relevant and useful are the proposed deliverables?

       The most common evaluation deliverable is a written report. I recommend that you look for several
       types of reports, each of which is used for a different purpose. Obviously, the most important reports
       are the final reports which present the evaluation results and the conclusions and recommendations of
       the evaluator. To ensure that the evaluation activities are being conducted as planned and within time-
       frames and budget, you should get monthly updates (these generally are brief reports). If the method-
       ology includes a pretest and posttest, then I recommend that you ask for a report of the pretest results
       as well as a report of the analysis of both the pretest and posttest. If the evaluation uses procedures
       that build on one another (for example, conducting a focus group and using the results to develop a
       survey) then a report on each component will be useful. For evaluations being conducted over a long
       time period, a comprehensive interim report might be useful.

        Helpful Hint: The larger the scope of the evaluation, or the more complex the methodology, the more
        there will be to report; it is commonly easier to digest several shorter and more focused reports than
        one large, detailed report.

   7. How reasonable is the proposed budget?

       Does the candidate believe your evaluation can be conducted for the available funds? Candidates must
       indicate that their proposed evaluation approach can be carried out for the funds you indicated would
       be available (or less!). You might find that a proposed evaluation plan is excellent, but unfeasible under
       your anticipated budget.

   Interviewing the Applicants

    The second step in selecting from among the applicants is to conduct formal interviews with the top three
candidates. The interviews should be used to clarify any concerns or questions raised during the proposal re-
view and to gauge evaluators' ability to work with your program.

   Request that each candidate interviewed provide you with previous evaluation reports they have written
and with references of people or organizations for whom they have previously conducted evaluation projects.

   8. How useful are the candidate’s previous evaluation reports?

       Look for evaluation reports for which the candidate served as lead author or as project direc-
       tor/manager (typically the second author in a report from a research institute). Assess the reports for
       clarity, organization, readability, and potential usefulness for decision makers. Pay particular attention
       to how well the reports would help a program improve its information collection, activities, or objec-
       tives. Candidates providing technical, poorly written, disorganized, difficult-to-understand, or lengthy
       evaluation reports will likely compose similar reports for your evaluation.

   9. Does the candidate have good references?

       References should be contacted to obtain their experiences with and opinions about the candidates.
       Some questions you might ask the references include:
       a. How well did the evaluation approach used by the evaluator address the needs and desires of your
          organization?
       b. Was the evaluation conducted in a timely fashion?
       c.   Was the evaluation conducted within your budget?
       d. What was the most useful aspect of the evaluation report? What changes, if any, did you make as
          a result of the evaluator's recommendations?
       e. Would you hire the evaluator to conduct another evaluation for you?




                                                          5
How to Hire the Right Evaluator

     The actual interview might begin with the evaluators presenting their proposed evaluation plan. Such pres-
entations provide you with much invaluable information about the candidates. You will learn about the evaluat-
ors' ability to organize information, communicate ideas, and respond to questions. Invite your staff, supervi-
sors, and program partners to the presentation to get their input. Remember: they also will be using the re-
sults.

    10. How effectively does the evaluator communicate with you and other stakeholders?

        If the discussion becomes very technical and a candidate is unable to present information in an easily
        understood manner, it is unlikely that this candidate will meet your needs. A candidate unable to com-
        municate effectively at this time will probably not overcome the problem during the evaluation. Effec-
        tive communication is a key for success, and the interview gives committee members a good idea of
        how effectively a candidate can communicate.

    In addition to whatever questions might arise during the presentation, the following questions might also
be useful.

    11. What is the candidate's prior evaluation experience?

        Experience is an important factor to consider. A candidate probably will not have performed exactly the
        same evaluation that you require, but many similarities between previous programs and your own can
        be found. A candidate that has performed evaluations of substance abuse prevention programs, for ex-
        ample, would have experiences that translate directly to tobacco prevention and control programs. De-
        pending on the emphases of your program, evaluation experience with law enforcement programs,
        school or community based educational programs, marketing, or community building might be relevant
        experience. The candidate's prior experience will be your main opportunity to discover and weigh that
        person's strengths and weaknesses.

    12. Will the candidate's existing professional commitments interfere with the planned evaluation?

        Good program evaluators are usually in high demand. A candidate who is engaged in several projects,
        however, may be unable to devote sufficient time to your program evaluation. However, you should
        expect that the quality of work you receive is comparable to any other project in which an evaluator is
        engaged.

        Ask the candidate to describe current and expected professional commitments. If the commitments
        seem excessive, ask how the candidate plans to conduct your program evaluation along with these
        other tasks. If the candidate indicates that other persons will be used to assist with the evaluation,
        determine which tasks win be performed by whom. Also determine if these other persons are capable
        of performing the tasks assigned to them, including requesting copies of resumes or vitas from the
        candidate's associates. Using a team of trained and experienced persons to perform an evaluation is
        common, but the team leader (i.e., the candidate) should be involved in all tasks that the you believe
        require this person's direct involvement.

        Finally, it is important to ensure that the evaluator you select does not have a conflict of interest. An
        evaluation is truly independent when the program evaluator has no stake in the program’s success or
        failure. For example, if the evaluator were a member of the management team, then he or she would
        obviously have a clear interest in the program results – both in terms of prestige and continued em-
        ployment. Such an interest in the program outcomes can affect the decisions of event the most honor-
        able evaluators. A conflict of interest questionnaire is included at the end of this guide that you can ask
        all evaluators to complete and include with their proposal.

    13. What is your general reaction to the candidate?

        During interviews, be alert to the candidate's ability to communicate in a straightforward manner, and
        be alert to your own expectations of how effectively you and your colleagues can work with this per-
        son. A clash of working styles can certainly be a problem, and the chemistry between a candidate and
        your program's stakeholders should not be ignored.

                                                        6
How to Hire the Right Evaluator

     Following the interview process, reviewers should individually rate the candidates on all of the issues previ-
ously described. Candidates might be rated on a five-point scale, ranging from "Definitely hire as our evaluator"
to "Definitely do not hire as our evaluator." "No opinion" should be the midpoint. (See Attachment 1 for a sam-
ple form for rating candidates.) Reviewers scores can be combined in a number of ways, from simply adding up
the total points for each criterion, to averaging the points for each criterion, to asking the reviewers to discuss
their individual ratings and then reach consensus on how a candidate should be rated on each criterion. After
combining reviewers' individual scores, the candidates should be ranked so that the project can be offered to
the candidate most acceptable to the reviewers.

    Negotiating and Writing a Contract

    The desired relationship between the evaluation committee and the external evaluator is one of partnership
and should be reflected as such in the contract. The contract should state, in a single paragraph if possible, the
evaluator's general responsibilities. If you have the evaluator produce a work plan, it can be incorporated into
your agreement. The contract should list the contract deliverables and provide a timetable for each one. Con-
sider including language describing how changes in the scope of work or work plan will be handled. Many
evaluation contracts also specify who owns the data gathered during the evaluation, as well as who has the
right to publish the results of the evaluation study. Finally, indicate how the evaluator will bill for services ren-
dered and include a schedule of payments. It is common to withhold between twenty and thirty percent of the
evaluator's fee until all deliverables have been submitted and reviewed to ensure that the deliverables respond
to the contracted scope of work.

    The contract should also detail the program's responsibilities. These responsibilities might include providing
the external evaluator with timely and appropriate guidance and reviewing and approving evaluation instru-
ments and documents in a timely and constructive manner. Your responsibilities might also include providing
the evaluator with program appropriate program records and other information, and assisting the evaluator in
solving problems that arise during the evaluation.


Guideline 3: Work closely with the evaluator.
    Successful evaluations are the result of good partnerships. Your involvement − and that of other
program stakeholders − is essential to the success of the evaluation. Your involvement in the evaluation proc-
ess should not be limited to periodic meetings with the evaluator. The greater your participation in all of the
various aspects of the evaluation, from planning to final report, the more complete the evaluation will be, and
the greater your understanding of the evaluation results and recommendations will be.

    You should continually monitor the evaluator and the evaluation. Review all major work elements, including
sampling plans, instrumentation, data collection plans, etc. You and your staff and partners can be of consider-
able service to the evaluator by facilitating involvement with host sites, service recipients, and others from
whom the evaluator wishes to collect information.

    Solicit feedback from your evaluator as the study progresses. Informal insights gained during the study pe-
riod can be valuable. Finally, manage the process actively to make sure you get what you need. A committee
that is intermittently involved in the evaluation process may discover that the study has gone in an inappropri-
ate direction after considerable time, money, and effort have been expended.

    As your evaluation nears completion, you and the evaluator should agree on a format for the evaluation re-
port. The evaluation report should address evaluation questions directly and briefly and should be understand-
able to the target audience. Any report, of course, must provide useful and direct guidance for program deci-
sion makers. You and the evaluator should also agree at this time on the evaluator's role in the release of the
evaluation's results. You might request that the evaluator be available to meet with decision makers, conduct
interviews with news media, and make public presentations of the results.

    Be involved in the development of the final report. This is not to say that you decide what the evaluator re-
ports; the point of an external evaluation is to get an independent assessment your program. However, your
knowledge of the target audience will be valuable in framing the report, and your knowledge of policy makers'


                                                         7
How to Hire the Right Evaluator

future plans for the program will help set the context for any recommendations made by the evaluators. You
should carefully review a draft of the final report and provide detailed, written feedback to the evaluator to use
in developing the final product.

    A cautionary note: If the changes suggested to the evaluator are significant and would have the effect of
changing the findings, recommendations, or overall focus of the report, they should be discussed at a meeting
with the evaluator. If the evaluator does not believe that the suggested changes are consistent with the data,
the evaluator has the right (and an ethical obligation) to be disassociated from the report. In such an instance,
the evaluator may make the requested changes and assign authorship of the report to the program, or may
hand over all work to the program to complete the report. Such situations should be avoided, however, as they
usually place the integrity and public acceptance of the report in jeopardy.


Guideline 4: Use the evaluation results.
     Once your study is complete, its value will be very limited unless the results are used to improve your pro-
gram. Use the insights gained to review your program design and modify, or replicate, program elements as
appropriate. Use your evaluator as a resource in this activity. Your evaluator's assistance in interpreting the
results for their program management implications can be very valuable. Address the most compelling findings
first. Don't get overwhelmed trying to do everything at once. Make note of findings that suggest further areas
of investigation. Future evaluations may shed light on unclear data.

    Use your study not only to improve operations, but to market your program. Potential funders and service
partners will be more willing to become involved with your program if you can show them evidence of effec-
tiveness and a willingness to make improvements based on evaluation data.

     Report the results of your evaluation not only to your staff and other program stakeholders, but also to fi-
nancial contributors, key community leaders, and organizational partners. Contact your local newspaper about
the possibility of doing a feature article on your program. Have program participants − especially youth mem-
bers − develop exhibitions including the evaluation results for use in community presentations and other activi-
ties. Present your results at professional conferences and other "issue networks" (groups with a common inter-
est).


Conclusion
     A program evaluation properly designed to provide interested individuals with information that they will use
will likely be a successful evaluation. However, while utilization of the results of the evaluation is clearly the
primary goal of an evaluation, important benefits may also be gained from the evaluation process itself. The
process of undergoing an evaluation can, for example, build shared meaning and understanding, support and
enhance the program (by building evaluation-based data collection and analysis into the program design), and
support human and organizational development by training staff in new skills. A well conceived evaluation will
provide benefits no matter the results.

    A well-conceived evaluation results from a good working partnerships between program stakeholders and
evaluators. An effective functional partnership is founded on agreement on the objectives of the evaluation, an
understanding of the responsibilities and authority of each partner, and mutual respect for the contributions
that each partner provides to the evaluation. If the steps identified here are followed, the evaluation process
should be enjoyable and productive for both the program and the external evaluator.




                                                        8
How to Hire the Right Evaluator

Evaluator Rating Guide                      Candidate Name:_______________________________________
                                                        2                1       0           -1              -2
1.   How well does the evaluator under-
                                              Clearly understands the        No Opinion           Has no understanding of
     stand the focus of the evaluation?      focus, including the dis-                             the evaluation focus or
                                             tinction between evalua-                             the distinction between
                                                 tion and research                                evaluation and research
                                                        2                1       0           -1              -2
2.   How well does the evaluator under-
                                             Has an excellent under-         No Opinion           Has no understanding of
     stand the program goals and objec-       standing of program                                  program goals and ob-
     tives?                                   goals and objectives                                        jectives
                                                        2                1       0           -1              -2
3.   How well do the proposed evaluation
                                            Evaluation questions fully       No Opinion            Evaluation questions to
     questions address the program goals     address program goals                                  not address program
     and objectives?                             and objectives                                     goals and objectives
                                                        2                1       0           -1              -2
4.   How appropriate is the proposed
                                             Appropriate: can clearly        No Opinion           Not appropriate: cannot
     methodology for the evaluation          see that the results ob-                             see how results obtained
     questions?                               tained will be useful                                 might help program
                                                        2                1       0           -1              -2
5.   How reasonable is the timeline of
                                              The evaluation can be          No Opinion           The evaluation cannot be
     evaluation activities?                 conducted well within the                               completed within the
                                              proposed time frame                                   proposed time frame
                                                        2                1       0           -1              -2
6.   How relevant and useful are the
                                             Very useful for program         No Opinion            Not useful for program
     proposed deliverables?                       improvement                                           improvement
                                                        2                1       0           -1              -2
7.   How reasonable is the proposed
                                             Evaluation is very likely       No Opinion           Evaluation can't be con-
     budget?                                 to be completed within                                ducted with available
                                             the available resources                                     resources
                                                        2                1       0           -1              -2
8.   How useful are the candidate’s pre-
                                              Previous reports were          No Opinion           Previous reports were not
     vious evaluation reports?                understood and used                                    understood or used
                                                        2                1       0           -1              -2
9.   Does the candidate have good refer-
                                                Received excellent           No Opinion           Received poor references
     ences?                                        references
                                                        2                1       0           -1              -2
10. How effectively does the evaluator
                                            Communicates effectively         No Opinion           Communicates effectively
    communicate with you and other           with most stakeholders                                with few stakeholders
    stakeholders?
                                                        2                1       0           -1              -2
11. What is the candidate's prior evalua-
                                            Has extensive experience         No Opinion            Has limited evaluation
    tion experience?                         evaluating similar pro-                                     experience
                                                     grams
                                                        2                1       0           -1              -2
12. Will the candidate's existing profes-
                                               Commitments will not          No Opinion           Commitments will inter-
    sional commitments interfere with        interfere with evaluation                              fere with evaluation
    the planned evaluation?
                                                        2                1       0           -1              -2
13. What is your general reaction to the
                                              Definitely hire as our         No Opinion            Definitely DO NOT hire
    candidate?                                      evaluator                                        has our evaluator




                                                               9
How to Hire the Right Evaluator

                       Evaluator Conflict of Interest Questionnaire


              Name of Program:_______________________________________

                                                                                Yes      No

1. Do you, your immediate family, or business partners have financial or
   other interests in the program to be evaluated?

2. Have you been employed by the program within the last 24 months?

3. Do you plan to obtain a financial interest, e.g. stock, in the program?

4. Do you plan to seek or accept future employment the program?

5. Are there any other conditions that may cause a conflict of interest?

If you answered Yes to any of the above questions, please provide a written explanation of
your answer.




I declare all of the above questions are answered truthfully and to the best of my knowledge.



signature                                                          date




                                                  10

Mais conteúdo relacionado

Mais procurados

Risk management overview
Risk management overviewRisk management overview
Risk management overviewHossam Maghrabi
 
PMBOK_5th_Initiating Process Group -
PMBOK_5th_Initiating Process Group -PMBOK_5th_Initiating Process Group -
PMBOK_5th_Initiating Process Group -Hossam Maghrabi
 
C8 logical framework approach (lfa)
C8 logical framework approach (lfa)C8 logical framework approach (lfa)
C8 logical framework approach (lfa)ocasiconference
 
project monitoring and evaluation
 project monitoring and evaluation project monitoring and evaluation
project monitoring and evaluationMir Mosharraf Hosain
 
Project: Monitoring
Project: Monitoring Project: Monitoring
Project: Monitoring Mfb Firoz
 
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...Dubai Quality Group
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]skzarif
 
Software development o & c
Software development o & cSoftware development o & c
Software development o & cAmit Patil
 
Lfa Logical Framework Analysis
Lfa   Logical Framework AnalysisLfa   Logical Framework Analysis
Lfa Logical Framework AnalysisJFM Lohith Shetty
 
TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)Research to Action
 
Monitoring and evaluation of human rights projects
Monitoring and evaluation of human rights projectsMonitoring and evaluation of human rights projects
Monitoring and evaluation of human rights projectsInka Pibilova
 
Taking the Mystery out of Project Evaluation
Taking the Mystery out of Project EvaluationTaking the Mystery out of Project Evaluation
Taking the Mystery out of Project EvaluationLisa Glickstein
 
Perform qualitative risk analysis
Perform qualitative risk analysis Perform qualitative risk analysis
Perform qualitative risk analysis Shereef Sabri
 
Measurement of project quality performance
Measurement of project quality performanceMeasurement of project quality performance
Measurement of project quality performanceK.K. PATHAK
 
PMP Training - 11 project risk management
PMP Training - 11 project risk managementPMP Training - 11 project risk management
PMP Training - 11 project risk managementejlp12
 
Proposal writing resource logical framework-
Proposal writing resource  logical framework-Proposal writing resource  logical framework-
Proposal writing resource logical framework-tccafrica
 

Mais procurados (20)

Risk management overview
Risk management overviewRisk management overview
Risk management overview
 
PMBOK_5th_Initiating Process Group -
PMBOK_5th_Initiating Process Group -PMBOK_5th_Initiating Process Group -
PMBOK_5th_Initiating Process Group -
 
C8 logical framework approach (lfa)
C8 logical framework approach (lfa)C8 logical framework approach (lfa)
C8 logical framework approach (lfa)
 
project monitoring and evaluation
 project monitoring and evaluation project monitoring and evaluation
project monitoring and evaluation
 
Project: Monitoring
Project: Monitoring Project: Monitoring
Project: Monitoring
 
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...
WQD2011 - Breakthrough Process Improvement - Tawam Hospital - The Surgical Ad...
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]
 
Software development o & c
Software development o & cSoftware development o & c
Software development o & c
 
Lfa Logical Framework Analysis
Lfa   Logical Framework AnalysisLfa   Logical Framework Analysis
Lfa Logical Framework Analysis
 
PMP_Project Risk Management
PMP_Project Risk ManagementPMP_Project Risk Management
PMP_Project Risk Management
 
TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)TTIPEC: Monitoring and Evaluation (Session 2)
TTIPEC: Monitoring and Evaluation (Session 2)
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 
Monitoring and evaluation of human rights projects
Monitoring and evaluation of human rights projectsMonitoring and evaluation of human rights projects
Monitoring and evaluation of human rights projects
 
Taking the Mystery out of Project Evaluation
Taking the Mystery out of Project EvaluationTaking the Mystery out of Project Evaluation
Taking the Mystery out of Project Evaluation
 
Perform qualitative risk analysis
Perform qualitative risk analysis Perform qualitative risk analysis
Perform qualitative risk analysis
 
Measurement of project quality performance
Measurement of project quality performanceMeasurement of project quality performance
Measurement of project quality performance
 
Project Monitoring & Evaluation
Project Monitoring & EvaluationProject Monitoring & Evaluation
Project Monitoring & Evaluation
 
PMP Training - 11 project risk management
PMP Training - 11 project risk managementPMP Training - 11 project risk management
PMP Training - 11 project risk management
 
Proposal writing resource logical framework-
Proposal writing resource  logical framework-Proposal writing resource  logical framework-
Proposal writing resource logical framework-
 
Project Risk Management
Project Risk ManagementProject Risk Management
Project Risk Management
 

Destaque (9)

Fidelity Defined
Fidelity DefinedFidelity Defined
Fidelity Defined
 
Next generation science standards for states by states
Next generation science standards for states by statesNext generation science standards for states by states
Next generation science standards for states by states
 
Strategic Plan Fy2011 Final
Strategic Plan Fy2011 FinalStrategic Plan Fy2011 Final
Strategic Plan Fy2011 Final
 
Ignoring Drug And Alcohol Prevention Is Costly
Ignoring Drug And Alcohol Prevention Is CostlyIgnoring Drug And Alcohol Prevention Is Costly
Ignoring Drug And Alcohol Prevention Is Costly
 
Cong Lujan Prevention
Cong Lujan PreventionCong Lujan Prevention
Cong Lujan Prevention
 
2015 YRRS SYNOPSIS 31 NM COUNTIES
2015 YRRS SYNOPSIS 31 NM COUNTIES2015 YRRS SYNOPSIS 31 NM COUNTIES
2015 YRRS SYNOPSIS 31 NM COUNTIES
 
Prevention Practices In The 21st Century FM
Prevention Practices  In The 21st Century FMPrevention Practices  In The 21st Century FM
Prevention Practices In The 21st Century FM
 
2017 Evidence-based prevention national standards
2017 Evidence-based prevention national standards2017 Evidence-based prevention national standards
2017 Evidence-based prevention national standards
 
Inter-professional Education for Collaboration: Learning How to
Inter-professional Education for Collaboration: Learning How toInter-professional Education for Collaboration: Learning How to
Inter-professional Education for Collaboration: Learning How to
 

Semelhante a Evalhire

Basic Guide to Program Evaluation (Including Outcomes Evaluation).docx
Basic Guide to Program Evaluation (Including Outcomes Evaluation).docxBasic Guide to Program Evaluation (Including Outcomes Evaluation).docx
Basic Guide to Program Evaluation (Including Outcomes Evaluation).docxJASS44
 
IFRC Framework for Evaluation
IFRC  Framework for  EvaluationIFRC  Framework for  Evaluation
IFRC Framework for Evaluationsgchaplowe
 
\\Storage\Personal Files\Module V\Utec
\\Storage\Personal Files\Module V\Utec\\Storage\Personal Files\Module V\Utec
\\Storage\Personal Files\Module V\Utechildagomez
 
INTRODUCTION TO PROGRAMME DEVELOPMENT..ppt
INTRODUCTION TO PROGRAMME DEVELOPMENT..pptINTRODUCTION TO PROGRAMME DEVELOPMENT..ppt
INTRODUCTION TO PROGRAMME DEVELOPMENT..pptmodestuseveline
 
Management oriented evaluation approaches
Management oriented evaluation approachesManagement oriented evaluation approaches
Management oriented evaluation approachesJessica Bernardino
 
Training on Evaluation.pptx
Training on Evaluation.pptxTraining on Evaluation.pptx
Training on Evaluation.pptxssusere0ee1d
 
Peer Review white paper
Peer Review white paperPeer Review white paper
Peer Review white paperAlvyn Yeoh
 
Evaluation of health programs
Evaluation of health programsEvaluation of health programs
Evaluation of health programsnium
 
PDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation PlanPDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation Plankpravera
 
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxPJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxinfantsuk
 
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxPJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxstilliegeorgiana
 
Planning and Designing Evaluation
Planning and Designing EvaluationPlanning and Designing Evaluation
Planning and Designing EvaluationMarlin Dwinastiti
 
June 20 2010 bsi christie
June 20 2010 bsi christieJune 20 2010 bsi christie
June 20 2010 bsi christieharrindl
 
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptx
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptxPROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptx
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptxleamangaring12
 

Semelhante a Evalhire (20)

Basic Guide to Program Evaluation (Including Outcomes Evaluation).docx
Basic Guide to Program Evaluation (Including Outcomes Evaluation).docxBasic Guide to Program Evaluation (Including Outcomes Evaluation).docx
Basic Guide to Program Evaluation (Including Outcomes Evaluation).docx
 
IFRC Framework for Evaluation
IFRC  Framework for  EvaluationIFRC  Framework for  Evaluation
IFRC Framework for Evaluation
 
Program Evaluation 1
Program Evaluation 1Program Evaluation 1
Program Evaluation 1
 
A2011214171642 1
A2011214171642 1A2011214171642 1
A2011214171642 1
 
\\Storage\Personal Files\Module V\Utec
\\Storage\Personal Files\Module V\Utec\\Storage\Personal Files\Module V\Utec
\\Storage\Personal Files\Module V\Utec
 
INTRODUCTION TO PROGRAMME DEVELOPMENT..ppt
INTRODUCTION TO PROGRAMME DEVELOPMENT..pptINTRODUCTION TO PROGRAMME DEVELOPMENT..ppt
INTRODUCTION TO PROGRAMME DEVELOPMENT..ppt
 
Management oriented evaluation approaches
Management oriented evaluation approachesManagement oriented evaluation approaches
Management oriented evaluation approaches
 
Training on Evaluation.pptx
Training on Evaluation.pptxTraining on Evaluation.pptx
Training on Evaluation.pptx
 
Peer Review white paper
Peer Review white paperPeer Review white paper
Peer Review white paper
 
Evaluation of health programs
Evaluation of health programsEvaluation of health programs
Evaluation of health programs
 
PDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation PlanPDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation Plan
 
OECD Due Diligence Alignment Assessment Tool for responsible supply chains in...
OECD Due Diligence Alignment Assessment Tool for responsible supply chains in...OECD Due Diligence Alignment Assessment Tool for responsible supply chains in...
OECD Due Diligence Alignment Assessment Tool for responsible supply chains in...
 
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxPJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
 
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docxPJM6125 Project Evaluation Selecting Evaluation Tools .docx
PJM6125 Project Evaluation Selecting Evaluation Tools .docx
 
Planning and Designing Evaluation
Planning and Designing EvaluationPlanning and Designing Evaluation
Planning and Designing Evaluation
 
Mentoring 418
Mentoring 418Mentoring 418
Mentoring 418
 
June 20 2010 bsi christie
June 20 2010 bsi christieJune 20 2010 bsi christie
June 20 2010 bsi christie
 
UP ACE MTL Unit 3 Session 7
UP ACE MTL Unit 3 Session 7UP ACE MTL Unit 3 Session 7
UP ACE MTL Unit 3 Session 7
 
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptx
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptxPROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptx
PROJECTPROGRAM IMPLEMENTATION, MONITORING AND EVALUATION.pptx
 
evaluation
evaluationevaluation
evaluation
 

Mais de University of New Mexico

BEST PRACTICES Comprehensive Resources Compendium (1)
BEST PRACTICES Comprehensive Resources Compendium (1)BEST PRACTICES Comprehensive Resources Compendium (1)
BEST PRACTICES Comprehensive Resources Compendium (1)University of New Mexico
 
Einstein's unfinished symphony listening to the sounds of space time
Einstein's unfinished symphony  listening to the sounds of space timeEinstein's unfinished symphony  listening to the sounds of space time
Einstein's unfinished symphony listening to the sounds of space timeUniversity of New Mexico
 
Forum on investing in young children globally
Forum on investing in young children globallyForum on investing in young children globally
Forum on investing in young children globallyUniversity of New Mexico
 
Financing population health improvement IOM
Financing population health improvement IOMFinancing population health improvement IOM
Financing population health improvement IOMUniversity of New Mexico
 
Business engagement in building healthy communities
Business engagement in building healthy communitiesBusiness engagement in building healthy communities
Business engagement in building healthy communitiesUniversity of New Mexico
 
Promising and best practices in total worker health
Promising and best practices in total worker healthPromising and best practices in total worker health
Promising and best practices in total worker healthUniversity of New Mexico
 
Building capacity to reduce bullying (2014); NAP-IOM
Building capacity to reduce bullying (2014); NAP-IOMBuilding capacity to reduce bullying (2014); NAP-IOM
Building capacity to reduce bullying (2014); NAP-IOMUniversity of New Mexico
 
WHO CDC Preventing Suicide: A Global Imperative
WHO CDC Preventing Suicide: A Global ImperativeWHO CDC Preventing Suicide: A Global Imperative
WHO CDC Preventing Suicide: A Global ImperativeUniversity of New Mexico
 
Data based planning for effective prevention
Data based planning for effective preventionData based planning for effective prevention
Data based planning for effective preventionUniversity of New Mexico
 

Mais de University of New Mexico (20)

BE ABOVE THE INFLUENCE NEWS MARCH 2017
BE ABOVE THE INFLUENCE NEWS MARCH 2017BE ABOVE THE INFLUENCE NEWS MARCH 2017
BE ABOVE THE INFLUENCE NEWS MARCH 2017
 
ATI NEWS #1
ATI NEWS #1ATI NEWS #1
ATI NEWS #1
 
BEST PRACTICES Comprehensive Resources Compendium (1)
BEST PRACTICES Comprehensive Resources Compendium (1)BEST PRACTICES Comprehensive Resources Compendium (1)
BEST PRACTICES Comprehensive Resources Compendium (1)
 
Einstein's unfinished symphony listening to the sounds of space time
Einstein's unfinished symphony  listening to the sounds of space timeEinstein's unfinished symphony  listening to the sounds of space time
Einstein's unfinished symphony listening to the sounds of space time
 
YOUTH BEHAVIORAL HEALTH 2015
YOUTH BEHAVIORAL HEALTH 2015YOUTH BEHAVIORAL HEALTH 2015
YOUTH BEHAVIORAL HEALTH 2015
 
SUNPORT ATI FY16 BANNER
SUNPORT ATI FY16 BANNERSUNPORT ATI FY16 BANNER
SUNPORT ATI FY16 BANNER
 
Forum on investing in young children globally
Forum on investing in young children globallyForum on investing in young children globally
Forum on investing in young children globally
 
Interventions for mental & substance use
Interventions for mental & substance useInterventions for mental & substance use
Interventions for mental & substance use
 
ATI FY16 BUS AD2
ATI FY16 BUS AD2ATI FY16 BUS AD2
ATI FY16 BUS AD2
 
IOM BUILDING CAPACITY TO REDUCE BULLYING
IOM BUILDING CAPACITY TO REDUCE BULLYINGIOM BUILDING CAPACITY TO REDUCE BULLYING
IOM BUILDING CAPACITY TO REDUCE BULLYING
 
Iom building capacity to reduce bullying
Iom building capacity to reduce bullyingIom building capacity to reduce bullying
Iom building capacity to reduce bullying
 
Alcohol Prices Study nihms441745
Alcohol Prices Study nihms441745Alcohol Prices Study nihms441745
Alcohol Prices Study nihms441745
 
Financing population health improvement IOM
Financing population health improvement IOMFinancing population health improvement IOM
Financing population health improvement IOM
 
Business engagement in building healthy communities
Business engagement in building healthy communitiesBusiness engagement in building healthy communities
Business engagement in building healthy communities
 
Promising and best practices in total worker health
Promising and best practices in total worker healthPromising and best practices in total worker health
Promising and best practices in total worker health
 
Building capacity to reduce bullying (2014); NAP-IOM
Building capacity to reduce bullying (2014); NAP-IOMBuilding capacity to reduce bullying (2014); NAP-IOM
Building capacity to reduce bullying (2014); NAP-IOM
 
WHO CDC Preventing Suicide: A Global Imperative
WHO CDC Preventing Suicide: A Global ImperativeWHO CDC Preventing Suicide: A Global Imperative
WHO CDC Preventing Suicide: A Global Imperative
 
COMMUNITY PLANNING SAMHSA
COMMUNITY PLANNING SAMHSACOMMUNITY PLANNING SAMHSA
COMMUNITY PLANNING SAMHSA
 
Connected learning
Connected learningConnected learning
Connected learning
 
Data based planning for effective prevention
Data based planning for effective preventionData based planning for effective prevention
Data based planning for effective prevention
 

Último

Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...
Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...
Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...narwatsonia7
 
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbers
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbersBook Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbers
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbersnarwatsonia7
 
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Availablenarwatsonia7
 
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Me
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near MeHigh Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Me
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Menarwatsonia7
 
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbai
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service MumbaiLow Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbai
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbaisonalikaur4
 
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdf
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdfHemostasis Physiology and Clinical correlations by Dr Faiza.pdf
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdfMedicoseAcademics
 
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment Booking
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment BookingCall Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment Booking
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment BookingNehru place Escorts
 
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...narwatsonia7
 
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipur
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service JaipurHigh Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipur
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipurparulsinha
 
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Availablenarwatsonia7
 
See the 2,456 pharmacies on the National E-Pharmacy Platform
See the 2,456 pharmacies on the National E-Pharmacy PlatformSee the 2,456 pharmacies on the National E-Pharmacy Platform
See the 2,456 pharmacies on the National E-Pharmacy PlatformKweku Zurek
 
Statistical modeling in pharmaceutical research and development.
Statistical modeling in pharmaceutical research and development.Statistical modeling in pharmaceutical research and development.
Statistical modeling in pharmaceutical research and development.ANJALI
 
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...narwatsonia7
 
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...narwatsonia7
 
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Availablenarwatsonia7
 
Pharmaceutical Marketting: Unit-5, Pricing
Pharmaceutical Marketting: Unit-5, PricingPharmaceutical Marketting: Unit-5, Pricing
Pharmaceutical Marketting: Unit-5, PricingArunagarwal328757
 
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment Booking
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment BookingCall Girl Koramangala | 7001305949 At Low Cost Cash Payment Booking
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment Bookingnarwatsonia7
 
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort Service
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort ServiceCollege Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort Service
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort ServiceNehru place Escorts
 
Hematology and Immunology - Leukocytes Functions
Hematology and Immunology - Leukocytes FunctionsHematology and Immunology - Leukocytes Functions
Hematology and Immunology - Leukocytes FunctionsMedicoseAcademics
 
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking ModelsMumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking Modelssonalikaur4
 

Último (20)

Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...
Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...
Call Girls Electronic City Just Call 7001305949 Top Class Call Girl Service A...
 
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbers
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbersBook Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbers
Book Call Girls in Kasavanahalli - 7001305949 with real photos and phone numbers
 
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hsr Layout Just Call 7001305949 Top Class Call Girl Service Available
 
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Me
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near MeHigh Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Me
High Profile Call Girls Mavalli - 7001305949 | 24x7 Service Available Near Me
 
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbai
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service MumbaiLow Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbai
Low Rate Call Girls Mumbai Suman 9910780858 Independent Escort Service Mumbai
 
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdf
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdfHemostasis Physiology and Clinical correlations by Dr Faiza.pdf
Hemostasis Physiology and Clinical correlations by Dr Faiza.pdf
 
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment Booking
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment BookingCall Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment Booking
Call Girls Service Nandiambakkam | 7001305949 At Low Cost Cash Payment Booking
 
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...
Housewife Call Girls Hsr Layout - Call 7001305949 Rs-3500 with A/C Room Cash ...
 
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipur
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service JaipurHigh Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipur
High Profile Call Girls Jaipur Vani 8445551418 Independent Escort Service Jaipur
 
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Hebbal Just Call 7001305949 Top Class Call Girl Service Available
 
See the 2,456 pharmacies on the National E-Pharmacy Platform
See the 2,456 pharmacies on the National E-Pharmacy PlatformSee the 2,456 pharmacies on the National E-Pharmacy Platform
See the 2,456 pharmacies on the National E-Pharmacy Platform
 
Statistical modeling in pharmaceutical research and development.
Statistical modeling in pharmaceutical research and development.Statistical modeling in pharmaceutical research and development.
Statistical modeling in pharmaceutical research and development.
 
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...
Call Girls Frazer Town Just Call 7001305949 Top Class Call Girl Service Avail...
 
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...
High Profile Call Girls Kodigehalli - 7001305949 Escorts Service with Real Ph...
 
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service AvailableCall Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Available
Call Girls Jayanagar Just Call 7001305949 Top Class Call Girl Service Available
 
Pharmaceutical Marketting: Unit-5, Pricing
Pharmaceutical Marketting: Unit-5, PricingPharmaceutical Marketting: Unit-5, Pricing
Pharmaceutical Marketting: Unit-5, Pricing
 
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment Booking
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment BookingCall Girl Koramangala | 7001305949 At Low Cost Cash Payment Booking
Call Girl Koramangala | 7001305949 At Low Cost Cash Payment Booking
 
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort Service
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort ServiceCollege Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort Service
College Call Girls Vyasarpadi Whatsapp 7001305949 Independent Escort Service
 
Hematology and Immunology - Leukocytes Functions
Hematology and Immunology - Leukocytes FunctionsHematology and Immunology - Leukocytes Functions
Hematology and Immunology - Leukocytes Functions
 
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking ModelsMumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
Mumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
 

Evalhire

  • 1. How to Hire The Right Evaluator For Your Tobacco Prevention and Control Program Jamey Wise, MS Director of Evaluation, Florida Tobacco Pilot Program Bureau of Epidemiology Florida Department of Health February 2000
  • 2. How to Hire the Right Evaluator How to Hire the Right Evaluator for Your Tobacco Prevention and Control Program1 Introduction Program evaluation is an important aspect of every tobacco prevention and control program. The essential components of successful programs can be identified and replicated through well-designed, systematic evalua- tion. Evaluation also identifies less effective program elements that need to be modified. Ultimately, evaluation results provide stakeholders − including the intended audience, community leaders and policy makers − with evidence of the impacts of tobacco prevention and control activities, and help them make decisions about how to improve these activities to achieve the desired outcomes. Many programs find hiring an external evaluator to be an effective means of obtaining a useful evaluation. An external evaluator must be chosen carefully, however, because program personnel can easily become dis- satisfied with an inappropriate or inadequate evaluation and may come to question the usefulness of the results obtained. Well-planned and well-conducted evaluations are invaluable in determining where a tobacco prevention and control program succeeds and how it can be improved. The following four guidelines can assist you in selecting and working with an appropriate external evaluator. 1. Define the evaluation. 2. Select an evaluator. 3. Work closely with the evaluator. 4. Use the results. Guideline 1: Define the evaluation. Defining the evaluation includes three primary steps: a) defining the focus of the evaluation; b) defining the evaluation activities; and c) developing a request for evaluation proposals. Evaluation Focus The focus of the evaluation is what you want to have evaluated. You need a clear understanding of what you want the evaluation to accomplish. Do you want an assessment of how well you planned and implemented your services? Do you want an assessment of the results of your services? Keep in mind that the evaluation should be decision-focused. At each step of the design process ask your- self how the data collected will inform programmatic decisions. Evaluation can assist you in making many types of decisions, including: 1. information decisions: options for changes in collection and use of program performance information. 2. management decisions: options for changes in program activities; and 3. policy decisions: options for changes in program resources or objectives. 1 This guide is based on an anonymously authored document, “Hiring the Right Evaluator for Your Program,” developed for the Corporation for National Service, to which I contributed advice but did not author. I would also like to express my appreciation to the people who provided helpful comments: Edward Trapido, Norman Weatherby, and Sharon Luebbers. 1
  • 3. How to Hire the Right Evaluator The focus in defining the scope of the evaluation is to identify the issues intended to be the focus of the evaluation. Once the focus is established, then the key evaluation questions follow: ♦ How well are we managing our program? ♦ How well did we plan and implement our program? ♦ What are the outcomes achieved by our program? An evaluation can address any or all of these questions. These questions will eventually serve as the basis for the evaluator to elaborate more detailed questions in an evaluation plan. Once you know what questions you want answered, then you are ready to define the evaluation activities. Evaluation Activities Evaluation activities are the scope of work expected from the evaluator. These tasks should relate directly to the conduct of the evaluation. The evaluator may be expected to: ♦ develop the evaluation plan; ♦ develop evaluation instruments; Using an Evaluation Committee ♦ select the sampling procedures and draw The question of what should be evaluated is often best an- swered by committee, and an Evaluation Committee can be a the evaluation sample; very valuable work group for all aspects of the evaluation. ♦ collect evaluation data; An evaluation committee is useful for overseeing the entire ♦ analyze evaluation data; evaluation process, from initial planning through implementa- tion and crafting of the final report. Persons representing the ♦ compose the evaluation report; and various aspects of your program, including Partnership and ♦ present the evaluation results. SWAT Members, supervisors and staff, partner agency staff, and other key people involved in programmatic planning best Requesting Evaluation Proposals constitute this committee. The committee's size will depend upon the size of your Once the evaluation tasks have been program. A large program might support an evaluation com- determined, you are ready to develop a request for mittee of five to seven persons, chaired by the program direc- evaluation proposals (RFP)2. The RFP is designed tor. Smaller programs, of course, may require no more than to solicit written evaluation proposals from which three people; some, only a single staff member. In other cases, you can select the best candidate. The RFP must the evaluation committee might need to include the entire pro- follow your agency’s purchasing guidelines, and gram staff. should include a statement of the evaluation focus, The evaluation committee has primary responsibility for all scope of work, and any other requirements. An aspects of the evaluation and receives regular reports from the overall description of your tobacco prevention and external evaluator. Committee members must be actively in- control program, an estimate of the available volved in the evaluation process to increase their understand- ing of it and to allow them to feel ownership of, and make use evaluation funds, and all required meetings (e.g. of, evaluation results. planning meeting, regular updates, interim report presentation, final debrief) should also be included in this document. The RFP should require evaluators to include several main sections in their evaluation proposal: 1. Statement of the purpose of the evaluation. (This section ensures that the evaluator understands the intended focus of the evaluation being sought.) 2. Statement of the program goals and objectives that are relevant to the focus of the evaluation. (This section ensures that the evaluator understands the program design.) 3. Statement of proposed evaluation questions. These should be developed from the information provided under Guideline 2. 4. Proposed methodology for addressing the evaluation questions. 2 In Florida, the service procurement document may take the form of a less formal “invitation to negotiate” or more formal “request for proposals” (or some other form). Check with your procurement office to determine which approach you should use. The steps outlined here work well under a number of procurement methods. 2
  • 4. How to Hire the Right Evaluator 5. Proposed timeline of evaluation activities. 6. Proposed deliverables – typically a) evaluation instruments; b) data set; c) monthly reports; d) interim report (if needed); and e) final report. 7. Proposed budget. Once the RFP has been completed, it should be distributed to a variety of local organizations. Suitable can- didates with good credentials in program evaluation can usually be found in social science research organiza- tions, institutions of higher education, and independent consultants. Try using your existing network to identify persons who have conducted program evaluations for other organizations in your community or for other to- bacco prevention and control programs. Helpful Hint: Under some circumstances it is not necessary that an RFP or other formal competitive process be used. Check with your budget or contract office for advice on the appropriate procedure for your program. If you do not conduct an RFP, the review and selection guidelines outlined her are still applicable – even for only one application. Guideline 2: Select an evaluator. There are three basic steps in selecting the evaluator. a) reviewing and ranking the applications; b) interviewing the candidates; and c) negotiating and writing a contract. An evaluator is not actually selected until a contract is executed. Steps a and b are used to assess and rank candidates. During these phases of the selection process, the evaluators are also deciding whether to select you. If you and the evaluator both agree that a good match is in the offing, then steps a and b become the background work for step c, developing the contract. Each of these steps is described below. Reviewing and Ranking the Applicants Depending on your available resources and the number of available evaluators in your area, you will proba- bly receive several applications or proposals. I recommend a two-step review process involving a review of the proposals and then formal interviews with the top candidates. You would then use both the results of the pro- posal review and of the interviews to hire the right evaluator for your program. The first step in selecting from among the applicants is to winnow down the proposals to the top three ap- plicants through a thorough review of the proposals. The following review criteria will allow to you assess the proposals based on the requirements established in the RFP. A sample evaluation review form is provided at the end of this guide which includes these seven criteria and six additional criteria for interviewing the appli- cants. 1. How well does the evaluator understand the focus of the evaluation? If your intent is to assess how well services were planned and implemented, does the evaluator clearly state this focus in the proposal? If you want an assessment of the results or impact of the services, does the evaluator clearly state this focus in the proposal? Does the evaluator propose that the evalua- tion will be decision-focused − that is, that the evaluation findings will help you make decisions about your program, including the need for additional information, what activities might be changed, or new objectives that might be implemented? Importantly, the applicant's statement about the focus of the evaluation will be an indication of whether the evaluator understands the distinction between research and evaluation. Research is con- ducted primarily to expand our knowledge of a topic or program. Evaluation will help you decide what information to use and how to use it, how you might improve service activities, or how you might 3
  • 5. How to Hire the Right Evaluator change your program's objectives. It is crucial that the evaluator understand that evaluation must help you make decisions about how to improve your program. Program Evaluation or Research? Evaluation is closely related to, but distinguishable from social research. Evaluation utilizes many of the same methods used in social research. According to Bill Trochim, evaluation takes place within a political and organ- izational context and therefore it requires additional skills, including group process skills, management ability, political savvy, and sensitivity to multiple stakeholders. A common definition of program evaluation is: the systematic assessment of the worth or merit of some human service program in order to improve it. (Adapted from Michael Scriven) In contrast, a common definition of research is: any systematic activity designed to develop or con- tribute to generalizeable knowledge. (Federal Regulations Regarding Human Subjects Protection) Research emphasizes acquiring and assessing information. The primary purpose of research is to generate new knowledge. Evaluation emphasizes assessing worth or merit in order to improve. The primary purpose of evaluation is to determine the value of program strategies and improve the ones that can be improved. 2. How well does the evaluator understand the program goals and objectives? A good evaluation will be based on the goals and objectives you have established for your program. Does the evaluator communicate in the proposal a clear understanding of your program's goals and objectives? Does the evaluator understand your program's priorities? For example, if you emphasize retailer edu- cation over school-based instructional programs, the evaluation proposal should note these priorities. 3. How well do the proposed evaluation questions address the program goals and objectives? Evaluation questions are based on program goals and objectives. The most direct evaluation questions are simply objectives written as questions. For example, To what extent have comprehensive tobacco use prevention education programs been implemented in grades 9-12? (implementation assessment). Or, How have community leaders' attitudes toward tobacco changed during the past 12 months? (out- come assessment). 4. How appropriate is the proposed methodology for the evaluation questions? This can be a difficult issue to assess. Selecting evaluation methodologies can be a lot like selecting foods; not only are the needs of the people (program) important, but also their individual tastes. How- ever, the evaluator should clearly explain in the proposal why the methods being proposed are the most appropriate. In general, you will readily see the logic behind using surveys, records reviews, pre/post tests, or other approaches for answering questions about your program. Being intentionally redundant, whatever approach is used should yield information that you can use to make information, management, or policy decisions about your program. If you are not familiar or comfortable with assessing the proposed methodology, then I recommend that you enlist the services of an experienced evaluator to assist you with the review of proposals. An evaluator on the review team will serve two primary purposes: 1) assessing the appropriateness of the methodology; and 2) making recommendations for improving the evaluation methodology, if improve- ments are needed. 5. How reasonable is the timeline of evaluation activities? The reasonableness of timelines depends on several factors, including your program budget period, when you need the evaluation results, and the evaluation methods used. Don't rush the evaluation un- necessarily, but do allow yourself enough time after the evaluation is completed to use the results for your program planning. 4
  • 6. How to Hire the Right Evaluator 6. How relevant and useful are the proposed deliverables? The most common evaluation deliverable is a written report. I recommend that you look for several types of reports, each of which is used for a different purpose. Obviously, the most important reports are the final reports which present the evaluation results and the conclusions and recommendations of the evaluator. To ensure that the evaluation activities are being conducted as planned and within time- frames and budget, you should get monthly updates (these generally are brief reports). If the method- ology includes a pretest and posttest, then I recommend that you ask for a report of the pretest results as well as a report of the analysis of both the pretest and posttest. If the evaluation uses procedures that build on one another (for example, conducting a focus group and using the results to develop a survey) then a report on each component will be useful. For evaluations being conducted over a long time period, a comprehensive interim report might be useful. Helpful Hint: The larger the scope of the evaluation, or the more complex the methodology, the more there will be to report; it is commonly easier to digest several shorter and more focused reports than one large, detailed report. 7. How reasonable is the proposed budget? Does the candidate believe your evaluation can be conducted for the available funds? Candidates must indicate that their proposed evaluation approach can be carried out for the funds you indicated would be available (or less!). You might find that a proposed evaluation plan is excellent, but unfeasible under your anticipated budget. Interviewing the Applicants The second step in selecting from among the applicants is to conduct formal interviews with the top three candidates. The interviews should be used to clarify any concerns or questions raised during the proposal re- view and to gauge evaluators' ability to work with your program. Request that each candidate interviewed provide you with previous evaluation reports they have written and with references of people or organizations for whom they have previously conducted evaluation projects. 8. How useful are the candidate’s previous evaluation reports? Look for evaluation reports for which the candidate served as lead author or as project direc- tor/manager (typically the second author in a report from a research institute). Assess the reports for clarity, organization, readability, and potential usefulness for decision makers. Pay particular attention to how well the reports would help a program improve its information collection, activities, or objec- tives. Candidates providing technical, poorly written, disorganized, difficult-to-understand, or lengthy evaluation reports will likely compose similar reports for your evaluation. 9. Does the candidate have good references? References should be contacted to obtain their experiences with and opinions about the candidates. Some questions you might ask the references include: a. How well did the evaluation approach used by the evaluator address the needs and desires of your organization? b. Was the evaluation conducted in a timely fashion? c. Was the evaluation conducted within your budget? d. What was the most useful aspect of the evaluation report? What changes, if any, did you make as a result of the evaluator's recommendations? e. Would you hire the evaluator to conduct another evaluation for you? 5
  • 7. How to Hire the Right Evaluator The actual interview might begin with the evaluators presenting their proposed evaluation plan. Such pres- entations provide you with much invaluable information about the candidates. You will learn about the evaluat- ors' ability to organize information, communicate ideas, and respond to questions. Invite your staff, supervi- sors, and program partners to the presentation to get their input. Remember: they also will be using the re- sults. 10. How effectively does the evaluator communicate with you and other stakeholders? If the discussion becomes very technical and a candidate is unable to present information in an easily understood manner, it is unlikely that this candidate will meet your needs. A candidate unable to com- municate effectively at this time will probably not overcome the problem during the evaluation. Effec- tive communication is a key for success, and the interview gives committee members a good idea of how effectively a candidate can communicate. In addition to whatever questions might arise during the presentation, the following questions might also be useful. 11. What is the candidate's prior evaluation experience? Experience is an important factor to consider. A candidate probably will not have performed exactly the same evaluation that you require, but many similarities between previous programs and your own can be found. A candidate that has performed evaluations of substance abuse prevention programs, for ex- ample, would have experiences that translate directly to tobacco prevention and control programs. De- pending on the emphases of your program, evaluation experience with law enforcement programs, school or community based educational programs, marketing, or community building might be relevant experience. The candidate's prior experience will be your main opportunity to discover and weigh that person's strengths and weaknesses. 12. Will the candidate's existing professional commitments interfere with the planned evaluation? Good program evaluators are usually in high demand. A candidate who is engaged in several projects, however, may be unable to devote sufficient time to your program evaluation. However, you should expect that the quality of work you receive is comparable to any other project in which an evaluator is engaged. Ask the candidate to describe current and expected professional commitments. If the commitments seem excessive, ask how the candidate plans to conduct your program evaluation along with these other tasks. If the candidate indicates that other persons will be used to assist with the evaluation, determine which tasks win be performed by whom. Also determine if these other persons are capable of performing the tasks assigned to them, including requesting copies of resumes or vitas from the candidate's associates. Using a team of trained and experienced persons to perform an evaluation is common, but the team leader (i.e., the candidate) should be involved in all tasks that the you believe require this person's direct involvement. Finally, it is important to ensure that the evaluator you select does not have a conflict of interest. An evaluation is truly independent when the program evaluator has no stake in the program’s success or failure. For example, if the evaluator were a member of the management team, then he or she would obviously have a clear interest in the program results – both in terms of prestige and continued em- ployment. Such an interest in the program outcomes can affect the decisions of event the most honor- able evaluators. A conflict of interest questionnaire is included at the end of this guide that you can ask all evaluators to complete and include with their proposal. 13. What is your general reaction to the candidate? During interviews, be alert to the candidate's ability to communicate in a straightforward manner, and be alert to your own expectations of how effectively you and your colleagues can work with this per- son. A clash of working styles can certainly be a problem, and the chemistry between a candidate and your program's stakeholders should not be ignored. 6
  • 8. How to Hire the Right Evaluator Following the interview process, reviewers should individually rate the candidates on all of the issues previ- ously described. Candidates might be rated on a five-point scale, ranging from "Definitely hire as our evaluator" to "Definitely do not hire as our evaluator." "No opinion" should be the midpoint. (See Attachment 1 for a sam- ple form for rating candidates.) Reviewers scores can be combined in a number of ways, from simply adding up the total points for each criterion, to averaging the points for each criterion, to asking the reviewers to discuss their individual ratings and then reach consensus on how a candidate should be rated on each criterion. After combining reviewers' individual scores, the candidates should be ranked so that the project can be offered to the candidate most acceptable to the reviewers. Negotiating and Writing a Contract The desired relationship between the evaluation committee and the external evaluator is one of partnership and should be reflected as such in the contract. The contract should state, in a single paragraph if possible, the evaluator's general responsibilities. If you have the evaluator produce a work plan, it can be incorporated into your agreement. The contract should list the contract deliverables and provide a timetable for each one. Con- sider including language describing how changes in the scope of work or work plan will be handled. Many evaluation contracts also specify who owns the data gathered during the evaluation, as well as who has the right to publish the results of the evaluation study. Finally, indicate how the evaluator will bill for services ren- dered and include a schedule of payments. It is common to withhold between twenty and thirty percent of the evaluator's fee until all deliverables have been submitted and reviewed to ensure that the deliverables respond to the contracted scope of work. The contract should also detail the program's responsibilities. These responsibilities might include providing the external evaluator with timely and appropriate guidance and reviewing and approving evaluation instru- ments and documents in a timely and constructive manner. Your responsibilities might also include providing the evaluator with program appropriate program records and other information, and assisting the evaluator in solving problems that arise during the evaluation. Guideline 3: Work closely with the evaluator. Successful evaluations are the result of good partnerships. Your involvement − and that of other program stakeholders − is essential to the success of the evaluation. Your involvement in the evaluation proc- ess should not be limited to periodic meetings with the evaluator. The greater your participation in all of the various aspects of the evaluation, from planning to final report, the more complete the evaluation will be, and the greater your understanding of the evaluation results and recommendations will be. You should continually monitor the evaluator and the evaluation. Review all major work elements, including sampling plans, instrumentation, data collection plans, etc. You and your staff and partners can be of consider- able service to the evaluator by facilitating involvement with host sites, service recipients, and others from whom the evaluator wishes to collect information. Solicit feedback from your evaluator as the study progresses. Informal insights gained during the study pe- riod can be valuable. Finally, manage the process actively to make sure you get what you need. A committee that is intermittently involved in the evaluation process may discover that the study has gone in an inappropri- ate direction after considerable time, money, and effort have been expended. As your evaluation nears completion, you and the evaluator should agree on a format for the evaluation re- port. The evaluation report should address evaluation questions directly and briefly and should be understand- able to the target audience. Any report, of course, must provide useful and direct guidance for program deci- sion makers. You and the evaluator should also agree at this time on the evaluator's role in the release of the evaluation's results. You might request that the evaluator be available to meet with decision makers, conduct interviews with news media, and make public presentations of the results. Be involved in the development of the final report. This is not to say that you decide what the evaluator re- ports; the point of an external evaluation is to get an independent assessment your program. However, your knowledge of the target audience will be valuable in framing the report, and your knowledge of policy makers' 7
  • 9. How to Hire the Right Evaluator future plans for the program will help set the context for any recommendations made by the evaluators. You should carefully review a draft of the final report and provide detailed, written feedback to the evaluator to use in developing the final product. A cautionary note: If the changes suggested to the evaluator are significant and would have the effect of changing the findings, recommendations, or overall focus of the report, they should be discussed at a meeting with the evaluator. If the evaluator does not believe that the suggested changes are consistent with the data, the evaluator has the right (and an ethical obligation) to be disassociated from the report. In such an instance, the evaluator may make the requested changes and assign authorship of the report to the program, or may hand over all work to the program to complete the report. Such situations should be avoided, however, as they usually place the integrity and public acceptance of the report in jeopardy. Guideline 4: Use the evaluation results. Once your study is complete, its value will be very limited unless the results are used to improve your pro- gram. Use the insights gained to review your program design and modify, or replicate, program elements as appropriate. Use your evaluator as a resource in this activity. Your evaluator's assistance in interpreting the results for their program management implications can be very valuable. Address the most compelling findings first. Don't get overwhelmed trying to do everything at once. Make note of findings that suggest further areas of investigation. Future evaluations may shed light on unclear data. Use your study not only to improve operations, but to market your program. Potential funders and service partners will be more willing to become involved with your program if you can show them evidence of effec- tiveness and a willingness to make improvements based on evaluation data. Report the results of your evaluation not only to your staff and other program stakeholders, but also to fi- nancial contributors, key community leaders, and organizational partners. Contact your local newspaper about the possibility of doing a feature article on your program. Have program participants − especially youth mem- bers − develop exhibitions including the evaluation results for use in community presentations and other activi- ties. Present your results at professional conferences and other "issue networks" (groups with a common inter- est). Conclusion A program evaluation properly designed to provide interested individuals with information that they will use will likely be a successful evaluation. However, while utilization of the results of the evaluation is clearly the primary goal of an evaluation, important benefits may also be gained from the evaluation process itself. The process of undergoing an evaluation can, for example, build shared meaning and understanding, support and enhance the program (by building evaluation-based data collection and analysis into the program design), and support human and organizational development by training staff in new skills. A well conceived evaluation will provide benefits no matter the results. A well-conceived evaluation results from a good working partnerships between program stakeholders and evaluators. An effective functional partnership is founded on agreement on the objectives of the evaluation, an understanding of the responsibilities and authority of each partner, and mutual respect for the contributions that each partner provides to the evaluation. If the steps identified here are followed, the evaluation process should be enjoyable and productive for both the program and the external evaluator. 8
  • 10. How to Hire the Right Evaluator Evaluator Rating Guide Candidate Name:_______________________________________ 2 1 0 -1 -2 1. How well does the evaluator under- Clearly understands the No Opinion Has no understanding of stand the focus of the evaluation? focus, including the dis- the evaluation focus or tinction between evalua- the distinction between tion and research evaluation and research 2 1 0 -1 -2 2. How well does the evaluator under- Has an excellent under- No Opinion Has no understanding of stand the program goals and objec- standing of program program goals and ob- tives? goals and objectives jectives 2 1 0 -1 -2 3. How well do the proposed evaluation Evaluation questions fully No Opinion Evaluation questions to questions address the program goals address program goals not address program and objectives? and objectives goals and objectives 2 1 0 -1 -2 4. How appropriate is the proposed Appropriate: can clearly No Opinion Not appropriate: cannot methodology for the evaluation see that the results ob- see how results obtained questions? tained will be useful might help program 2 1 0 -1 -2 5. How reasonable is the timeline of The evaluation can be No Opinion The evaluation cannot be evaluation activities? conducted well within the completed within the proposed time frame proposed time frame 2 1 0 -1 -2 6. How relevant and useful are the Very useful for program No Opinion Not useful for program proposed deliverables? improvement improvement 2 1 0 -1 -2 7. How reasonable is the proposed Evaluation is very likely No Opinion Evaluation can't be con- budget? to be completed within ducted with available the available resources resources 2 1 0 -1 -2 8. How useful are the candidate’s pre- Previous reports were No Opinion Previous reports were not vious evaluation reports? understood and used understood or used 2 1 0 -1 -2 9. Does the candidate have good refer- Received excellent No Opinion Received poor references ences? references 2 1 0 -1 -2 10. How effectively does the evaluator Communicates effectively No Opinion Communicates effectively communicate with you and other with most stakeholders with few stakeholders stakeholders? 2 1 0 -1 -2 11. What is the candidate's prior evalua- Has extensive experience No Opinion Has limited evaluation tion experience? evaluating similar pro- experience grams 2 1 0 -1 -2 12. Will the candidate's existing profes- Commitments will not No Opinion Commitments will inter- sional commitments interfere with interfere with evaluation fere with evaluation the planned evaluation? 2 1 0 -1 -2 13. What is your general reaction to the Definitely hire as our No Opinion Definitely DO NOT hire candidate? evaluator has our evaluator 9
  • 11. How to Hire the Right Evaluator Evaluator Conflict of Interest Questionnaire Name of Program:_______________________________________ Yes No 1. Do you, your immediate family, or business partners have financial or other interests in the program to be evaluated? 2. Have you been employed by the program within the last 24 months? 3. Do you plan to obtain a financial interest, e.g. stock, in the program? 4. Do you plan to seek or accept future employment the program? 5. Are there any other conditions that may cause a conflict of interest? If you answered Yes to any of the above questions, please provide a written explanation of your answer. I declare all of the above questions are answered truthfully and to the best of my knowledge. signature date 10