1. Junior Project Management course
13-15 April 2011
Budapest, Hungary
Evaluation of Project Proposals
Ms. Gabriella Lovasz
Assistant Managing Director
Europa Media

2. Selection of the Experts – Who?
Who are the Evaluators?
Balance between
• Academic and industrial expertise
• Geographical representation of Europe
• Gender
Expertise in the following fields
• Research in the given scientific area
• Industrial/commercial applications
• International project design and implementation
• Administrative, policy-oriented or other
knowledge as required by the Call
Register as an
”expert”:
https://cordis.europa.
eu/emmfp7/

3. The Evaluation Process
Proposal
Administrative check by the EC
Evaluation by the experts:
Eligibility
Briefing for the experts Pre-check by
Commission
Individual evaluation
Consensus evaluation
Final ranking Individual Individual Individual
evaluation evaluation evaluation
Contract negotiations start
Consensus meeting
Scores, ESR
Tresholds
Ethical
review if
needed
Panel review
with
Hearing (optional – IP/NoE)
Applicants informed of
results
ESR
Commission
Ranking

4. On-site vs Remote Evaluation
On-site Remote
Confidentiality Applies Applies
Controlled No control
Payment 450 EUR/day 450 EUR/day
Time available for Limited Open (but maximised by
individual evaluation payment)
Access to extra resources No access Unlimited
(Internet)
Stay in Brussels Fully Only for the consensus and
panel meeting
Benefit Controlled confidentiality Background check can be
Better focus done – better
understanding

5. Briefing for the Evaluators
Introduction to the Work Programme objectives
Introduction to the Guide for Applicants and formal criteria
Overview on the general evaluation principles, and the details of the evaluation
process
Overview on the scores and the definitions of „Excellent” and „Poor”, etc.
Introduction to any special EC priorities/issues
Clarify the possible “conflict of interest”
Discussions
Evaluation principles:
Excellence
Transparency
Confidentiality
Fairness & Impartiality
Ethical and Security considerations
Efficiency and Speed

6. Individual Evaluation
The evaluators work independently
• No communication is allowed
• Computer is provided if needed (personal laptops are not allowed)
• The internet might be used (depending on the programme)
Each proposal is evaluated by 3-5 evaluators
Each evaluator fills out the „Individual Assessment Report” (IAR) forms

7. Individual Evaluation
2-4 proposals per day
Max 2 hours per proposal
First impressions
Title
Summary
Objectives
Partnership
Consistency & formatting
Length
First 15-30 minutes of the evaluation are crucial

8. Evaluation Criteria
„Within scope” assessment Detailed reading
First Reading = First impression Read specific chapter for the
= ABSTRACT, OBJECTIVES, specific criterion (find it)
Partner list Make rough notes
Decide score
Complete IAR
Technical issues
Management
Finances/Resources
Impact

9. Evaluation Criteria - CP
From the Guide for Applicants:
Evaluation criteria applicable to
Coordination and support actions (Supporting)
S/T QUALITY IMPLEMENTATION IMPACT
“Scientific and/or technological “Quality and efficiency of the “Potential impact through the
excellence implementation and the development, dissemination
(relevant to the topics addressed by management” and use of project results”
the call)”
• Soundness of concept, and quality • Appropriateness of the • Contribution, at the European
of objectives management structure and [and/or international] level, to the
• Progress beyond the state-of-the-art procedures expected impacts listed in the work
• Quality and effectiveness of the • Quality and relevant experience of programme under the relevant
support action mechanisms, and the individual participants topic/activity
associated work plan • Quality of the consortium as a • Appropriateness of measures for
• Appropriate comparative perspective whole (including dissemination and/or exploitation of
in relation to the proposed research complementarities, balance) project results, and management of
• Appropriateness of the allocation intellectual property.
and justification of the resources to
be committed (staff, equipment, …)

10. Individual Evaluation Report
In FP7: a 4-5 page document completed by each expert
I. Evaluation summary
II. Recommendation
Criteria:
S/T QUALITY “Scientific and/or technological excellence
(relevant to the topics addressed by the call)”
IMPLEMENTATION “Quality and efficiency of the implementation and the management”
IMPACT “Potential impact through development, dissemination and use of project results”
Criterion Threshold Max
Scientific quality 3 5
Implementation 3 5
Impact 3 5
Total 10 15

11. Marking System
Scores
0-5 points awarded for each main
0 Fails to address the criterion or cannot be
and sub-criteria (0-Fail/Poor, 5-
judged
Excellent)
1 – 1,5 Poor.
Half points can be given! There are serious weaknesses
Detailed explanation required
2 – 2,5 Fair.
• should be given by the
Broadly addresses the criterion, but ….
evaluators, not only simple
scores
Thresholds for each main criterion 3 – 3,5 Good.
Addresses well, but improvements are
Threshold for the overall mark necessary
This can be more than the sum of
4 – 4,5 Very good.
the thresholds of the sub-criteria! Certain improvements are still possible
5 Excellent.
Any shortcomings are minor

12. Individual Evaluation - Results
Total score, overall comments
• for the whole project (strengths and weaknesses, overall recommendations)
Qualification
• recommended or not recommended for funding – each threshold must be reached!
The amount of the requested funding is also evaluated: realistic or too much, should
be reduced, adequate to the work plan, etc.
When all experts have finished, the proposal is ready for Consensus evaluation

13. The Consensus Meeting
Experts meet together to come to a consensus
Minutes of the meeting are recorded by the „Rapporteur”
The Commission representative is the „Moderator”
Experts have a few minutes to read and understand each others comments
Preliminary discussions followed by detailed assessment of all criteria

14. The Consensus Meeting
Roundtable discussion on each criteria:
• The higher/lower marks of an expert have to be explained and justified
• Discussion. Arguments - counterarguments
Consensus must be obtained for
• Scores for each main criterion (the consensus score is NOT the mathematical
average of the individual evaluators score)
• Text of explanation and justification
The scores of all criteria + the corresponding text agreed upon: IT IS A
CONSENSUS
Consensus Report – drafted by the “Rapporteur”

15. If There Is No Consensus:
1. New evaluator(s) can be appointed:
Individual evaluation
2. Extended Consensus Meeting with the involvement of the new
evaluators
3. If there is no consensus afterwards:
Decision by majority of votes

16. Goals of the Panel Meeting
To assess and compare the consensus reports of the different sub-panels:
Overall quality under the different topics
Number of proposals evaluated
Special attention to
Proposals that scored very high but failed on one criteria
Proposals with equal score near the funding thresholds
Final ranking according to scores:
Established for each topic
Established for the programme as a whole
Proposals with equal scores will be ranked:
Objectives, Relevance, Impact
Horizontal issues

17. Final Decision, Negotiation
Commission prepares the following lists:
• Ranking list of the eligible proposals (over the thresholds)
• List of those proposals for which the negotiation process can start – taking
into account the available total fund for the given panel
• Reserve list
• in case of withdrawal – if the negotiation is not successful from any
side!
• List of rejected proposals

18. Thank you for your attention!
info@europamedia.org
© Europa Media
It is not allowed to use or distribute the content and design
of the presentation without prior agreement.