This document describes a methodology used to evaluate innovation projects for an electric power company in Brazil. It uses multi-criteria decision making (MCDM) and the analytic hierarchy process (AHP) to establish criteria and weights for evaluating projects. Key criteria included originality, applicability, relevance, and cost reasonableness. The methodology decomposed the evaluation hierarchy and used pairwise comparisons to determine relative importance of criteria. This allowed establishing priority scores for projects and simulating different evaluation scenarios. The results provided an effective tool to support the company's evaluation and selection of research and development projects.

1. Using Multi-Criteria Decision and knowledge representation
methodologies for evaluation of Innovation Projects
Tania C. D. Bueno1,a, Claudia de O. Bueno1,b, Angela I. Zotti1,c,Vinicius
Mirapalheta1,d , Thiago P. de Oliveira1,e, Hugo C. Hoeschl1,f
1
Instituto de Governo Eletrônico, Inteligências e Sistemas – i3G Florianópolis – SC Brasil
a
tania.bueno@i3g.org.br, bclaudia.bueno@i3g.org.br, ciara.zotti@i3g.org.br,
d
vinicius.mirapalheta@i3g.org.br, ethiago.paulo@i3g.org.br, fhugo.hoeschl@i3g.org.br
Keywords: Analytic Hierarchy Process (AHP), knowledge representation, multicriteria decision,
projects evaluation, decision making.
Abstract. The performance of R&D sector has a high importance degree as a competitive
advantage in energy market, so there is the application of a large amount of investments which
causes a great number of offered projects. Therefore, it has been necessary to propose a set of
criteria in order to help on R&D projects prioritization for Electric Power Companies in Brazil. The
normalization of criteria for projects selection on the Innovation field will guarantee quality on the
promotion of technological innovation in electric sector, which has a fundamental role on the
integration among economic, social and environmental factors inside the company. This work used
aspects of knowledge representation methodologies and multicriteria decision problem to support
the process of quality measurement of R&D Projects.
Introduction
The evaluation process for R&D projects is the main bottleneck for the Electric Energy institutions
in Brazil because of its subjectivity and lack of relevant and adequate criteria for each situation. To
help on this demand, the Electric Energy National Agency (ANEEL) [1,2] has defined some
criteria, so that the companies in electric sector could evaluate R&D projects under legal Acts.
These criteria, however, are too broad and subjective, becoming difficult the creation of patterns for
the evaluation process for the electric energy companies. The absence of solid criteria comes out
with delays, and in many cases, might cause the selection of inappropriate projects.
The selection problem of R&D projects has being seen as a multicriteria decision problem. In
fact, due to a number of criteria ranging from financial investments and materials to possible
competitive advantages, we intend to prioritize projects according to their importance, to be
implemented at first moment [3]. On every evaluation system it is fundamental the correct selection
of criteria, relevant and adequate for each situation. The normalization of such criteria – for
innovation project selection – will guarantee quality on the promotion of technological innovation
in electric sector, which plays a key role for the integration of economic, social and environmental
factors inside Electric Power Companies. Therefore, a project evaluation model was developed.
This model used aspects from Albertyn [4,5], Saaty [6,7] and Bueno at al [8,9] to support the
process of quality measurement of R&D Projects. In order to turn this process evident, this work
was divided on the following sections: on Introduction is presented the subject of the present work,
followed by the presentation of the R&D projects evaluation process. The third item explains the
methodology. The fourth and fifth items present the Results and Conclusion, respectively. Finally,
the sixth item presents the References consulted for this study.
The R&D Projects Evaluation Process
The Brazilian Electric Energy National Agency – ANEEL, in initial evaluation, defines that the
proposed Research and Development (R&D) projects should be evaluated solely based on the

2. content of the XML file based on predefined criteria. In this criterion, the state of the art, challenges
and proposed advances in scientific and/or technology terms must be analyzed, considering the
main product of the project. The problem to be solved and the absence or the high cost solution
available in the market must be considered when is relevant. Since the law creation, the electric
power companies have difficulty in sustaining a significant portfolio of investments in innovation.
Among the main factors is the delay in approving the projects by ANEEL [1,2].
Criteria by ANEEL. The criteria established and adopted by ANEEL are described in sequence,
which currently parameterize the selection and evaluation process of the companies. The evaluation
of a R&D project is performed based on the criteria: Originality, Applicability, Relevance and
Reasonableness of Costs and the main parameters analyzed in each criterion. The projects are
evaluated at the end, necessarily, and there is also the possibility of an initial evaluation, if
requested, not obligatory though. Nevertheless, the initial evaluation is highly recommended in
order to "evaluate the framework of the project as R & D activity, its relevance to the technological
challenges facing the sector and the reasonableness of the planned investments in the face of
expected results and benefits".
Currently, the electric power company submits the project proposal to the initial evaluation of
ANEEL for analysis of the following criteria: Originality, Applicability, Relevance and
Reasonableness of Costs.
Criteria by the Electric Power Company. The project proposal is reviewed by the Company
area where the knowledge to be gained in the research, and/or the potential outcomes may be used.
If there is a favorable analysis by the area, the R&D Commission reviews the project proposal,
based on parameters and evaluation criteria provided in the Research and Development Guide
Program. If the Commission considers relevant, the proposals coordinators are invited for a face
presentation [9].
After this step and after the Regulatory Agency (ANNEL) initial evaluation, the R&D
Commission establishes a conclusive record containing a list of the projects considered "able" to
join the R&D Program of Energy. Projects can only be considered "able" if they answer the
following premises:
a) To obtain at least the concept “3” in Originality criterion given by ANEEL;
b) To achieve grade equal or higher than 2.8, the average applied to the concepts that guide the
project (Originality, Applicability, Relevance and Reasonableness of Costs);
c) The favorable assent of the R & D Commission. In final evaluation in case of inadequacy as
an R&D activity the project is disapproved. The Originality criterion has eliminatory character, so
the project must have grade equal to or higher than three (3) and still be characterized as R&D
project.
Otherwise the value of the project should be reversed for the account of R&D [10].
Methodology
The structure of the Electric Power Company´s R&D Management System allowed the inclusion of
an evaluation process by the following organization. This organization is a result of the addition of
a meta-model proposed by Albertyn [4,5]. The reason for joining was to realize the occurrence of
some classes of the metamodel that present connections with classes pointing to quality aspects
adapted to the Electric Power Company R&D Project. Before starting the analysis, we defined the
purpose and expected outputs from the process.
The knowledge engineering team determined that the objective pursued by the application of the
model was to identify a set of questions and expressions which could fit the criteria set by ANEEL
through knowledge engineering methodology proposed by Bueno at all [8,9] where the definition of
the relevance of the expressions is related to frequency of use of these expressions in context.
Set the goal of the analysis - it was the choice of the projects that were using the process - we
chose to work on projects that were running. In the case of a system of Multi-Criteria Decision to

3. formalize and validate the process of evaluation and selection of projects aimed to a practice that
was structured in line with the Mind Engineering Methodology® [8]. The MCDA worked by
Albertyn [5] is a method to support evaluators who are faced with many different and conflicting
solutions to a problem.
Deriving this method, we chose to focus the evaluation on the application of the AHP, because
the method is based on the innate human ability to make judgments about different problems
[11].The method is characterized by decomposing the problem into descendant hierarchical levels,
starting with the overall goal, criteria, subcriteria and alternatives in successive levels, until it
reaches a prioritization of its indicators, approaching a better response. The AHP has the advantage
of allowing comparison of quantitative and qualitative criteria. At each stage (criteria and
subcriteria) pairwise comparisons are made to determine the relative importance of each criterion to
reach the goal. After this hierarchical structure phase, AHP includes other two important steps: the
judgment of value and priority, where the evaluator establishes a peer comparison of elements of
the various hierarchical levels, prioritizing them and in sequence, the analysis of consistency of
these trials. This pairwise comparison is done according to Saaty Scale [6,7] that allows the
conversion of the analysis on a scale from 1 to 9, ending in an array for each level of criteria and
sub-criteria, showing the result of the comparisons made in pairs.
Through pairwise comparisons, the priorities calculated by AHP capture subjective and objective
measures and demonstrate the strength domain of a criterion over another or one alternative over
another. Through synchronization meetings, the teams began the process of knowledge engineering,
with the task of standardizing the language that would apply in the development of Knowledge-
Based System - KMS. We identified the main concepts used in the domain of evaluation and
selection of R&D projects and their understanding and determination of national and regional
context of the procedures adopted so far the Electric Power Company. Prospecting criteria, in
agreement with the criteria established by ANEEL, and definition of the Energy Power Company
strategic criteria were issues discussed.
The elements used by the domain experts, knowledge engineers and programmers could pass on
their knowledge in a better structured way in order to build a KMS based on the Mind Engineering
Methodology® [8]. Inventories of contents, processes and people were conducted following the
premises of knowledge sharing, visualization and definition of relevance. It is the synchronization
of these factors that enables the knowledge understanding or expertise in a particular field. There is
a need to change the formal model of innovation management in the Electric Power Company to
support and become innovation more effectively within the parameters of ANEEL and thus
adapting it to the national context.
To do so, the innovation management models from other companies in power sector were
studied and the proposal of a new model came out from the synergic teamwork. This process took
two months and four meetings were held for sharing knowledge among six knowledge engineers
and three domain experts.
Results - AHP Application for R&D Evaluation
While inventories allowed the selection of the questions available to the R&D Projects Evaluation
System and therefore the adoption of criteria to be evaluated and prioritized, the application of AHP
method allowed calculating the weights for each alternative of evaluation project in relation to the
proposed criteria. At this stage, based on the questionnaire already prepared, we performed pairwise
comparisons using the intensities of importance by Saaty Scale. The pairwise comparison followed
the next hierarchy: pairwise evaluation among criteria, pairwise evaluation among questions within
the criteria, pairwise subcriteria evaluation. Thus, the intensities of importance were converted into
numbers, according to that scale. The test calculations were performed in Excel software. The
following tables present the values resulting from the evaluation of researchers and the matrix
obtained from calculations that make up the method.

4. Pairwise evaluation among criteria
Figure 1 represents the comparison among all the criteria (Originality, Applicability, Relevance and
Reasonableness of Costs). According to the data presented the Originality criterion is the one that
has a greater impact over the other criteria defined by the company. This result quantifies the
guidance established by the energy national agency and the extreme importance of the originality
criterion for the evaluation and selection of R&D projects.
Fig. 1. Matrix with the pairwise criteria evaluation
Pairwise evaluation among questions within the criteria
Figure 2 presents the comparison among the questions that form the Originality criterion. It shows
the “Unpublished” as the most important.
Fig. 2. Matrix of Pairwise evaluation of alternatives within the criteria Originality
Pairwise Subcriteria Evaluation
The alternatives inside each question of each criterion were considered as subcriteria. Figure 3
presents the result of pairwise comparison made among the subcriteria in question 4 of Originality

5. criterion that evaluates the innovative character of the proposed project (Innovativeness). The most
relevant subcriterion was “It is well defined and innovative, generating new product, process and
methodology”.
Fig. 3. Matrix of pairwise comparison for the subcriteria in question 4 (Originality)
Conclusion
This methodology allowed the identification of the project that best meets the criteria defined by the
company, considering the different weights for each alternative. As main result it produced a tool
that allowed simulating scenarios depending on the questions to evaluate R&D projects.
Every new scenario allows using the same methodology to support decision making in other
situations, specific to each edict or company's proposal. It means that depending on the proposal,
another criterion as Relevance can be considered the most important. Therefore, each scenario
created can permit the use of the methodology applied in this study as support for decision making
in other circumstances.
Thus, for each new scenario will be necessary to compare the questions according to each edict
adopted, calculating the weights again and subsequently, ranking criteria, and producing a new
result.
Since ANEEL´s R&D Program is recent, results from this research have contributed effectively
for the Electric Power Company intention, on the challenge for an evaluation and selection of
submitted projects with good quality, about a task previously assumed by ANEEL.
Although there is already a model for textual knowledge representation inside this Electric
Power Company´s knowledge base, this base is not filled with enough documents for
experimentation. In future jobs, this knowledge base will be completed with a considerable set of
proposed R&D projects, in order to do more tests about knowledge representation by using
ontologies.
Acknowledgments. Our thanks to Centrais Elétricas de Santa Catarina – CELESC, especially to
Celesc Team: Mr. Amaro Koneski Filho, Mr. Luiz Afonso Pereira Athayde Filho e Mrs. Débora
Simoni Ramlow. Special thanks to i3G Team: Mrs. Alessandra Zoucas and Ms. Angela Iara Zotti
who helped us to finish this job.

6. References
[1] ANEEL - Agência Nacional de Energia Elétrica (2002). Atlas de Energia Elétrica do
Brasil.ANEEL, Brasília: ANEEL, 2002. Retrieved
from:<www.aneel.gov.br/arquivos/PDF/livro_atlas.pdf>
[2] ANEEL - Agência Nacional de Energia Elétrica (2010). Guia do avaliador de projetos de
P&D:Programa de Pesquisa e Desenvolvimento Tecnológico do Setor de Energia Elétrica.
Brasília:ANEEL, 2010. 30f.
[3] A. P. H Gusmão. Modelo Multicritério para Priorização de Portfólio de Projetos de Pesquisa e
Desenvolvimento. Recife, 2005. 101p. Dissertação (Mestrado em Engenharia de Produção) –
Universidade Federal de Pernambuco.
[4] F. Albertyn. E-Process Selection using Case-Based Reasoning Techniques. IADIS international
conference E-Society 2007. Lisbon, Portugal. (http://www.esocietyconf.org/) part of the IADIS
MultiConference on Computer Science and Information Systems (MCCSIS 2007) Lisbon,
Portugal, 3 – 8 July 2007 (http://www.mccsis.org).
[5] F. Alberty. A Decision Making Approach: a meta-model for e-process selection. In: IADIS
International Conference e-Commerce, 2010, Freiburg, Germany, 26-30 July. ISBN (Book):
978-972-8939-24-3.
[6] T. L. Saaty, (2005). Theory and Applications of the Analytic Network Process: Decision
Making with Benefits, Opportunities, Costs, and Risks. Pittsburgh: RWS Publications.).
[7] T. L. Saaty, (2009). Extending the Measurement of Tangibles to Intangibles. International
Journal of Information Technology & Decision Making, Vol. 8, N. 1, p. 7-27, 2009. Available
at SSRN: http://ssrn.com/abstract=1483438.
[8] T. C. D. Bueno , H. C. Hoeschl , A. Bortolon , R. M. Barcia. Engineering of Minds: The
syncrhonicity between Artificial Intelligence and Management of knowlegde in collaborative
networked organazations. Conference Proceedings IADIS International Conference
WWW/Internet - ICWI IADIS International Conference WWW/Internet 2004. ISBN: 972-
99353-0-0.
[9] T. C. D. Bueno, C. Stradiotto, S. Bedin, V. Mirapalheta, H. Hoeschl, T. Oliveira, L. A. Filho, P.
A. Junior. Case-based System for Innovation Management in Electric Power Enterprises. 2nd
Workshop on Applications of Knowledge-Based Technologies in Business (AKTB 2010) in
13th International Conference on Business Information Systems (BIS 2010). Berlin, Germany.
May 3-5, 2010.
[10] Celesc (2010). Centrais Elétricas de Santa Catarina S.A.. Relatório Anual de Responsabilidade
Socioambiental 2010. CELESC. 64f.
[11] J. R. Grandzol. Improving the Faculty Selection Process in Higher Education: a case for the
analytic hierarchy process. IR Applications, v. 6, 13 p., 2005. Available in:
<http://airweb.org/images/IRApp6.pdf>