This document describes a system called Indiopedia that uses crowdsourcing games to acquire legal knowledge from non-experts to expand lightweight ontologies in specific domains. Indiopedia builds on an existing legal knowledge system called Ontojuris that uses ontologies to improve legal search, but ontology building is time-consuming. Indiopedia addresses this by developing crowdsourcing games that turn the ontology expansion process into fun games for non-experts to play and contribute knowledge. The games aim to efficiently acquire semantic structures from many users to develop the legal ontologies in a scalable way. Initial experiments involved users playing games to help expand ontologies related to indigenous legislation.

1. Indiopedia: a System to Acquire Legal Knowledge from the Crowd Using Games
Techniques
Tania C. D. Bueno, Karina Girardi Roggia, Angela Iara Zotti, Hugo C´esar Hoeschl
Institute of Electronic Government, Intelligences and Systems
Florian´opolis, Brazil
{tania.bueno,karina.roggia,iara.zotti,hugo.hoeschl}@i3g.org.br
Abstract
This paper describes a new way to expand
lightweight ontologies in specific domains getting
help from the crowd in the internet. More specifi-
cally, we had to construct an ontology concerning
indigenous legislation: a domain very strict where
the corpus is restricted. To solve this, we pro-
pose a system to get from non-specialists in knowl-
edge engineering (but that know about the indige-
nous area) working with gaming techniques to ac-
quire the desired knowledge needed to improve le-
gal search systems.
Keywords: crowdsourcing, knowledge engineer-
ing, legal ontologies, gamification.
1 Introduction
How to compile a knowledge domain that is know only by
few people and they are not necessarily in the same place? In
a projet dealing with the improvement on the legislation ac-
cess to indigenous people and specially trying to map indige-
nous law, our team was with this question on mind. From this
problem, started the Indiopedia project.
The search for legal information in the Web requires legal
knowledge and are limited due to problems such as syntactic
and semantic ambiguity, and also the uncertainty existing in
the text of the documents. In practice, these databases often
recover a large number of irrelevant information, and require
a recasting of repeated search to achieve a satisfactory result.
Ontojuris System [Stradiotto et al., 2009] was created to solve
this problem.
Based on ontological representation, the system allows the
organization of legal knowledge representation to improve
the process of recovery and automatic extraction of the at-
tributes of new documents to be included in the database of
cases. Lightweight ontologies are used to improve search ac-
curacy, among others. A knowledge managing system based
on ontologies will be able to retrieve only pages and docu-
ments which are relevant to the user, by considering the con-
text of the research item. Therefore, it may infer that ontolo-
gies are a fundamental part of intelligent retrieval systems
which will search, combine, or integrate information from
various sources. However, ontology building is a time con-
suming activity that requires a lot of effort for domain ac-
quisition. Several methods have been developed to overcome
these problems, including tools that automatically or semi-
automatically allow ontologies generation. Crowdsourcing
games emerged as an alternative to solving computational
problems, hard or impossible to be solved by machine com-
putation (which includes acquiring semantic structures), via
aggregation of knowledge provided by many non-expert users
(e.g., for image annotation) making it more accurate. Crowd-
sourcing games transform problems into games that motivate
players to solve them via fun and thus eliminate the need to
pay for them. As many game instances can be played simul-
taneously, they are suitable for larger scale problems divisi-
ble into smaller tasks [Casanovas, 2012]. Compared to other
crowdsourcing techniques, the knowledge gained in crowd-
sourcing games is not just a by-product of another user activ-
ity (e.g., annotating web resources for personal use), but the
primary objective, so their design is tuned to maximize that
ability. With this in mind, an approach to use crowdsourcing
to ontology expansion came as a possibility to minimize the
issue of slowly, hardworking and expensive legal ontology
building. In addition, the most important is the disruptive im-
pact that this type of approach emphasizes.
So, from the Ontojuris system, we developed Indiopedia
with the purpose of expand lightweight legal ontologies about
indigenous people with the help of those who knows about in-
digenous questions and/or law but not necessarily know about
knowledge engineering. Figure 1 resumes the architecture of
Indiopedia system, the upper part works on Ontojuris System
that communicates with our system by a webservice.
1.1 Legal Ontologies
“In computer and information science, ontology is a technical
term denoting an artifact that is designed for a purpose, which
is to enable the modeling of knowledge about some domain,
real or imagined.”[Gruber, 2009]
Legal ontologies can enhance information extraction from
semi-structured and non-structured data, adding a new di-
mension to legal research. One of the most complicated is-
sues is, however, the definition of prior knowledge either from
the process or the domain. Furthermore, the different theo-
ries about the nature of Law may inspire different ontologies
that are highly structured and elaborated, often called heavy-
weight semantics. Though they provide advanced capabili-

2. ties, they are also hard to obtain and are almost exclusively
built by (costly) human experts. The cheaper and more easy-
to-create are the lightweight semantics with lightweight rep-
resentations. These include taxonomies (hierarchical organi-
zations of entities) and free association networks (which ex-
press general relatedness of entities) sometimes referred to as
folksonomies (in case they were created using a crowdsourc-
ing technique) [Casanovas, 2012]. Semantic structures some-
times use simple terms instead of concepts (also due to ease
of creation), which of course, brings drawbacks like semantic
ambiguity. Thus, crowdsourcing games allow environment
of ontology construction in the Crowdsourcing Knowledge
Engineering Suite (see Section 3.1), which allows the acqui-
sition, management and sharing of ontologies. The advan-
tage of crowdsourcing approaches against the expert-based
approaches is a much greater scale of discovered semantics.
We use lightweight ontologies in our system: without ax-
ioms and instances. The necessity of information needed in
order to amplify a legal search is in the classificational level,
organizing the legal knowledge in a kind of taxonomy where
we use relations of synonym, conected term (in a restricted
way), parts and types of a concept. Besides, it is simplier
to start with a taxonomy style ontology for non specialists
in technology or information as anthropologists, for instance,
what is the case of most of experts in the domain of indige-
nous area (the domain that the system were developed). [Cor-
cho et al., 2015] also points out that most of the ontologies
that have been developed or that are being reused so far fall
into the lightweight ontology category.
1.2 Gamification
ICT provided the organization of social platforms that enable
strong interaction between people, but this is not enough to
achieve the purpose of organizing work. Social platforms are
broadened with the use of games to allow sharing of special-
ized content, enhanced by the context of use and the commu-
nities that form around them. The introduction of games can
arouse motivation, engagement and personal satisfaction.
This section contextualizes the use of gamification as a new
element to be considered in the construction of ontologies in
a collaborative platform. For this article we consider the def-
inition of gamification as the game mechanics use in differ-
ent contexts, in order to increase participation and generate
engagement and commitment of potential users. [Juho and
Veikko, 2011]
In general, the application of gamification points to cir-
cumstances involving the creation or adaptation of the user
experience to a product, service or process. It also points to
arouse positive emotions, explore personal skills or tow vir-
tual or physical rewards to accomplish tasks. [Vianna et al.,
2013]
In gamification two terms are closely related: the mechan-
ics of games and dynamic games. The game mechanics are
the various actions, rules and benefits that make up the game,
behavior and control mechanisms that are used to insert game
elements in an activity. They are also the aspects that make it
difficult, fun, satisfying, or any other emotion that is expected
to invoke. [Bunchball, 2010]
These emotions are the result of desires and motivations we
call game dynamics. The dynamics of games is related to the
universal human needs and desires: recognition and reward,
status, outcome, competition, self-expression, altruism, and
other needs.
In summary, the dynamics refer to the purposes and the me-
chanics to the means. The mechanics comprises points, lev-
els and challenges while the dynamics extends the rewards,
status and achievements and keeps people’s interest because
they create an active environment, but it can be short-lived
without the addition of game mechanics [Law et al., 2011].
Table 2 illustrates the interaction of basic human desires and
game play. Dots signify the primary desire a particular game
mechanic fulfills, and crosses show the other areas that it af-
fects.
Figure 2: Game Mechanics and Human Desires
This article is divided on the following sections: Section 1
presents the introduction and important related themes. Sec-
tion 2 gives the background used in the research: Ontojuris
System and Mind Engineering methodology. Section 3 are
about the development of ontologies using crowdsource, pre-
senting the proposed system to ontology expansion through
crowdsourced games and a brief experiment with a few users.
Section 4 has final remarks about this work.
2 Background
In earlier works, we used a methodology called Mind Engi-
neering [Bueno et al., 2004; 2005; 2006] to identify and orga-
nize ontologies using a collaborative web tool called Knowl-
edge Engineering Suite (see Section 2.2). This tool is a mod-
ule of Ontojuris System. Mind Engineering allows building
a knowledge base, improving the construction of the ontol-
ogy of the domain and the automatic representation of cases
in knowledge-based systems, either in the legal area or any
other knowledge management domain. The Ontojuris Sys-
tem provides the knowledge base and the initial ontologies
for this work [Bueno et al., 2014].
2.1 Knowledge Based System - Ontojuris
The Ontojuris framework embraces the whole cycle of strate-
gic information production, from the collection to the re-
covery for the user. The System includes Artificial Intelli-
gence artifacts, which gives best quality and answering speed
through the organization of Law information in form of cases,
and allows recovering information based on similarity mea-
sures. The system is based on ontologies for indexing and
storing the documents inserted on the knowledge base. The
vision of the storage structure is physical, containing the

3. items collected in folders organized by domains. Collec-
tions are converted for a common pattern that allows com-
munication with structured databases. The chosen format
was XML (Extensible Markup Language). Text Mining is
destined to extract concepts, statistics and important words
of a group of documents to minimally structure them. In
that phase, three types of information are extracted: metadata
(common document information, such as title and author), in-
dexes (terms that describe the content of the document) and
concepts (terms that describe the context of the document).
These concepts are based on the ontologies defined in the
Knowledge Engineering Suite. (see Section 2.2). In order to
create those systems, the necessity of a specific tool emerges
the Ontology Editor.
The Ontology Editor intends to improve specialists’ work
in ontology creation. It is a part of Ontojuris System infras-
tructure, which relates complex terms, considering its con-
cept in the specific knowledge, allowing the application to
recognize context. Applying semantics in information search
tools is essential, however, it is verified that the development
of such ontology construction tools is still shy. There is few
ontology editors based on web semantics showed in publi-
cations and systems, emphasizing semantics rather than con-
text. Prot´eg´e is an example, using OWL, RDF, RDFS and
XML languages. Ontology specification in this editor type is
presented in a complex way, requiring the specialist to master
the application, with knowledge on classes, subclasses and
attributes which allows the ontology to make sense semanti-
cally, and that repels daily use in institutions. In that order,
the development of user friendly Ontology Editors is essential
to turn it into an everyday use program, since context analy-
sis will allow the retrieval of more precise information which
will interact in a way more reliable and more relevant to hu-
man knowledge.
For the common user, the Ontojuris System allows the
search in a wide database of texts about law taking advan-
tage of the similarity search: the system do not only looks
for the expressions inserted by the user but also brings results
based on the internal legal ontologies created in the Ontology
Editor and indexed by the system. This feature allows bet-
ter search results even when the users vocabulary doesnt have
specific expressions of the domain, what is common in Law.
The formula
Sim(A, B)=
n
i=0 Sim1(Tai)∗ht +
m
j=0 Sim2(Kaj)∗hk
n ∗ ht + m ∗ hk
where
Sim1(Ta) = wl if ∃Tb ∈ B|rl(Ta, Tb)
Sim2(Ka) =
1 if ∃Kb ∈ B|Ka = Kb
0 otherwise
is used to measure the similarity of a text B given a search
expression A. In the formula, T and K are terms and
keywords, respectively. In Sim1, w is the respectively
weight of one of the relations r between terms in a on-
tology: equality (weight=1), synonym (weight=0.99), con-
nected (weight=0.7), type (weight=0.4) or part (weight=0.3).
In the similarity formula, ht is a weight for terms on the on-
tology and hk for keywords in order to balance the relation
between these two.
2.2 Mind Engineering
In the era of the digitalized information, researchers of dif-
ferent areas of knowledge face a new subject: the semantic
organization of the data. The matter became important for
the fact of the information digitally available be disposed, in
majority, as non structured data. Though, it is known that the
structuring of data is a complex problem that can be solved
through the construction of formal models and of languages
of the Computer Science, on which it is necessary to observe
that those specific areas possess an own culture and a sin-
gular way of communicating. Observing the involved fac-
tors, in the context of Knowledge and Ontology Engineer-
ing, Mind Engineering Methodology was developed as a pro-
cess of knowledge synchronization to establishing concep-
tual models based on Artificial Intelligence and to identify
and systematize intellectual abilities of knowledge engineers.
Thus, the higher the synchronicity between the experts and
the knowledge engineers, the greater the effectiveness of the
system. This methodology includes people, processes and
technologies, through the knowledge sharing, visualization
and the definition of relevance.
Illustrating the application of this methodology for the con-
struction of the dictionary of ontologies in the legal scope
takes into consideration, not only the study and the extrac-
tion of the terms of the legislation applicable to the domain,
but also the usual terms used by the specialists found in the
didactic jurisprudences, doctrines and materials. The defini-
tion of related concepts implies research work or help from a
knowledge specialist on the matter. They are terms that can be
considered as synonyms of themes and secondary themes, as
well as close to the application context. An identifiable limit
does not exist for the number of related concepts. Therefore
it is important to observe the application of the terms in real
cases. The specialists are helped in this task by a technologi-
cal structure.
The concepts were defined starting from the fact that con-
nections previously exist and so similarities with the theories
and developments of acquisition methodologies and structur-
ing of knowledge already described. When ontologies had
being established, the search engine of the system is apt for
a better definition of the context that each term searched is
inserted.
The process of testing and evaluate ontologies define the
effectives of relationships considering the amount of docu-
ments retrieved. The goal is to keep users highly engaged, re-
turning online to the app and participating, which then leads
to real-world results [Wohlgenannt et al., ]. Nevertheless,
processing a task only by non-experts raises the problem of
ensuring quality. Therefore, the similarity of ontologies in the
texts of laws and decrees in the knowledge base of Ontojuris
system enables users to balance this problem.
The methodology works with a model in which the knowl-
edge engineer and the domain specialist exchange informa-
tion or, in other words, the engineer draws out the knowl-
edge structure from the specialist. The legal domain, as an

4. example, makes the construction of ontologies, easier since
all information is within the norms or the Case Law. The
difficulty of the validation is due to the imposition by the
Brazilian Laws themselves, which dictate to the citizen the
need/necessity of knowing the law. For that reason, a non-
specialist is obligated to not only know, but also to compre-
hend every law. Making the legal language common and un-
derstandable is one of the great challenges of AI. In this per-
spective, the recall test identify if the ontology relationships
improve de quality of the results in Ontojuris System. Thus,
the ontology is inserted in the analysis for evaluation in two
ways: textually, in the analyzer search where the document is
presented orderly based on the similarity of the ontology and
ontology relationships, or still in form of graphs that allow
the evaluation the quantity of documents where that ontology
and its relationships appear.
3 Crowd-based Approach for Ontology
Engineering
Crowdsourcing is an extreme case of dealing with the un-
known, where emergence and the reactions to emerging be-
havior play an important role: The individuals of the crowd
are a priori unknown and contingency plans for unexpected
behavior of this interacting mass cannot be fully prepared be-
forehand [B¨ucheler et al., 2010; Bcheler and Sieg, 2011].
Moreover, in a crowdsourcing scenario there are no pre-
defined contracts between parties like in traditional outsourc-
ing. [Lane, 2010] points out that risk is involved when using
crowdsourcing for decision making: “However, mechanisms
also need to be in place to protect against competition sab-
otaging the crowd system. Therefore, systems that leverage
the crowd for creation decisions should ensure that the final
decision passes through a governing body”. For this reason,
the proposed game dynamics here has two levels about the
inclusion of terms in the ontology.
Using the advantage of legal domain over other domains
of knowledge, a vocabulary defined exclusively by the law, a
new environment for expansion and validation of ontologies
supported by techniques of crowdsourcing games was built.
The Crowdsourcing Knowledge Engineering Suite (CKES) is
an independent computational structure for the development,
creation and modification of ontologies. These editors are
tools designed to assist the crowd to help the working team
of knowledge engineers and specialists in the construction of
legal ontologies. The results consist in a network structure
that connects words, considering their concepts in a defined
field of knowledge, inside a specific application. Such net-
works enable knowledge management applications to recog-
nize the context of the searched documents, i.e., the applica-
tions based on this kind of knowledge representation can do
contextualized searches for the stored documents. The most
important factor regards the search results in Ontojuris sys-
tem where the search by similarity helps the user in ontology
definition and alignment.
Starting from the search for a term (indicative expression)
the user has access to all the laws pertaining to that context
and also to the terms that are related to the document pre-
viously and are part of the domain ontologies. We also ob-
served that many projects that used crowdsourcing were get-
ting meaningful results. This work does not completely au-
tomatize the ontology construction processes, but allow for
a large and growing share of people interested in a partic-
ular knowledge domain, regardless of the degree of knowl-
edge. This process requires the system developer to provide
initial parameters, i.e., a knowledge base identified with in-
dexes and ontologies, easy and integrated access to social net-
works in order to allow the sharing of documents (laws and
jurisprudence) with other collaborators and followers. The
basic components for these ontology editors are classes (ar-
ranged in taxonomies), relations between terms (represent the
type of interaction between the concepts of a domain), and
subject domains. In this project the ontology representation
did/does not use axioms (used to model sentences that are al-
ways true) and did/does not use instances (used to represent
specific elements,(i.e. the data).
3.1 Crowdsourcing Knowledge Engineering Suite
The Crowdsourcing Knowledge Engineering Suite (CKES)
is an independent computational structure to expand legal on-
tologies. For non-expert users, it is a game that allows the
user to search an expression of her/his choice or presents an
indicative expression belonging to the ontology and performs
the search based on similarity on the juridical database look-
ing for decrees and laws about that term. The user can access
to all laws pertaining to that context and also to terms that
are related to the document and are part of the domain on-
tologies. Documents related to the term are ordered by the
similarity measure presented in Section 2.1. When the user
chooses some document, she/he can:
• include meronyms and hypernyms based in her/his
knowledge and the document chosen, and
• evaluate terms included by other users.
Each action can give to the user two kinds of points: im-
mediate and ontological. The first corresponds to points given
right after evaluation or insertion of a term or expression fit-
ting the system criteria (must be a term not present on the base
ontology, cannot belong to a previous defined list, etc). On-
tological points are distributed by the knowledge engineering
team that consolidates new terms on the base ontology. In the
game, the difference of these points corresponds to the status
given to the player: a player with a big amount of immediate
points is seen as a participative user, while a player with lots
of ontological points is seen as a kind of wise user.
CKES gives to expert users (the knowledge engineering
team) the most cited expressions inserted by the crowd clas-
sified according to the formula
Rank(R(t,exp)) =
(D(t) + 2 ∗ SH90(t)) ∗ 10 + Q(R(t,exp)) if Q(R(t,exp)) ≥ 0
−(D(t) + 2 ∗ SH90(t)) ∗ 10 − Q(R(t,exp)) if Q(R(t,exp)) < 0
where R(t, exp) indicates that the inserted term t is re-
lated to the indicative expression exp as a R = {meronym,
hypernym}, D(t) is the number of legal documents in the
database that contains the term t, SH90(t) is the number of
searches of t in the system in the last 90 days and
Q(R(t, exp)) =
n
i=1
PVi(R(t, exp)) − NVi(R(t, exp))
n

5. with n being the number of documents where R(t, exp)
was inserted, PVk(R(t, exp)) and NVk(R(t, exp)) being the
number of positive and negative votes of R(t, exp) in docu-
ment k, respectively.
Expert users then can select new expressions to modify the
base ontology and include terms in the list of prohibited ex-
pressions. It is interesting to notice that all new terms are re-
lated to legal documents (the ones chosen by the non-expert
user to insert expressions) given the juridical justification of
inclusion of terms.
3.2 An Experiment Using CKES
We submitted four users without any specialization or train-
ing in Law to use the system: two knowledge engineers and
two also without training in this area. Each user was free to
choose any expression related to indigenous population and
registered their path in the system with annotations about rea-
son of their choices in a document for posterior evaluation.
Table 3 presents the chosen terms related to the users profile.
User Term
Knowledge Engineer 1 Indigenous Culture
Knowledge Engineer 2 Indigenous Human Rights
Common User 1 Indigenous Language
Common User 2 Indigenous Access to Information
Table 3: Users and Terms in the Experiment
All users performed two kinds of search in order to find
legislation about the desired subject: exact term search and
search by similarity. While the search for the exact term
returned less than ten documents (with one search with no
document) in all cases, the similarity search found hundreds.
Users found a legislation text to explore in the first results
using the similarity search.
In what concerns about the task of insertion of new terms
in the ontology, we observed that the users feel the needed to
use another search window to look for different vocabulary:
something frequent when the knowledges engineers execute
this tasks. There was no surprise also that the users without
training in knowledge engineering had inserted around one
third terms than the knowledge engineers.
4 Final Remarks
Developing ontologies to represent knowledge in a particular
law is a bit more complex than developing ontologies to rep-
resent jurisprudence (Case Law). Laws can involve a wide
variety of areas, for example, the Brazilian Statute of the In-
dian covers health, education, security, equity, etc. and there
are no similar laws (as they tend to be revoked). In jurispru-
dence, however, the opposite happens; there are several laws
regarding the same matter and it is possible to use other sim-
ilar decisions to design and build ontologies. The laws tend
to be unique and need a thorough knowledge on the topics
it covers, making it unfeasible to create a body of experts to
build ontologies for each law specifically.
Another problem that adds in complexity is the amount of
time available for domain specialists to guide the knowledge
engineer, and, since domain specialists typically use a knowl-
edge structure developed after decades of study, identifying
and transferring these structures in a time frame measured in
hours is unlikely [Spreeuwenberg et al., 2012].
Currently there is a large and growing number of applica-
tions using collective intelligence or crowdsourcing to give
speed and amplify the update of databases. In the devel-
opment of ontologies it hasn’t been different (eg. Semantic
Web). The process is interactive in a way that the domain
specialist can describe knowledge structure to the knowledge
engineers, and the engineers can then validate their developed
structures with the specialists. However, despite the process,
once ontology is produced there are still significant problems
in verifying its correctness.
The advantage of crowdsourcing approaches against the
expert-based approaches is a much greater scale of discov-
ered semantics. Ontologies are becoming increasingly large
and complex, the legal vocabulary is even more complex. To
translate to colloquial language, the great difficulty is to un-
derstand the vocabulary of the citizen and/or parties involved
in the judicial dispute, making the juridical documents acces-
sible for him or her. This tool will allow ordinary citizens to
collaborate in the construction of this legal ontology. The rea-
son is the intrinsic motivation of the non-expert user of mak-
ing understandable legal language and the metric of similarity
allowing him or her to have an initial knowledge, avoiding the
problem of low quality results involving non-specialist users.
It is still early to assess the effectiveness of the tool and the
techniques used. But based on previous experience with of
gamification [Juho and Veikko, 2011], it is possible to con-
clude there will be a great improvement in quality and rapid-
ity for ontology expansion.
Acknowledgements
Our thanks to researchers and developers of Instituto i3G who
helped us for Indiopedia system evaluation, especially Clau-
dia Bueno and Jernimo Velasquez. This work was partially
funded by CNPq through a PDJ scholarship to Karina Rog-
gia.
References
[B¨ucheler et al., 2010] T B¨ucheler, R M F¨uchslin, R Pfeifer,
and J H Sieg. Crowdsourcing, open innovation and col-
lective intelligence in the scientific method: a research
agenda and operational framework. In H Fellermann,
M D¨urr, M M Hanczyc, L L Laursen, S Maurer, D Merkler,
P A Monnard, K Stoy, and S A Rasmussen, editors, Arti-
ficial Life XII – Proceedings of the Twelfth International
Conference on the Synthesis and Simulation of Living Sys-
tems, pages 679–686, August 2010.
[Bueno et al., 2004] Tˆania C. D’Agostini Bueno, Hugo Ce-
sar Hoeschl, Andre Bortolon, and Ricardo Miranda Bar-
cia. Engineering of minds: the synchronicity between ar-
tificial intelligence and the management of knowledge in
collaborative networked organizations. In Proceedings of
the IADIS International Conference WWW/Internet 2004,
Madrid, Spain, 2 Volumes, pages 1043–1046, 2004.

6. [Bueno et al., 2005] Tania C.D. Agostini Bueno, Hugo Ce-
sar Hoeschl, Andre Bortolon, Eduardo Mattos, and
Cristina Souza Santos. Knowledge engineering suite: A
tool to create ontologies for automatic knowledge repre-
sentation in knowledge-based systems. In MariaA. Wim-
mer, Roland Traunmller, ke Grnlund, and KimV. Ander-
sen, editors, Electronic Government, volume 3591 of Lec-
ture Notes in Computer Science, pages 249–260. Springer
Berlin Heidelberg, 2005.
[Bueno et al., 2006] Tania C.D. Bueno, Sonali Bedin, Fabri-
cia Cancellier, and Hugo C. Hoeschl. Evaluating the effec-
tiveness of a knowledge representation based on ontology
in ontoweb system. In HenrikLegind Larsen, Gabriella
Pasi, Daniel Ortiz-Arroyo, Troels Andreasen, and Henning
Christiansen, editors, Flexible Query Answering Systems,
volume 4027 of Lecture Notes in Computer Science, pages
625–633. Springer Berlin Heidelberg, 2006.
[Bueno et al., 2014] Tania C. D’Agostini Bueno, Karina G.
Roggia, and Hugo Cesar Hoeschl. Using crowdsourcing
games techniques and similarity metrics to improve legal
ontologies expansion. In Legal Knowledge and Informa-
tion Systems - JURIX 2014: The Twenty-Seventh Annual
Conference, Jagiellonian University, Krakow, Poland, 10-
12 December 2014, pages 201–202, 2014.
[Bunchball, 2010] Bunchball. Gamification 101: An intro-
duction to the use of game dynamics to influence behavior,
2010.
[Bcheler and Sieg, 2011] Thierry Bcheler and Jan Henrik
Sieg. Understanding science 2.0: Crowdsourcing and open
innovation in the scientific method. Procedia Computer
Science, 7(0):327 – 329, 2011. Proceedings of the 2nd
European Future Technologies Conference and Exhibition
2011 (FET 11).
[Casanovas, 2012] Pompeu Casanovas. Legal crowdsourc-
ing and relational law: what the semantic web can do for
legal education. Journal of Australian Law Teachers As-
sociation, 5(1-2):159–175, 2012.
[Corcho et al., 2015] Oscar Corcho, Mar´ıa Poveda-Villal´on,
and Asinci´on G´omez-P´erez. Ontology engineering in the
era of linked data. Bulletin of the Association for Infor-
mation Science and Technology, 41(4):13–17, April/May
2015.
[Gruber, 2009] Tom Gruber. Ontology. In Ling Liu and
M. Tamer ¨Ozsu, editors, Encyclopedia of Database Sys-
tems, pages 1963–1965. Springer US, 2009.
[Juho and Veikko, 2011] Hamari Juho and Eranti Veikko.
Framework for designing and evaluating game achieve-
ments. In DiGRA &#3911 - Proceedings of the 2011 Di-
GRA International Conference: Think Design Play. Di-
GRA/Utrecht School of the Arts, January 2011.
[Lane, 2010] Sean Lane. Collective intelligence for competi-
tive advantage: crowdsourcing and open innovation. PhD
thesis, University of Oregon, 2010.
[Law et al., 2011] Foong Li Law, Z. Mohd Kasirun, and
Chun Kiat Gan. Gamification towards sustainable mobile
application. In Software Engineering (MySEC), 2011 5th
Malaysian Conference in, pages 349–353, Dec 2011.
[Spreeuwenberg et al., 2012] Silvie Spreeuwenberg, Jeroen
van Grondelle, Ronald Heller, and Gartjan Grijzen. Us-
ing cnl techniques and pattern sentences to involve do-
main experts in modeling. In Michael Rosner and Nor-
bertE. Fuchs, editors, Controlled Natural Language, vol-
ume 7175 of Lecture Notes in Computer Science, pages
175–193. Springer Berlin Heidelberg, 2012.
[Stradiotto et al., 2009] Cesar K. Stradiotto, Tania C.D.
Bueno, and Hugo C. Hoeschl. Ontology graphical edi-
tor for multilingual document search system. In Troels
Andreasen, RonaldR. Yager, Henrik Bulskov, Henning
Christiansen, and HenrikLegind Larsen, editors, Flexible
Query Answering Systems, volume 5822 of Lecture Notes
in Computer Science, pages 453–464. Springer Berlin Hei-
delberg, 2009.
[Vianna et al., 2013] Y. Vianna, M. Vianna, and S. Tanaka.
Gamification, Inc: como reinventar empresas a partir de
jogos. MJv Press, Rio de Janeiro, 2013.
[Wohlgenannt et al., ] Gerhard Wohlgenannt, Marta Sabou,
and Florian Hanika. Crowd-based Ontology Engineering
with the uComp Protege Plugin. Semantic Web Journal.

7. Figure 1: Architecture of Indiopedia, using Ontojuris System