Collaborative Projects And Self Evaluation Within A Social Reputation Based Exercise Sharing System

1. 2009 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technologies
2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology
Collaborative projects and self evaluation
within a social reputation-based
exercise-sharing system
Andrea Sterbini Marco Temperini
Dept. of Computer Science Dept. of Computer and Systems Science
”La Sapienza” University, Rome, Italy ”La Sapienza” University, Rome, Italy
sterbini@di.uniroma1.it marte@dis.uniroma1.it
Abstract—We present the design issues and motivations of an where they can interact, exchange information and collaborate
enhanced version of the web-based system S OCIAL X, supporting over common problems (e.g. mandatory exercises in a subject
collaborative and social aspects of learning. This web application matter).
allows to share solutions to exercises and development of project-
(possibly group-) work, through the management of a reputation In this paper we deal mainly with the designing aspects of a
system. With the aim of enhancing collaboration and to help web application, called S OCIAL X, taking care of collaborative
students working on exercises, we introduce contextual FAQs and social aspects of learning in the aforementioned system
and micro-forums and a currency-based concretization of the and allowing the management of an augmented learning
perceived usefulness of other’s answers. The tokens exchanged model, based on the analysis of learner behavior in a specifi-
are used also to help the teacher/tutor in choosing the best
question/answer pairs to be promoted to the FAQ. To introduce cally devised reputation system environment.
group responsibilities, peer-pressure and self-evaluation we de- II. T HE OLD S OCIAL X
fine group-based projects with self/peer-evaluated phases. The
different phases of a project are given to different groups, so Our previous system [5], paired a reputation system together
that the produced deliverables are both self-evaluated when they with an exercise-sharing web tool. It aimed at:
are submitted and peer-evaluated by the group working on the
• increasing the motivation of students in doing home-
next phase. The system is its last stages of development and will
be tested with real students in the next academic year. works,
• increasing/encouraging higher cognitive learning activi-
ties (as in the Bloom cognitive taxonomy [1]), both by
I. I NTRODUCTION rewarding the student grading other’s solutions, and by
Cooperative learning is an indispensable element (in the rewarding the reuse and correction of other’s solutions
e-learning field) to help learners sharing and combining ex- To obtain this we built a reputation-based system, within
pertise, with the goal to prepare to join team-based working which a student was able to work on homeworks, share
environments [2]. Whereas cooperative learning is usually his/hers solutions and judge and reuse other’s solutions. A rep-
discussed and applied on small groups, a further aspect of utation system is normally used to motivate interaction and to
interest is then in the vision of e-learning as a community elicit good behaviors by awarding points to the user’s actions
and social activity [7]. Cooperative learning can improve that are deemed more useful to the community. In our case the
teaching and learning considerably; moreover its implications student’s reputation is a blend of five facets that describe how
and effects in the extension of present e-learning standard well s/he is working within the class: involvement, usefulness,
(namely the IMS Learning Design [4]) have just started to competence, judgment, and critical thinking (see later for
be considered [8]. Fundamental didactic tools in collaborative details). Reports of the student’s reputations can be shown
learning are reputation systems, that capture (and make evident both at the course and topic level, with details displaying all
to the learner) the contributions s/he is giving to the group, to the facets to allow the student to improve his/her reputation
the class and to the course. A reputation system is both a mo- by focusing on the type of social activity s/he likes more.
tivational tool and a way to evaluate and understand learner’s
psychological preferences, relations with others, ability to III. T HE NEW S OCIAL X
analyze/judge others’ work, and thus conceptual competences. SocialX is being extended with the added goals of:
We are also working on a comprehensive approach to the • increasing collaboration and peer-based help by intro-
management of personalized courses [3], [6], exploiting social ducing contextual micro-forums within which ”direct
aspects of learning to enrich the definition of learner’s model. rewards” (tokens) are used to explicitly capture the per-
In particular, opposite to the usual approaches to collaboration ceived usefulness of other’s help,
based on small groups, we are defining a model including • introducing peer-pressure and responsibilities towards the
the idea that learners are participating to a social network, group by managing group-based projects.
978-0-7695-3801-3/09 $26.00 © 2009 IEEE 243
239
DOI 10.1109/WI-IAT.2009.273

2. • support the teacher, which is a very valuable yet limited tokens that can exchange with good answers. Whenever s/he
resource, by exploiting to our best all the recorded social needs an information s/he can pose a question (consuming
interactions among the intervening students. one token) and reward the best answer received. Tokens are a
Moreover, we introduce a new facet in the reputation system limited resource, and thus a student needing answers should
to explicitly capture the self judgment ability of the student. first ”work” for the community to collect the tokens needed
Definition 1: The reputation of a learner is an overall to ask more questions. The total number of tokens received
representation of certain learner’s qualities as they come out is a direct indication of the usefulness of the student in the
from his/her interaction with the S OCIAL X system. It can be community, and thus it contributes to the usefulness and to
calculated at different levels of detail in the system: course the competence factors of his/her reputation. The number of
topic, whole course, and whole system (encompassing several tokens spent, instead, counts how many times the student has
courses). There are six basic aspects that are taken care of in asked questions to the community, and thus it contributes to
the system: the involvement factor of his/her reputation.
• involvement: the degree of active participation in the To avoid students cheating the system (e.g. by exchanging
system, measurable by the amount of work that the useless questions/answers) we mildly discourage ”off-topic”
learner has been available to submit, also in terms of and ”dummy” discussions. The teacher/tutor flags this kind of
participation (such ad the number of solutions submitted, useless exchanges so that they contribute zero to the reputation
questions proposed and grades given, as well as the and the token spent to create the question is lost (for both
propriety and extension of judgments); the students involved). Discouraged exchanges may affect the
• usefulness: how the learner’s work is beneficial to others reputation of both parties involved.
in the system (such as the reuse of learner’s solutions, Therefore the participation of the students at the contextual
and the appreciation of her/his questions); micro-forums produces reputation through the rules:
• competence: an appraisal of the skills shown by the 1) the answers given to others (even if not awarded with the
learner (deriving from the grades and judgments coming token) contribute to the student’s involvement factor,
from peer students and from the teacher; 2) the tokens received show how much a student has been
• judgment: how well the student has evaluated other’s useful to the others, increasing her/his usefulness factor,
solutions, questions, answers and products (with respect 3) the tokens spent to propose questions show how much
to the teacher’s grades and evaluations) the student has participated, and contribute to his/her
• self-judgment: how well the student has evaluated her/his involvement factor,
own answers and products (with respect to the teacher’s 4) a Q/A promoted to FAQ shows that that contribution is
grades and evaluations) important, thus contributing to the answering student’s
• active critical thinking: a measure of the conceptual competence factor,
work issued to understand and critically appraise others’ 5) ”dummy” and ”off-topic” discussions are completely
work, in order to modify, reuse, and start from such ignored and loose the corresponding token
work (such as when a solution is the first produced for a To make the best use of the teacher’s time we highlight the
problem, or is the correction of another) token exchanges to help him/her to evaluate faster the dummy
A. Increasing collaboration and FAQ candidates.
We introduce both contextual micro-forums attached to each C. The teacher is the bottleneck
exercise, so that students help each other by asking/answering
questions, and FAQs to collect the most interesting discus- The teacher’s work in the system is a crucial, limited
sions. The students exchanges are moderated by the teacher, resource. S/he should correct solutions, moderate answers,
that can ”promote” the discussion threads by refactoring the promote good Q/As to the FAQs, manage the group projects.
most interesting pairs of questions/answers to the exercise’s We must make the best use of the teacher’s expertise, even if
FAQ. When a discussion is refactored as a new FAQ entry, the s/he would be able to check/test/correct just a small part of
students involved in the originating discussion are rewarded by the solutions submitted. To this aim we exploit the network of
increasing their usefulness and competence reputation levels. social exchanges between the students to guide the teacher by
To make the best use of the teacher’s time, we highlight the selecting the most interesting items to be evaluated. Then, the
exchanges that the students have already selected as the most social network is used to propagate the evaluation results to the
interesting/appropriate. neighbor items to adjust the authors’ reputations accordingly.
E.g., the tokens exchanged in the contextual micro-forums
B. Perceived usefulness are used (also) to highlight the most useful answers, thus
To enhance the motivation of the students in helping each- reducing the number of Q/As to peruse while looking for
other, and to make explicit the perceived usefulness of others good candidates for FAQ promotion. In the exercise evalu-
we apply the classical currency-based approach (tokens) that ation the teacher is guided by the judgments expressed, their
students can use to acquire services by other students or by agreement/disagreement, and the reputation of the intervening
the teacher. Each student is awarded an initial number of students.
244
240

3. IV. S OCIAL C OLLABORATIVE P ROJECTS A WF is a set of projects {Pj }j∈(1,...,nWF )
A SCP-path in a WF is a set of tasks
As seen, S OCIAL X allows the use of a reputation system
in an e-learning environment, supporting the development of {ti,j }i∈(1,...,nT ),j∈(1,...,nWF )
collaborative-social exercising activities within a potentially
where ti,j is the i-th task in the j-th project of the work-field.
large group of students. Such “exercising activities” so have
been made of single exercises, freely reusable by each learner.
In a work-field, the projects are supposed to share a common
So, in the context of S OCIAL X the learning activity is a trade-
structure, meaning that the number of steps and their logical
off between individual work (selection and comprehension
sequence are homogeneous, so that it is acceptable that a SCP-
of others’ work, reuse and adaptation, development of new
path provides group learners with a reasonably standard and
solutions) and social exchange.
complete project activity in the course topic. Once a suitable
However in certain courses the development of projects is a
work-field is defined, SCP-paths can be assigned to groups.
relevant part of learning, in both cases of an activity performed
The following is an example of path assigned to a group g:
by the individual learner or a collaborative work carried on
by a small group of learners. So we extended S OCIAL X to {t1,k1 t2,k2 · · · tnT ,kn }
g g g where
T
g
embed also the support to a partially social approach to the ∀h ∈ (1 . . . nT ) kh ∈ (1, . . . , nWF )
development of projects. The approach is called “partially”
(the path is made by nT tasks (to make the overall activity
social collaborative because, while a small group (possibly
complete according to the course topic definition); each i-th
singleton) of learners is still the basic operating unit, the
task is the i-th task in one of the projects of the work-field).
products of such units are submitted to social exchange with
If we can assume that each task in a project depends on the
the other units (to be reused and assessed). A project is usually
previous and is depended upon by the following, we can also
a prolonged and organized activity, made of a sequence of
assume that for almost each task undertaken by a group in
tasks, each one depending on previous one and depended
its SCP-path, the group is going to depend on the work done
upon by the following ones. Usually a project is entrusted
by other groups and will produce material for other groups to
to a small group of learners, and collaborative work among
use. This gives the social dimension to the activities in a SCP-
them is instructed and supported, to produce the deliverable
work-field, and gives also the opportunity to add feedbacks
for the whole project. We add to S OCIAL X the support to
over the reputation of learners, beyond the evaluation of their
a partially social collaborative approach to the development
technical skills related to project deliverables. (In considering
of projects. Instead of having a small group working on the
the dependences of a tasks from others, we limit the scope
various steps of a single project, the idea is to have the
to those immediately preceding and succeeding, in order to
group working on different steps of different projects: all
simplify a bit the notation, with no prejudice for the general
the projects share a similar structure, made of a sequence of
discussion).
tasks (the steps); the n-th task of a project is expected to be
“similar” to the n-th task of another (wrt the general learning
Definition 3: (fulfillment of a task by a group)
goals related to the project development methodology); so the
Given a task ti,j assigned to group g = {lp }p∈Ig , and
group would be assigned a path of tasks, each one possibly
assuming that the previous and successive tasks in the same
involving a step in a different project; at each step the group
project Pj , ti−1,j and ti+1,j , are resp. assigned to groups g
should deliver a product; moreover, the learners in the group
and g, g fulfills ti,j when it provides the system with
provide evaluations of the product(s) received from earlier
step(s) in the same project (from which the group should • A product p(g, ti,j ),
start to work on its task) and of the deliverable released by • A set of evaluations { VAL (lp , g, ti−1,j )}p∈Ig over the
the group (to show self-evaluation skills). We define a social product received from the previous task in the project
collaborative project (SCP), in a given course topic T , as a (one explicit evaluation for each member of the group),
set of tasks P T = {ti }i∈(1,...,nT ) . Each task is assigned to • A set of self-evaluations { AVAL (lp , g, ti,j )}p∈Ig over
a group of learners (g T = {li }i∈(1,...,ngT ) ), that will do the the product released by the group itself (one explicit
corresponding learning activity (such as the construction of evaluation for each member of the group),
a deliverable product). Moreover, the sequence of tasks in When the group g = {lq }q∈Ig has fulfilled its task
a SCP provides a complete span of learning activities about ti+1,j another set of evaluations {VAL(lq , g, ti,j )}q∈Ig ) will
the related project methodology. (Henceforth, where possible be available over the work of group g.
we’ll assume that projects are all on the same T and avoid
the related indexes.) So, from the work of a group of learners g in a SCP-work-
In the following definition, an SCP-path is a sequence of field, and from Def.3, many items may produce a feedback
tasks, selected from different projects in such a way to provide over the reputation of the group members:
the aforementioned complete span of learning activities. 1) for each task ti,j of the SCP-path assigned to g we have
a set of evaluations of the product p(g, ti,j ), issued by
Definition 2: (work-field - WF - and SCP -path) the members of group g that followed g in the same
245
241

4. project Pj ; each grade, as well as the principal one given 3) Self evaluation through open-answers quizzes: We want
by teachers, is imparted to the whole group and can to introduce open-form quizzes with a very simple mechanism
easily be spread, mediated by the teachers’ judgment, to that allow the student both to engage in self-evaluation and
feedback over usefulness and competence of each lp ∈ g. to do high-level cognitive work (respect to the Bloom’s
2) for each task, ti,j , we also have the evaluations issued hierarchy). The student is proposed a question, which s/he
by group members about the product p(g, ti−1,j ) inher- answers. Then s/he is proposed a selection of peer’s answers
ited from the previous task in the same project: those to the same question (including his own) from which s/he
are single learner’s evaluations, that can be compared could choose the best answer. In doing this, the student is
with teachers’, affecting both learner’s competence and analyzing his and the other’s answers, comparing them to
judgment. each other. The above evaluations could be wrong, because
3) the various evaluations mentioned at point 1) are also to it’s affected by the student competence on the topic. The
be taken into account to measure the ability of group preferences expressed in the system are then used to analyze
g to build a good product, basing on the one they such level of competence. Answers that collect high number
received from previous task: the relationship between the of choices are probably more correct, and contribute to the
grades of the former, p(g, ti,j ), and those of the latter, competence part of the author’s reputation. As the preference
p(g, ti−1,j ) can provide a feedback over the active criti- relation expressed should be transitive, if all preferences are
cal thinking component in the reputation of the members correct they should build a poset or a total order. If a student
of group g. Of course, as it is apparent that the evalua- has a limited knowledge of a topic and makes a wrong choice
tions over the previous product, {VAL(lp , g, ti−1,j )}p∈Ig then s/he could introduce a preference going in reverse order
are coming from g’s members, for it only the teachers’ respect to the ”correct” order, which could introduce cycles.
grades will be taken into account. Thus, cycles in the preference graph highlight the presence of
4) finally, for each ti,j , task assigned to g, we also have a misunderstanding and could be used to select which answers
the evaluations issued by group members about their should be examined first to find the mistake (and to correct
own product p(g, ti,j ): those are single learner’s self- the corresponding student’s competency level on that topic).
evaluations, that can be compared with teachers’ evalu- Moreover, while the teacher evaluates part of the answers,
ations, affecting learner’s self-judgment. the graph could be used to propagate the marks given to
student to other (yet not examined) answers and to assess the
V. F UTURE W ORK competency levels of others. At a given moment, depending
on the preferences expressed so far on the presented answers,
We have presented the new SocialX system, which in- the answers can be ranked as: 1) best answers, which have
troduces collaborative group projects, and contextual micro- been chosen by many; 2) worst answers, which have been
forums with rewards for best answers within its reputation proposed but never chosen; 3) unseen answers, which haven’t
system. The system is its last stages of development and will been proposed yet. Therefore, the choice of answers to propose
be tested with real students during the next academic year. In to the student is better done by choosing an appropriate mix
a near future, we intend to continue the SocialX expansion in of the above three types: 1) some best answers: to allow the
several directions: switch to a better answer; 2) some worst answers: to show
1) The teacher as a quality rater: Our initial approach good distractors; 3) some unseen answers: to evaluate all the
uses the token exchanges as a simple indicator of hot topics, answers.
while the student’s judgments are used to pinpoint the most
important solutions to mark. R EFERENCES
Our final aim is to transform the teacher into the ”quality [1] B.S. Bloom (Ed). Taxonomy of Educational Objectives. David
McKay Company Inc, New York (1964).
assessor” of the system, by efficiently highlighting the most [2] Y. Cheng, H. Ku. An investigation of the effects of reciprocal peer
important didactic decisions and by leveraging the student’s tutoring. Comp.in H.Behav. 25 (2009).
social network structure with its reputation levels, instead than [3] G. Fernandez, A. Sterbini, M. Temperini. On the Specification of
Learning Objectives for Course Configuration. Proc. Int. Conf. on
keeping him engaged in tedious repetitive tasks. Web-Based Education (WBE), (2007)
2) Student’s Fairness: As we have seen with the discour- [4] IMS Learning Design Best Practice and Implementation Guide;
aged exchanges, to keep a high level of quality we discourage IMS Learning Design Information Model; IMS Learning Design
XML Binding. http://www.imsglobal.org/learningdesign/index.cfm.
misbehaviors. The penalization should be done very mildly to [5] A. Sterbini, M. Temperini. Learning from peers: motivating stu-
avoid discouraging also normal participation, thus we currently dents through reputation systems. Int. Symp. on Applications and
just make all misbehaviors void. We would like to introduce the Internet, Social and Personal Computing for Web-Supported
Learning Communities (SPeL). Turku, Finland, (2008).
a fairness factor to capture how much the student agrees [6] A. Sterbini, M. Temperini. Adaptive Construction and Delivery
with the ”didactic pact”, i.e. with the proper behavior rules of Web-Based Learning Paths. accepted for publication in Proc.
in the course. This factor is probably meaningful only for the Frontiers in Education (FIE). San Antonio, Texas, (2009).
[7] E.Wenger. Communities of practice: Learning, meaning, and iden-
teacher/tutor, and is updated whenever the student misbehaves tity. Cambridge Un. Press (1998).
within the system, either by annoying others or by trying to [8] Yu D., Chen X. Supporting Collaborative Learning Activities with
cheat the system. IMS LD. Proc. ICACT2007 (2007).
246
242