1. A Practical Method for
Courseware Evaluation
Aldo de Moor
CommunitySense
PragWeb 2007

2. The Pragmatic Web of what?
Communal effective use of
Information
Functionalities

3. Courseware and the Pragmatic Web
Pragmatic Web = context
How to evaluate web functionalities in their context
of use?
Courseware provides an interesting domain
technological environments consisting of multiple
functionality components, together offering a
complete system of info/comm services required for
supporting course needs
Functionality evaluation needed
Too much (costly) functionality
Gaps between required and available functionality
Conflicting functionality requirements

4. Functionality
Functionality
A set of functions and their specified properties that satisfy
stated or implied needs
Levels of granularity
Systems
Courseware environments
Tools
Blackboard
Modules
Announcements
Functions
Post announcement

5. A context model of courseware
evaluation

6. A practical courseware evaluation
method
Portfolio methods
Scores weighed by infrastructure/architecture of organization.
Bedell’s method for ICT investment selection
Functionalities scored on both effectiveness and importance for the
activities to be supported.
Practical method
Simplification of Bedell
No higher-order analyses
Actors (users in their roles) provide, interpret and decide
upon scores
Context: courseware evaluation:
Actors: students, software manager
Tool system level: module
Two questions
How well are course activities supported by various functionality
components?
To what extent are the functionality modules used?

7. Scores
Elements
I(a) = importance of activity
E.g. I(Information Collection) = 9
I(f,a) = importance of functionality f in supporting
activity a
E.g. I(Virtual Chat, Information Collection) = 4
Q(f,a) = quality of functionality in supporting an
activity
E.g. Q(File Transfer, Submission of Results) = 8

8. Activity and functionality scores
Activity scores
Σ I(fi,a) * Q(fi,a), for all functionalities 1..i.
Usefulness of the combined technologies for a
particular activity
Relevant for technology users (lectures, students)
Functionality scores
Σ I(aj) * I(f,aj) * Q(f,aj), for all activities 1..j
Usefulness of a particular functionality component
for the combined activities
Relevant for technology developers/maintainers

9. Experiment: evaluating group
assignment functionality
Two courseware tools: Blackboard, CourseFlow
Goal: making group assignments
Four activities, 11 functionality modules
Actors: 2nd year Information Management students,
software manager
2002: 62 students, 16 groups
2003: 46 students, 12 groups
Questions
Quality of tools for various group assignment activities?
Usefulness of various functionality modules?

10. Activity scores 2002/2003
Avg. Activity Scores in 2002 and 2003
450
400
350
300
Avg. score
2002
250
200 2003
150
100
50
0
Info coll. Discuss. Subm. Feedback
Activity

11. Avg. score
se
nd
di
0
200
400
600
800
1000
1200
1400
sc e-
m
us
ai
si
l
on
bo
ar
d
vi
rtu
di al
sc st ch
ud
us at
en
si
on tr
os
bo
te
r
ar
d
vi
rtu (g
ro
al
up
ch
)
fi l
e at
(g
tra
ro
ns
an up
fe )
no r(
un gr
co ce ou
m
ur p)
se
Functionality
en
in ts
fo
co (i n
rm
ur fo
se )
at
io
n
do
cu (i n
m fo
)
en
as ts
si
Avg. Functionality Scores in 2002 and 2003
(in
gn
m fo
)
en
ts
(in
Functionality scores 2002/2003
fo
)
2003
2002

12. Activity Scores BB/CF (2003)
Avg. Activity Scores Blackboard and CourseFlow (2003)
450
400
350
300
Avg. score
Blackboard
250
200 CourseFlow
150
100
50
0
Info coll. Discuss. Subm. Feedback
Activity

13. Avg. score
di s en
sc
0
200
400
600
800
1000
1200
1400
1600
us d e
s i -m
on a
b il
di v i oa
r
sc
us s t tua rd
l
u
si
on de c h
at
b nt
v i oa ros
rtu rd te
a r
fi l l c (g ro
e h u
an tra at ( p )
g
nn
c o o u s fe rou
ur nc r (g p)
e
s r
(2003)
c o e i n m e ou
p
n
f
ur
s e orm ts ( )
Functionality
do ati i nfo
)
cu o n
as me (i n
f
si
gn nt s o)
m (in
en fo
)
ts
(in
fo
)
Avg. Functionality Scores Blackboard and CourseFlow
Blackboard
CourseFlow
Functionality Scores BB/CF (2003)

14. Conclusion
Practical method, shown to be useful for initial
courseware functionality selection
Context-bound, in toto, ex-post evaluation
Simple measures useful for quickscan and
discussion purposes
Open source can perform just as well
“E-learning organization”: continuous evolution /
evaluation needed