Presentation at AVI 2004 (Gallipoli, Italy) to accompany short paper - doi: 10.1145/989863.989886 Basically a matrix of scores, assessors v candidates, where rows/columns can be sorted and scores grouped on the crosshair axes to see them more easily in context Also a matrix where the cells (not rows/columns) are hierarchical

1. Exploring & Examining
Assessment Data via a Matrix
Visualisation
Mar tin Graham & Jessie
Kennedy
Napier University, Edinburgh

2. Background
• Work part of OPAL – Online Partner Lens
• Prospective business partners, employers,
employees assess each other on various
characteristics
• ‘Lazy’ users would like to search these
assessments rather than perform their own
AVI 2004 - Gallipoli, Italy 2 of 13

3. Why Visualisation?
• Why not just search the numbers?
• Simple search – may just pick out lenient
assessors rather than quality candidates
• Statistical analysis – can give averages, but not in
context of related assessments
• Recommender system - only possible if user has
previously performed assessments of their own
• Lack of feedback & freedom to browse
• Use a visualisation to convey context of
users and assessments
AVI 2004 - Gallipoli, Italy 3 of 13

4. Why Matrix Visualisation?
• Node-link visualisations give precedence
to nodes
• Also clutter rapidly when edges >>> nodes
• In our case, the interesting data is
primarily the assessments – the edges
• Matrix visualisations have edges/links at the
centre of attention
• Directed nature of edges mean assessors and
candidates map naturally to axes
AVI 2004 - Gallipoli, Italy 4 of 13

5. Initial Layout
AVI 2004 - Gallipoli, Italy 5 of 13

6. Assessment Context
• Investigating single assessments doesn’t
tell us much as score is product of both
assessment and candidate
• Showing related assessments could
reveal context of assessment, and of the
participants
AVI 2004 - Gallipoli, Italy 6 of 13

7. Assessment Context
• Reveal context by overlaying related
assessments as ordered bars
• For example, say the candidate crosshair
intersects two assessments, both coloured blue
• The assessors who gave these ratings have
their other evaluations collected and ordered
around these points
• In this case, the bars show the candidate got
the worst scores that each assessor handed
out
• Not only are the scores poor on an ‘absolute’
scale, they are poor ‘relatively’ too
AVI 2004 - Gallipoli, Italy 7 of 13

8. Assessment Context
• This candidate has been involved in 7
evaluations in total, all of them poor
• Furthermore, they have received the
lowest score each assessor has handed
out
AVI 2004 - Gallipoli, Italy 8 of 13

9. Assessment context
• Brushing a point in the matrix will show value
bars for assessors and candidates
• Here, we see a low score obtained even though the
candidate has a very high average
• The bars along the vertical crosshair show that this assessor
has a history of handing out harsh evaluations.
AVI 2004 - Gallipoli, Italy 9 of 13

10. Filtering / Focusing
• Candidates and assessors may be filtered
by position in matrix
• I.e. remove all assessors with < n
assessments
• Assessments are hierarchical in nature
• User may filter matrix to include only
attributes they are interested in
• User may zoom on portions of matrix to see
assessment details
AVI 2004 - Gallipoli, Italy 10 of 13

11. Specific Attributes
AVI 2004 - Gallipoli, Italy 11 of 13

12. Conclusions
• Collating and overlaying related
assessments acts as a context for
verifying a candidate’s or assessor’s
associated evaluations
• Allows users to see whether a candidate’s
scores are consistent or not given the
assessors who have applied them
AVI 2004 - Gallipoli, Italy 12 of 13

13. Acknowledgements
• OPAL – EU Project IST-2001-33288
• http://www.opal-tool.net
AVI 2004 - Gallipoli, Italy 13 of 13