2011 10 07 (uam) emadrid mfreire ucm sistema analisis usabilidad herramientas autoria contenidos e learning

1. A Usability Analysis System for
e-Learning Authoring Tools
Manuel Freire Morán
eMadrid seminar on Adaptive Systems - 7 October 2011 manuel.freire@fdi.ucm.es

2. Presentation
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
1/20

3. Introduction: authoring & tool evaluation
Contents
System adoption requires content
Introduction
Story-line authoring  authoring
Ratings and outcomes  reuse
Time-lapse visualization
Future work Better authoring requires creativity support
Concluding remarks
 imagine, create, play, share, reflect
creative thinking spiral - Resnick, 2008
 low threshold, high ceiling, wide walls
 make it as simple as possible - and maybe simpler
Shneiderman et al, 2006
 evaluate your tools
2/20

4. Story-line authoring: adventure game authoring
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
3/20

5. Story-line authoring: Weev
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
4/20

6. Story-line authoring: initial question
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
5/20

7. Ratings and outcomes: a-b testing with ratings
Contents
A-B testing: very popular in web usability
Introduction
Story-line authoring  Split participants randomly into two groups
Ratings and outcomes  Each group uses an interface variant (A and B)
Time-lapse visualization
 Outcomes of each group are compared
Future work
Concluding remarks
Outcome of a creative task?
 Objective measures: correctness, completeness
 Subjective measures: questionnaires (author
satisfaction), ratings (comparative quality)
6/20

8. Ratings and outcomes: evaluating ratings
Contents
N users are requested to rate M users each
Introduction
Story-line authoring  All users rated same number of times
Ratings and outcomes  No user rated twice by same rater
Time-lapse visualization
 No user rates self
Future work
Concluding remarks
Unknown individual rating distribution. How
to determine if results are significant?
 H0: count(A rated better than B) is a fair coin toss
 one-sided binomial distribution
7/20

9. Ratings and outcomes: statistical treatment
Contents
1x b>a
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization 5/10 4/10 7/10 8/10 6/10
a b a a b
Future work
Concluding remarks
4 x a>b
8/20

10. Ratings and outcomes: experiment
Contents
Experiment with
Introduction
Story-line authoring  20 users (first-time Weev users)
Ratings and outcomes  5-minute tutorial on tool use (applicable to A&B)
Time-lapse visualization
 "Little Red Riding-Hood" script and resources
Future work
Concluding remarks
 1 hour's time; after editing, rating screen
Outcome
 70 total A vs B comparisons; in 40, B > A
 p-value: 0.141: does not reject null hypothesis
9/20

11. Ratings and outcomes: simulations
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
10/20

12. Ratings and outcomes: more simulations
Increase ratings-per-user
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
Increase number of users
11/20

13. Ratings and outcomes: increasing ratings
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
12/20

14. Ratings and outcomes: increasing users
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
13/20

15. Time-lapse visualization: idea
Contents
UI instrumentation sends data to server
Introduction
Story-line authoring  server timestamps records for each user
Ratings and outcomes  UI screen captures
Time-lapse visualization
 Action logs
Future work
Concluding remarks
 last screen capture sample used for rating
 ratings-file is just one more record type
How do you begin to analyze this data?
 Build videos from screen-captures
 Visualization tool to scan time-lapse data
14/20

16. Time-lapse visualization: interface
user / record
Contents selection time-lapse view of
Introduction records for first
selected user
Story-line authoring
time-lapse view for
Ratings and outcomes next selected user
Time-lapse visualization
Future work
Concluding remarks
detail /
difference view
15/20

17. Time-lapse visualization: insights
User is stuck
Contents
Introduction
Story-line authoring
Ratings and outcomes Creative flow
Time-lapse visualization
Future work
Concluding remarks
Rectangular placement
16/20

18. Time-lapse visualization: tasks and directions
Contents
 Tasks for UI evaluation
Introduction
Story-line authoring What did this user do?
Ratings and outcomes
- glance at timeline
Time-lapse visualization  What happened between here and there?
Future work - use difference function
Concluding remarks
 Still missing
 Variable-length time lapses ("time zoom")
 Graphical display of textual log data
 Filter & Sort
17/20

19. Future work: next experiment
Contents
 Server
Introduction
Story-line authoring  Finer-grained data collection
Ratings and outcomes  Better rating distributions, more ratings
Time-lapse visualization
 Support for experimenter note-taking
Future work
Concluding remarks
 Analysis tool improvements
 Test new automatic layout assistant
 Do people use it?
 Do people rate "automatically-improved"
storylines as better than-before?
18/20

20. Concluding remarks
Contents
A/B usability testing methodology for
Introduction
Story-line authoring
creative/aesthetic tasks
Ratings and outcomes
Reusable server
Time-lapse visualization
Future work  Collects images, logs, rating results
Concluding remarks  Configurable rating, allows real-time monitoring
Reusable client
 Specific handling for each record-type (ie.: logs)
19/20

21. Questions or comments?
Contents
Introduction
Story-line authoring
Ratings and outcomes
Time-lapse visualization
Future work
Concluding remarks
20/20