This document summarizes an educational tool called ShowNTell that allows users to create voice-over recordings with annotations to answer students' questions. It describes ShowNTell's features, design and implementation challenges, evaluation through a user study, and areas for future work. The tool was found to perform well and be easy to use according to the user study. Future work may include adding features like importing video clips, video chat, and comments/likes, as well as deeper analytics.

1. +
ShowNTell - An easy-to-use tool for
answering students’ questions with voice-
over recording
Dr BHOJAN ANAND
IEEE/ASEE Frontiers in Education 2015

2. +
ShowNTell
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

3. +
ShowNTell
Problem
Statement &
Motivation
Features & Demo
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

4. +
Problem Statement
A simple-to-use screen and audio
recording tool to effectively
answer student questions

5. +
Motivation
• Existing products in market usually not
cross-platform
• Existing products require installation
• Recent improvements in browser
technology and network infrastructure
provide feasibility to implement a
recording solution for the web

6. +
Solution
n Novelty - plugin-free implementation of a
whiteboard recording application.
n Combine technologies on server and
client-side to provide functionality usually
found only on native applications

7. +
New browser technologies allow development of rich
interactive applications
Media Capture AudioCanvas
WebWorkers XHR2 + Blob
Feasibility

8. +
New browser technologies allow development of rich
interactive applications
WebRTCWebSockets
Feasibility

9. +
0
20
40
60
80
100
120
Apr-14
May-14
Jun-14
Jul-14
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14
Jan-15
Feb-15
Mar-15
Mbps
Average Internet broadband speed in Singapore (NetIndex)
Feasibility
Changes in resource capabilities allows handling of heavier data:
• Improvements in bandwidth and data transfer rates
• Improvements in performances of web browsers
• Improvements in performances of mobile devices

10. +
Features V 1.0
Insert and Annotate Images/Slides
Record Screen and Audio
Whiteboard Drawing Features
Export recordings to video format
Share recordings for viewing/editing
Share Videos for Viewing

11. ShowNTell is supported on
Desktop/Laptop Android
Browser support

12. +
Launch and Demo
n Access: www.sntboard.com
n Login: fiedemo
n Pwd: fiedemo

13. +
Features V 2.0
n Real-time collaborative editing
(WebSockets)

14. +
Features V 2.0
n Non-Linear Editing (Simple Video Editing
Tool)
n Provide ability to manipulate the recording
using a similar interface as a video editor

15. +
Features V 2.0
n Multimode Login and consolidation of
documents

16. +
ShowNTell
Related Works/
Products
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

17. Browser Desktop Mobile
ShowNTell
ShowMe
ExplainEverything
Ink2Go
CamStudio
Camtasia
BigBlueButton
Other software
Our project is unique in that it:
• Works in the web browser
• Works on mobile
• Does not require installation (no plugins)

18. • Sandboxed environment
• Tighter resource constraints compared to native
• Inconsistent support of APIs
• Inconsistent behavior among browsers
• Behavior changes between release cycles
Challenges – Browser Environment

19. Challenges – Mobile Environment
• Mobile devices have lower performance
• Limited memory and resources
• Inconsistent performance among devices

20. Examples: Camstudio, Fraps, Open Broadcasting Software (OBS)
• Desktop applications capture the screen and audio
• Captured data written to a file during recording
• Some support longer duration by distributing recording into multiple
files.
• Limitations: Overkill if using for simple recordings
Relation to ShowNTell:
• We capture audio which acts as continuous input stream
• We distribute recordings into multiple units to improve
responsiveness
Related work – Desktop applications

21. Examples: ShowMe, ExplainEverything
• Screen capture APIs not accessible, so do not capture screen
• Offer annotation capabilities with recording functions
• Capture events and replay events to simulate recordings
• Limitations: Not cross-platform
Relation to ShowNTell:
• Main inspiration for project
• Features available in mobile implementations emulated in ShowNTell
• Adopt event recording approach for recording implementation
Related work – Mobile applications

22. Examples: BigBlueButton
• Supports recording and playback
• Requires plugins like Adobe Flash
• Limitations: May not work on mobile browsers
Relation to ShowNTell:
• ShowNTell offers similar capabilities but does not require plugins
• ShowNTell works on mobile browsers
• Inspiration for other features that could be supported by ShowNTell
Related work –Web applications

23. Examples: Google Docs
• Supports editing by multiple users simultaneously
• Limitations: Uses native mobile apps to get around
compatibility issues
Relation to ShowNTell:
• Utilizes derivative of Google Wave OT algorithm, which was main
inspiration for OT algorithm used in ShowNTell
• ShowNTell handles multiple types of data (annotation, image, recorded
events) layered on top of each other, not just ordered elements in a
document.
Related work – Collaborative Editors

24. +
ShowNTell
Design &
Implementation
Challenges
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

25. Client-side: JavaScript
Server side: PHP and JavaScript (NodeJS)
Editor
Recorder/
Player
Input
capture
VMS
Video
render
er
MySQL
Compressio
n
Client HTTP
Server
Data
API
Conten
t
deliver
y
Dispatch
er
Storage
Authenticati
on
Renderer
Dispatche
r
Server
PDF
Converte
r
Design

26. Client-side: JavaScript
Server side: PHP and JavaScript (NodeJS)
Editor
Recorder/
Player
Input capture
VMS
Video
renderer
MySQL
Compression
Client
HTTP
Server
Data API
Content
delivery
DispatcherStorage
Authentication
Renderer
Dispatcher
Server
PDF
Converter
Design
Collaboration
Server
WebSockets
Bridge
Real-time
Collaboration
WebRTC Chat
WebSockets
Bridge
Chat
Handshaker
Real-time
Collaboration
Tracker

27. Data is stored on server
• Small data (< 128KByte): store on MySQL
• Large data (≥ 128 Kbyte): store on File
System
Hybrid solution is needed:
• MySQL does not scale well for large
transactions
• File system incur overhead for small
transactions
• Relational information needs to be stored
[large data]
[small data]
File
Database
record
Record
0
50
100
150
200
250
300
350
400
0 200 400 600 800 1000
TimeTaken(ms)
Fragment Size (KB)
FileSystem Put FileSystem Get
Database Put Database Get
Implementation (Storage)

28. We have two mechanisms to record audio:
• GetUserMedia API
• Adobe Flash (fallback)
From audio stream to storage:
• Poll the stream for PCM (Pulse Code Modulation) data
• Accumulate data to a buffer, until reach a threshold (e.g. 5 seconds)
• Convert PCM data to WAV format
• Upload WAV fragment to server
Implementation (Audio Recording)

29. • WAV: Raw uncompressed data generated by microphone.
• MP3: Supported by all major browsers.
• MP4/AAC: Supported by all major browsers.
• Ogg/Vorbis: Seamless. Supported by Chrome and Firefox.
• WebM/Vorbis: Seamless property. Supported by Chrome, Firefox, and
IE.
Implementation (Audio Compression)
786,476
72,339
74,553
82,211
84,914
0 200,000 400,000 600,000 800,000
WAV (Uncompressed)
MP3
MP4/AAC
Ogg/Vorbis
WebM/Vorbis
Compression is used to reduce bandwidth requirements

30. Challenges:
• Audio broken up into fragments (non-contiguous)
• Seamless property needed for smooth gapless
playback
• Browsers have varying support for audio
Format Seamless Internet Explorer Chrome Firefox Safari
WAV
MP3
MP4/AAC
Ogg/Vorbis
WebM/Vorbis
Implementation (Audio Formats)

31. Compression is done by server instead of client:
• Experiments reveal that client-side compression is expensive
• Compressed data is cached to reduce time for subsequent access
Client
50.4
271.4
171.2
223.6
185.8
297.4
50.4
41.4
39.2
46.5
43.8
42.4
0 100 200 300 400
WAV
OGG
AAC
MP3
MP4
WEBM
Time taken (ms)
Non-cached
Cached
Server
43.2
2590
3110
262
11390
14409
0 5000 10000 15000 20000
WAV
OGG
MP3
Time taken (ms)
Desktop Browser
Mobile Browser
Audio Duration (5 seconds)
Implementation (Audio Compression)

32. Main timeline
0Time slots (array)
[Event 1, Event 2] [Event 3]
53
[Event 4, Event 5]
121
• Event-oriented
• Events do not belong to any slide
• Covers audio and document-wide events
• Supports generic events
Slide timeline
Drawing 1Objects (array)
[Draw] [Move, Resize]
Image 1
[Draw]
Drawing 2
• Object-oriented
• Objects only belong to the specific slide
• Covers information to reproduce visual state
• Events follow a fixed format
Implementation (Event Recording)
We perform event capture instead of visual capture

33. Thresholds used for de-noising
• First point: Starting point is always accepted
• Euclidean distance: > 4 pixels.
• Trajectory angle: > 1 radians, and > 2 pixels.
De-noisedOriginal
Implementation (Drawing)
V2
V1
C
θ
Drawings are reproduced from set of points captured from user
Applied Bezier curves and de-noising to provide smooth
curves for drawings

34. Video rendering service:
1. Use node-canvas to render video frames
2. Use avconv to combine frames to video format
3. Use SoX to generate audio
4. Use avconv to combine video and audio
Download
Document
Render Frames
Render Frames
Render Frames
Render Frames
Multi-threaded render
Join MP4
fragments
Combine
audio chunks
Combine audio
collections
Combine
video and
audio track
MP4 Video
Implementation (Video Rendering)

35. Collaboration server:
• Document is loaded and kept in memory of collaboration server
• Changes made by client are applied immediately to memory
copy
• Copy is serialized to DB periodically, and only if changes are
made
Real-time Collaboration Tracker
Implementation (Collaborative Editing)
op
Transform op
with OT
Apply op’ to
document
Broadcast op’
to all clients
op’
op’
op’
Delayed
serialization
of document
Data API

36. Chat ‘Handshaker’
• Chat clients prompted to connect to each other by server
• Audio chat is transmitted over fully-connected P2P network
• Chat messages are transmitted to and broadcasted by server
Chat Handshaker
Implementation (Audio-enabled chat)
Handle WebRTC
Handshaking
Broadcast Chat
Messages

37. +
ShowNTell
Evaluation/User
Experience
Survey
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

38. Objectives of study:
• Determine the performance and stability of ShowNTell under
medium load
• Determine the capabilities of ShowNTell on different machines
• Gauge user experience with regards to usability and design
• Obtain suggestions for improvements that can be made on the
system
User study

39. Students from CS3247 (Game development) module have participated in
the study
During the study, participants are to:
• Utilize ShowNTell to answer questions related to the module
• Have supporting annotations or slides
• Complete the survey upon completion of the assignment
Some participants are provided access to a Samsung Galaxy Note 10.1
with stylus.
User study

40. Participants 45 (30 respondents)
Browser used
(may use multiple)
Google Chrome (80%)
Mozilla Firefox (17%)
Safari (7%)
Experiences with other
software (may use
multiple)
Camtasia (31%)
Camstudio (34%)
Others (20%)
None (34%)
Most participants have used Google Chrome
• Participants have used both desktop and mobile browsers
• Most are using ShowNTell on Windows
Many participants are familiar with at least one recording
software
• Most commonly used: CamStudio and Camtasia
User study demographics

41. Document duration ≤ 3 minutes (7%)
3 to 5 minutes (49%)
5 to 10 minutes (42%)
≥ 10 minutes (2%)
Used PDF import 97%
Annotations Extensive (51%)
Limited (17%)
None (32%)
Most recordings are under 5 minutes
• There are also many exceeding the expected 5 minutes.
Most participants used the PDF import tool
• Most used slides to create starting material
Most have used annotations (e.g. drawing)
• However, many created documents exhibited limited annotations.
• Some opt not to use annotations as not required for the assignment
User study demographics

42. Most participants indicate favorable performance
• Our techniques to improve responsiveness falls within tolerable levels
Many participants indicate that ShowNTell is easy to use
• User interface is sufficiently guided even when no explicit instructions
given to users on how to use the system
Many participants indicate a favorable rating for user experience
• Overall design is within acceptable levels
Many participants find ShowNTell comparable to other products
• However, they also indicate it could be improved further
0
5
10
15
Performance Ease of use User
experience
Compared to
other
products
Responses
1 (Poor)
2
3
4
5 (Great)
Evaluation of ShowNTell with User Study

43. Almost all participants (93%) have used a mouse input
• Users find ShowNTell to be fairly usable with a mouse
Some participants (40%) have used a touch input
• Users find ShowNTell to be mostly usable although they also face
difficulty in using it for drawing
Some participants (50%) have used a stylus input.
• More favorable ratings for stylus compared to touch, but still fall
behind mouse input
• It is likely that participants are not used to the stylus input
0
5
10
15
Usability
(Mouse)
Usability
(Touch)
Usability (Stylus)
Responses
1 (Poor)
2
3
Evaluation of different device inputs

44. Other evaluations
• It is feasible to implement a recording solution in a web browser environment
• Most browsers support the APIs or workarounds to allow feasibility of
implementation
• Modern mobile devices have sufficient performance for application usage
• Techniques applied allows ShowNTell to remain responsive to the end-user

45. Limitations
Not all browsers fully support ShowNTell yet
Not supported because browser do not support required API or fallbacks
iOS SafariiOS Chrome RT IE Android StockEdge

46. Client-side audio compression requires significant
memory and computational resources
Limitations
43.2
2590
3110
262
11390
14409
0 5000 10000 15000 20000
WAV
OGG
MP3
Time taken (ms)
Desktop Browser
Mobile Browser
Certain tasks too expensive on mobile devices

47. +
ShowNTell
Conclusion and
Future work
Related
Works/
Products
Features
&
Demo
Conclusion
and Future
work
Problem
Statement &
Motivation
Evaluation/
User
Experience
Survey
Design &
Implementation
Challenges

48. +
Future Work
n Import Video Clips
n Video Chat
n Comments/Like sections
n Deep Analytics and Context based
related videos