This document discusses the history and future of human-computer interaction and empathetic computing. It describes early visions of interactive computing from researchers like Vannevar Bush and Ivan Sutherland. More recent concepts discussed include ubiquitous computing, tangible interfaces, virtual and augmented reality. The document proposes a vision for "empathetic computing" using sensors and wearable devices to understand and convey emotions between users to enhance empathy. It outlines some initial experiments and analyses opportunities for future research in areas like real-world augmented reality, brain-computer interfaces, and city-scale social analytics.

2. From Interaction to Understanding
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
May 21st 2015

3. My Childhood Home

4. The Vision - Auckland

5. The Road Trip

6. Happy Trails

7. The Reality

8. Essential Supplies

9. Mount Messenger

10. Are We There Yet?

11. Vision vs. Reality

12. Lesson’s Learned
! Enjoy the journey
! Travel with others
! Have a well equipped vehicle
! The end always seems nearer than it really is
! Don’t be disappointed in the outcome
! Most Important: Have a Clear Vision

13. HUMAN COMPUTER
INTERACTION

16. Processing Power

17. User Interaction
Hard Wire
Gesture/Speech
???
Punch Card/Tape
Mouse/GUI

18. HCI VISIONS

19. Courtesy Matt Rettig, CMU

21. Vannevar Bush (1945)
! Memex – As we may think

22. A Memex is a device in which an individual stores all
his books … and which is mechanized so that it
may be consulted with exceeding speed and
flexibility. It is an enlarged intimate supplement to
his memory.
Vannevar Bush

24. Ivan Sutherland
! Sketchpad - 1963

25. The Ultimate Display
The ultimate display would .. be a room within which
the computer can control the existence of matter. A
chair displayed in such a room would be good enough
to sit in. Handcuffs .. would be confining, and a bullet ..
would be fatal.
With appropriate programming such a display could
literally be the Wonderland into which Alice walked.
Ivan E. Sutherland
Sutherland, I.E. The ultimate display. Information Processing 1965, Proc. IFIP
Congress 65, 506-508.

26. Realizing the Vision
! 3D computer graphics
! Physical input devices
! Intuitive interaction
Sutherland, I. E. (1968, December). A head-mounted three dimensional display. In Proceedings
of the December 9-11, 1968, fall joint computer conference, part I (pp. 757-764). ACM.

30. Richard Bolt
! Founder MIT Media Lab
! The Human Interface: Where People
and Computers Meet
! Van Nostrand Reinhold, 1984.

31. Interacting with machines via speech, gesture, and
gaze will not only be possible, but in fact will
become the way most people--most of the time, and
for most purposes--will deal with them.
Richard Bolt

32. Put That There (1979)
! Speech/gesture interaction

35. Ubiquitous Computing

36. “The most profound technologies are those that
disappear. They weave themselves into the fabric
of everyday life until they are indistinguishable
from it.”
Mark Weiser
July 23, 1952 - April 27, 1999

37. ! dadfas

38. Ubiquitous Computing
! Mark Weiser, Xerox PARC
! TAB, Slate, Wall display

39. Ubiquitous Computing
Smart Home Sensor Networks

45. Invisible Interfaces
Jun Rekimoto, Sony CSL

46. Hiroshi Ishii

47. Tangible User Interfaces (Ishii 97)
! Create digital shadows
for physical objects
! Foreground
! graspable UI
! Background
! ambient interfaces

48. i/O Brush (Ryokai, Marti, Ishii)

49. Virtual Reality

50. ! sddg

51. Virtual Reality
! Immersive VR
! Head mounted display, gloves
! Separation from the real world

52. 1977 – Star Wars
Augmented Reality

53. Augmented Reality
! Defining Characteristics
! Combines Real and Virtual Images
- Both can be seen at the same time
! Interactive in real-time
- The virtual content can be interacted with
! Registered in 3D
- Virtual objects appear fixed in space

54. 2008 - CNN

55. Example: Puteko
! http://colarapp.com/

56. Microsoft HoloLens
! Stereo see-through display
! Self contained
! Room scale tracking
! Gesture/speech interaction

57. ! df
From Hiroshi Ishii

58. ! asfas

59. MY VISION

60. ! daf

61. ! sdfs

62. ! sdfgs

63. ! axcvxca

65. Vision
“Using technology to create shared
emotional experiences between users
and so create a deeper sense of
empathy and understanding”

67. Empathic Computing
1. Computing systems that can
understand your feelings and emotions
2. Computing systems that help you
better understand the feelings of others

68. Affective Computing
! Ros Picard – MIT Media Lab
! http://affect.media.mit.edu

69. Appliances That Make You Happy
! Jun Rekimoto – Univ. Tokyo
! Smile detection + smart devices

70. Can we develop interfaces
that allow us to be more
empathetic to others?

71. Empathy Computing Requirements
! Basic Requirements
! Making the technology transparent
! Empathy Definition
! Seeing with the eyes of another
! Hearing with the ears of another
! Feeling with the heart of another

72. Movies are like a machine
that generates Empathy
Roger Ebert

73. Using VR for Empathy
! USC Project Syria (2014)
! VR Experience of Terrorist attack in Aleppo

74. Using AR for Empathy
! Augmented Reality can:
! Remove technology barriers
! Enhance communication
! Change perspective
! Share experiences
! Enhance interaction in real world

75. Wearable Computer
▪ Wearable computer – Ego-Vision view
▪ camera + processing + display + connectivity

76. Current Collaboration on Wearables
! First person remote conferencing/hangouts
! Limitations
! Single POV, no spatial cues, no annotations, etc

77. Sharing Space: Social Panoramas
! Capture and share social spaces in real time
! Enable remote people to feel like they’re with you

78. Technology
! Google Glass
! Image capture, viewpoint sharing
! Remote device (desktop, tablet)
! Immersive viewing, live annotation

79. Capture Emotions
Data Capture
Feature Detection
Emotion Recognition
Emotion Representation
Empathic User Interface
Hardware
User
Interface

80. Sensor Systems/Software
! Different systems developed/evaluated
! SSI framework (emoVoice, etc)
! e-Health, bitalino, arduino bio-sensors
! Wearscript wearable eye-tracker
!
!

81. Wearable Interface
! Google Glass + e-Health + Spydroid + SSI
! Measure GSR, pulse oxygen, ECG, pitch
! Share video and audio remotely
! Represent emotions back to Glass user (4 states)
! !

82. Desktop Interface
! Live video, real time emotion data
! See what sender sees, emotion representation
!

83. Emotion Representation
! How can we show what you’re feeling?
! Tested - Raw data, Visual tinting, Emotion labeling
!
! !
Excited Happy

84. Early Results
! Colour overlay + video stream preferred
! Easier to understand
! Disagreement over best colour coding
! Different emotional response with diff. stimuli
! Scary movie best
! Remote users felt connected to local user
! Understanding of emotions

85. Analysis
Table 2. Correlation between different streaming scenarios
obtained from the feedback of the local and remote users
Experiment
3
(Creating
a
shared
experience
while
watching
an
audio
Visual
Watching
an
audio
visual)
Happy
Sad
Neutral
Excited
Case1
(Only
video
stream)
0.77
0.84
0.85
0.85
Case
2
(Case
1
+
Color
Overlay)
0.92
0.91
0.85
0.98
Case
3
(Case
2
+
Graphical
Analysis)
0.86
0.75
0.85
0.94

86. Looking to the Future
What’s Next?

87. Capturing Space: Real World Capture
! Hands free AR
! Portable scene capture (color + depth)
! Projector/Kinect combo, Remote controlled pan/tilt
! Remote expert annotation interface

88. Remote Expert View

89. Brain to Brain Control
! Rajesh Rao, University of Washington
! First Brain to Brain control

90. System Architecture

91. Scaling Up
! Seeing actions of millions of users in the world
! Augmentation on city/country level

92. AR + Smart Sensors + Social Networks
! Track population at city scale (mobile networks)
! Match population data to external sensor data
! medical, environmental, etc
! Mine data to improve social services

95. Research Challenges
! How to convey emotion?
! How to measure empathy?
! Interface/interaction models?
! How to communicate emotion?
! Scaling up to city/country scale?

96. CONCLUSIONS

97. Take Home Messages
! Find people to travel with
! Equip the vehicle
! Enjoy the journey
! Don’t be disappointed in the outcome
Most Important: Find your vision

98. More Information
• Mark Billinghurst
– mark.billinghurst@hitlabnz.org
– @marknb00
• Website
– www.hitlabnz.org