8. 8
“As We May Think”
• Published in the Atlantic Monthly in 1945!
• What will the computer of the future look like?
– Wearable cameras for photographic records
– Automatic transcripts of speech
– Memex
– Trails of discovery
– Direct capture of nerve impulses
• “publication has been extended far beyond our present ability to
make real use of the record”
• 인간이 이루어 놓은 지식을 한데 보관하는 연구의 필요성 언급
• 여러 세대에 걸쳐서 축적한 방대한 양의 자료들을 쉽고 빠르게 검색하여
이용할 수 있다면 인간의 지성은 크게 증가할 것이라고 주장 → Memex
장치 고안
9. 9
“As We May Think”
Memex
• Memex: Memory + Index
• Store all personal books, records, communications
• Items retrieved through indexing, keywords, cross references,...
• Can annotate text with margin notes, comments...
• Can construct a trail through the material and save it
• Acts as an external memory
10. 10
“As We May Think”
Memex
• 사진과 같은 정보들을 스캐닝하여 저장할 수 있는 스캐너(왼쪽)
• 정보를 검색할 수 있는 타자기 (오른쪽)
• 검색된 결과를 출력해 주는 스크린 (가운데)
• 원통형의 장치 → 데이터를 저장하는 마이크로필름(micro film): 이
부분에 미세하게 저장된 데이터를 찾아 화면에 투사, 사용자가 볼 수
있게 한다.
15. 15
Sketchpad
Ivan Sutherland (1962)
• Ivan Sutherland
– Sketchpad는 memex에 영향을 받아 만들어졌으며, 화면 위의
geometry shape과 line을 라이트펜을 이용하여 조작하게 하였다 →
최초의 graphical user interface
– 펀치카드, 키보드 등을 이용하여 입력하던 기존의 방식에서 벗어나,
화면에 오브젝트를 그리고, 크기를 변화하고, 집어서 움직이고
삭제하는 등의 인터랙션을 제공
– 포인팅디바이스를 사용 → 최초의 direct manipulation interface (B.
Shneiderman, 1983)
16. 16
Sketchpad: “Direct Manipulation”
• Direct manipulation features:
– Visibility of objects
– Incremental action and rapid feedback
– Reversibility
– Exploration
– Syntactic correctness of all actions
– Replacing language with action
• Term coined by Ben Shneiderman1
1 Shneiderman, B., Direct manipulation: A step beyond programming languages, in IEEE Computer, 1983, August, 57-69.
18. 18
Beyond Sketchpad
• EverybodyLovesSketch
– Bae, Seok-Hyung, Ravin Balakrishnan, and Karan Singh. 2009. “EverybodyLovesSketch: 3D
Sketching for a Broader Audience.” In Proceedings of the 22Nd Annual ACM Symposium on
User Interface Software and Technology, 59–68. UIST ’09. New York, NY, USA: ACM.
doi:10.1145/1622176.1622189.
– Sketch-based 3D Modeling
20. 20
Invention of the Mouse
Doug Engelbart (1963)
Douglas Carl Engelbart
(January 30, 1925 – July 2, 2013)
21. 21
Douglas Engelbart
• The Problem (early ‘50s)
“...The world is getting more complex,
and problems are getting more urgent.
These must be dealt with collectively.
However, human abilities to deal
collectively with complex / urgent
problems are not increasing as fast as
these problems.
If you could do something to improve
human capability to deal with these
problems, then you'd really contribute
something basic.”
...Doug Engelbart
22. 22
Douglas Engelbart
• The Vision (Early 50’s)
…I had the image of sitting at a big CRT screen with all kinds
of symbols, new and different symbols, not restricted to our
old ones. The computer could be manipulated, and you could
be operating all kinds of things to drive the computer
... I also had a clear picture that one's colleagues could be
sitting in other rooms with similar work stations, tied to the
same computer complex, and could be sharing and working
and collaborating very closely. And also the assumption that
there'd be a lot of new skills, new ways of thinking that would
evolve "
...Doug Engelbart
23. 23
Douglas Engelbart
• A Conceptual Framework for Augmenting Human Intellect (SRI
Report, 1962)
"By augmenting man's intellect we mean
increasing the capability of a man to approach a
complex problem situation, gain comprehension
to suit his particular needs, and to derive
solutions to problems.
One objective is to develop new techniques,
procedures, and systems that will better adapt
people's basic information-handling capabilities to
the needs, problems, and progress of society."
...Doug Engelbart
24. 24
Invention of the Mouse
Doug Engelbart (1963)
• Mouse
– Stanford Research Institute
– 가장 대표적인 2차원 위치 입력장치
– 초기에는 두 개의 wheel 이 x, y 축을
움직임
– 볼(ball) 이 추가되어 wheel 을 움직이는
구조로 변화
– 최근에는 광학 마우스로 발전
최초의 마우스에 대한 특허
US 3,541,541 (X-Y position indicator for a display system)
25. 25
HCI’s First User Study1
A comparative evaluation of…
Mouse Joystick
Lightpen Grafacon Knee-controlled
lever
1 English, W. K., Engelbart, D. C., & Berman, M. L. (1967). Display selection techniques for text manipulation. IEEE Transactions on Human Factors in Electronics, HFE-8(1), 5-15.
26. 26
HCI’s First User Study
• Experiment Design
– Participants: 13
– Independent variable
• “Input method” with six levels: mouse, light pen, Grafacon, joystick
(position-control), joystick (rate-control), knee-controlled lever
– Dependent variables
• Task completion time, error rate
• (Note: task completion time = access time + motion time)
– Within-subjects
– Task:
• Press spacebar, acquire device, position cursor on target, select
target
27. 27
HCI’s First User Study
• Results (1)
Notes:
1 Access time with the knee-controlled lever was zero (since the device is always “acquired”).
2 Light pen use is fatiguing, since the user’s arm is held in the air in front of the display.
36. 36
Henry Mill - The first typewriter patent
1714
William Austin Burt - Typographer, used
"type bars"
1829
Augsust Dvorak - Dvorak simplified
keyboard is designed.
1936
Charles Thurber - used moveable carriage
1843
Malling Hansen - "Writing Ball", the first
commercial typewriter
1870
Latham Scholes - the "Sholes-Glidden" was
first placed on the market
1874
QWERTY is settle down
1887
1714 1756 1798 1840 1882 1924 1966
Palm - Graffiti
text entry
1996
T9 predictive
text entry
1998
Twiddler is
invented
2004
ShapeWriter
(similar with
Swype) is
invented
2004
History of Keyboards
37. 37
History of Keyboards
• Typewriters
Burt’s typewriter
(used type bars)
M. Hansen’s typewriter
Jamming
the first commercial (but not success)
“Carroll and Rosson: Paradox of the Active Use”, http://www.cwa.mdx.ac.uk/bis4407/lec12Examples/carroll.html
Sholes-Glidden typewriter
the “father” of the typewriter
Origin of QWERTY
To prevent to jam between levers
Assign a far distance to frequently
consequent characters
Dvorak layout Vs.
Production Paradox
(Carroll and Rosson, 1987)
38. 38
History of Keyboards
Newton (Apple) vs. Pilot (Palm)
Handwriting Recognition vs. Graffiti unistroke text entry
Twiddler
Early stage wearable device
39. 39
History of Keyboards
Predictive Text Entry
T9
Predictive Text Entry
Swype
Funding: $15.6M (150억원)
Sold: Nuance, $102.5M (1,025억원)
Founded: 2002, Employees: 5 (in CrunchBase)
41. 41
History of Mouse
Trackball - Royal Canadian Navy
1949
Trackball Mouse - Telefunken Rollkugel
1968
Mouse - D. Engelbart
1963
Optical Mouse - late 1970s
Late 1970
Commercial Mouse - Xerox(for 8010 system)
1981
Wireless Mouse - Logitech
1984
1960 1970 1980 1990 2000
Laser Mouse -
Logitech(MX1000)
2004
Consumer Mouse - Apple(for LISA)
1983
1949
3-button Mouse – SRI (D. Engelbart and B. English
1972
42. • “A visual history of the computer mouse”, http://www.dvice.com/archives/2011/04/a_visual_histor.php
• “A Brief History of the Personal Computer Trackball”, http://www.trackballworld.com/trackball-history.html
• “oldmouse.com”, http://www.oldmouse.com/
• “Mouse (computing)”, http://en.wikipedia.org/wiki/Mouse_(computing)
First Trackball Mouse “Rollkugel” (1968)
42
History of Mouse
First Trackball (1952)
Tom Cranston and Fred Longstaff (Canadian)
Canadian Navy’s DATAR(Digital Automated Tracking and Resolving) System
used a 4" Canadian duck pin bowling ball and weighed several pounds
First Mouse (1963)
Douglas Engelbart (Stanford Research Institute)
2-wheels and 1-button
Telefunken (Germany)
for TR66 computer system
Mystery Wheel Mouse (1968)
Douglas Engelbart and Bill English (Stanford Research Institute)
2-wheels and 3-button
First Wireless Mouse (1984)
Logitech (U.S.)
IrDA(1984) Radio (1991, Cordless MouseMan)
Xerox PARC (1971~)
Bill English moved from SRI to PARC, and his group
developed PARC’s first mouse
(Left) Bill’s track-ball mouse, (right) first commercial
mouse Star-2 optical mouse (center also)
First Laser Mouse (2004)
Logitech, MX1000
LED Laser
43. 43
Special Mice
6-DoF Mouse
Nintendo Wiimote (2006) Logitech MX Air (2007) Hillcrest Lab. Scoop Pointer (2011)
6-DoF (x, y, z,
roll, pitch, yaw) 인식
Wiimote 성공 이후
다양한 6DoF 개발됨
Multi touch Mouse
Apple Magic Mouse (2009) MS Touch Mouse (2011) Apple Magic Trackpad (2010) Logitech Wireless Touchpad (2011)
더 다양한 상호작용 입력 가능
44. 44
Future Mice
• Mouseless
– Mistry, Pranav, and Patricia Maes. 2010. “Mouseless.” In Adjunct Proceedings of the
23Nd Annual ACM Symposium on User Interface Software and Technology, 441–42.
UIST ’10. New York, NY, USA: ACM. doi:10.1145/1866218.1866261.
46. 46
Display - HMD
Sensorama
Prototype built in 1962
• Displayed stereoscopic 3D images in a wide-frame view
• Supplied stereo sound
• Simulated wind, body tilt and could produce aromas
Sword of Damocles
Prototype built in 1968
• First virtual reality head-mounted display
• Supported head tracking
• Displayed 3D graphics using wireframes
47. 47
Display - HMD
The EyePhone
Released in 1984
• First consumer head-mounted display
• Displayed color graphics
• Used hand gestures to interact with a virtual world
Nintendo Virtual Boy
Released in 1995
• Marketed as the first portable game console that
could display “true 3D graphics”
• Monochromatic display
• Used oscillating mirrors to create a 3D effect
48. 48
TOSHIBA Headgear
Released in 2006
• 360-degree image on a dome-shaped screen
• Wearing an old-style TV set on your head
• Toshiba has no plans yet to turn the helmet into
a commercial virtual-reality product.
Display - HMD
49. 49
Oculus Rift
Prototype released in 2011
• Uses lenses and a split screen to display 3D
graphics
• Tracks movement using gyroscope and
accelerometer
• 110 degree field of view
Display - HMD
52. 52
History of HCI
1. Past
2. Current and Future
3. HCI based Startups
53. 53
Current and Future Technology
• NUI
• Wearable Computing
• Augmented Reality
54. 54
NUI: Natural User Interface
Four Computing Eras*
C o m p u t i n g P a r a d i g m
C o m p u t i n g I n t e r f a c e
* Microsoft Research Cambridge, “Beging Human”, http://research.microsoft.com/en-us/um/cambridge/projects/hci2020/
55. 55
NUI: Natural User Interface
• 1970년대 MIT Media Lab.의 Wearable Computer
연구에 참여한 Steve Mann이 자신의 “Metaphor-free
Computing” 연구를 “Natural Interface”로 언급하며
시작됨*
• 2008년 MS의 Surface Computing의 Principal User
Experience Director였던 August de los Reyes가 “Web
Directions South 2008” 학회에서 “Predicting the
past”라는 세션에서 언급하면서 더움 알려짐**
• 현 IT 산업에서 “Natural”의 의미는 마우스나 키보드 같은
인공적 제어장치 없이도, 사람의 자연스런 감각/행동/인지
능력을 통해 직접 교감하는 방식을 의미함
* Steve Mann, “Metaphor-free Computing”, Intelligent Image Processing, John Wiley and Sons, 2001
** August de los Reyes, “Predicting the past”, http://www.webdirections.org/resources/august-de-los-reyes-predicting-the-past/
57. 57
Motion Controllers using Sensors
3축가속도 센서, Gyro 센서,
지자기 센서, Color LED Tracking
IR Tracking, 3축 가속도 센서
3축 가속도 센서, Gyro 센서
자기 센서
IR Tracking, 3축 가속도 센서
58. 58
2D Camera based
Skin-Color based
Movement based
[1] http://blog.cvisionlab.com/
[2] E. S. Nielsen, et. al., “Hand Gesture Recognition for Human-Machine Interaction”, Journal of WSCG, Vol. 12, No. 1-3, ISSN 1213-6972
[3] S. M. Benoit, “Monocular Optical Flow For Real-Time Vision Systems”, http://www.cim.mcgill.ca/~apl/Research/OpticalFlow/benoits.htm
Mobile NUI
63. 63
NUI: Leap Motion Family
Competitors
• announced at May 21, 2012
• Price: $99.99
• 3D in-air 추적(tracking), 인식(recognition)
• 0.01mm 의 정밀도로 객체 추적 제공
• 개발 플랫폼 제공
• 빠른 인식 속도
DUO
DIY 3D sensor kit
Funding unsuccessful ($62,529 / $110,000)
Nimble VR (Link)
Designed for VR Input
Funding successful ($103,922 / $62,500)
65. 65
Wearable Computing
Display Always
Sense Your Body Log Your Life
Wearable
Computing
66. 66
Display Always
스마트 Glasses
Google Glass META SpaceGlasses Vuzix Smart Glasses
Sony Smart Watch I’m Watch
스마트 Watch
Samsung Galaxy Gear Qualcomm toq
Wearable Computing 기기들은 항상 사용자의 신체에 부착되어
있기 때문에 Smart Phone에 비하여 접근성이 높다. 이러한 특징을
극대화 하기 위여 사용자에게 필요한 정보를 상시적으로 제공하는
Always Display 기기로써 기대되고 있다.
67. 67
Sense Your Body
Wearable Interface
Wearable Computing 기기들은 항상 사용자의 신체에 부착되어
있기 때문에 사용자의 신체의 변화를 가장 빠르고 정확하게 수집할
수 있 다 . 이 러 한 특 징 을 이 용 하 여 사 용 자 의 신 체 정 보 를
적극적으로 활용한 다양한 기기들이 등장하고 있다.
Fitness Devices
Fitbit Flex / Jawbone Up Nike FuelBand
Authentication
Misfit Shine
MYO Interaxon Nymi
68. 68
Log Your Life
Fitness Devices
Wearable Computing 기기들은 항상 사용자의 신체에 부착되어
있기 때문에 사용자의 경험들을 가장 근접하게 공유할 수 있다.
이를 활용하여 사용자가 체험하는 오감의 경험들을 기억하고
저장하기 위한 여러 기기들이 등장하고 있다.
Memoto – Video logging
Kapture – Audio logging
69. 69
Wearable Computing
• Mistry, Pranav, Pattie Maes, and Liyan Chang. 2009. “WUW - Wear Ur
World: A Wearable Gestural Interface.” In CHI ’09 Extended
Abstracts on Human Factors in Computing Systems, 4111–16. CHI EA
’09. New York, NY, USA: ACM. doi:10.1145/1520340.1520626.
70. 70
Augmented Reality
Wearable (HMD) Mobile Pervasive
Meta Space Glass
HMD의 대가인 Steve Mann과
Steven Feiner 와 합작
Display 뿐만 아니라, Interaction
쪽에도 상당한 개발
Atheer labs
https://www.atheerlabs.com/
Google Project Tango
Google의 Mobile 3D
Recognition Platform
증강현실 뿐만 아니라 NUI, 로봇
등의 분야에서 사용 예상
Microsoft IllumiRoom
프로젝션을 이용하여 현실
환경에 부가적인 정보를 제공
현실 환경에 몰입형 증강현실
구현
74. 74
History of HCI
1. Past
2. Current and Future
3. HCI based Startups
서종훈 (Jonghoon.seo@msl.yonsei.ac.kr)
강나형 (nahyung.kang@msl.yonsei.ac.kr)
양윤식 (yoonsikyang@msl.yonsei.ac.kr)
75. 75
Human Computer Interaction Startups
• https://angel.co/human-computer-interaction
76. 76
3Gear Systems
• At 3Gear, we believe that the future of human computer interaction isn't a better mouse or a bigger
touchscreen.
• We're building technology that engages the full dexterity of the human hand for gestural
computing.
• Our business model is to be the fastest and easiest way to add gestural control to any product,
whether that's a laptop, a car or a refrigerator.
77. 77
Tobii Technology
• Tobii creates market leading gaze interaction and eye tracking solutions for computer interaction,
integration as well as research and tests.
78. 78
Ubi Interactive
• ubi provides a plug-and-play gesture control solution that can turn any existing display or surface
into a 3D touch-screen.
• We are one of the eleven startups in the first class of Microsoft Accelerator for Kinect, powered by
Techstars.
79. 79
Tactus Technology
• Tactus Technology makes touch-screens tactile, providing real physical buttons that dynamically
appear from the surface of any touchscreen.
