Mobile Hci L Nigay

Laurence Nigay - laurence.nigay@imag.fr

Outline
Mobile Human-Computer Interaction
• Focus on Human-Computer Interaction
Introductory Course

• EHCI Group
Laurence Nigay • Mobility: a broad concept
University of Grenoble
• Interaction with handheld devices
Grenoble Informatics Laboratory - LIG
Engineering Human-Computer Interaction Group - EHCI

CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France

EHCI group & Mobile HCI
• EHCI Group • “EHCI” for
“Engineering for Human-Computer
Interaction”
– Created in 1990
– 7 professors and assistant-professors
– 15 PhD students
– 4 research engineers and 1 post-doc

EHCI group EHCI group: Research thematic
• Software Engineering for HCI
• EHCI group belongs to LIG
(Computer Science Laboratory of Grenoble)
Interaction
• 165 academic researchers Computer science Supported by
System Human
• 240 doctoral and post-doctoral students contribution Human sciences
• 65 technical and administrative staff

– 24 autonomous research teams • Development of conceptual and technical tools
based on HCI principles : Utility, usability, context
– 4 principal themes • Establishing links between psychology-
• Infrastructure (networks and data) ergonomics and software engineering
• Software (foundations and design models)
• Interaction (perception, action and dialog)
• Knowledge (learning, agent models and web-ontologies)

CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France


EHCI group: Research axes EHCI group: Research axes
• 4 complementary research axes • Models and processes for designing
– Models and processes for designing HCI, HCI, formal testing
formal testing
– Interaction plasticity • Task Hierarchy: a model at runtime
• Adaptation of HCI to interaction context
– Post-WIMP interaction
• Vision based finger tracking and two-handed
interaction …
• Mixed reality
– Modality and multimodality

• Interaction plasticity • Interaction plasticity
• Adaptation of HCI to interaction context • Adaptation of HCI to interaction context
• Context: interaction resources / user preferences /
social context / location etc.

• Interaction plasticity • Interaction plasticity

Control panel

Slides
2. PDA: Partial
migration (only the
1. PC only
1. PC control panel, not the
speaker video)



• Interaction plasticity • 4 complementary research axes
» camnote_shortRepetWeaving.mov – Models and processes for designing HCI,
formal testing
– Interaction plasticity
• Adaptation of HCI to interaction context
Control panel • Vision based finger tracking and two-handed
interaction
• Mixed reality

• WIMP • Vision-based tracking (finger, token)
– Windows: scroll, resize, move
– Icons: representations, drag/drop
– Menus: pop-up, pull-down
– Pointers: mouse, trackball

• Post-WIMP

• Mixed Reality • New technique of menu: multi-stroke
» Video Geonote_26.avi marking menu •Submenus
» Video MenuWave.wmv Previsualization
» Video WaveMenuIphone.mov
– Rapid scan of possibilities
get
•Parent menus:
1) Menu appears
– Always visible
2) Menu is enlarged – Can be directly selected
3) Submenu appears

4) Final item is selected •Visual stability of
the focus of attention



• 4 complementary research axes • Modality and multimodality
– Models and processes for designing HCI, • RAZZLE: Mobile game
– The goal of the player is to collect the pieces of a digital
formal testing puzzle
– Interaction plasticity – The digital puzzle pieces are scattered all over a
• Adaptation of HCI to interaction context modeled playground
• Vision based finger tracking and two-handed
interaction
• Mixed reality

• Modality and multimodality • Modality and multimodality
• Multimodality for the task of collecting a selected • RAZZLE experimental tests (10 subjects / 3 sessions)
puzzle piece – Use of the modalities
– Speech modality Get – All sessions / All subjects

Video RAZZLE
RAZZLEExpe2.dv
– Gestural modality
24% Tactile
45% Gestural
Speech

Video RAZZLE

– Tactile modality
RAZZLEExpe1.dv 31%

• Modality and multimodality
• Modality and multimodality
• RAZZLE experimental tests (10 subjects / 3 sessions)
• RAZZLE experimental tests (10 subjects / 3 sessions) – Subject 4: Use of the modalities per action
– Usage of the modalities per subject (for the three sessions)
100%
90% 100%

80% 90%

70% 80%
70%
60%
Tactile 60% Tactile
50%
Gestural 50% Gestural
40%
Speech 40% Speech
30%
30%
20%
20%
10%
10%
0% 0%
Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3
1 2 3 4 5 6 7 8 9 10



• Modality and multimodality • Modality and multimodality
• RAZZLE experimental tests (10 subjects / 3 sessions) • RAZZLE experimental tests (10 subjects / 3 sessions)
– Subject 9: Use of the modalities per action – The subjects used all of the modalities
(for the three sessions)
100%
=> Usefulness of multimodality
90%
80%
– Individual preferences leading in some cases to
70%
60%
specialization
Tactile • Subject 4: Tactile modality for rotating a puzzle piece
50%
Gestural
40%
• Subject 9: Speech for rotating and collecting a puzzle piece
Speech
30% => Usefulness of multimodality
20%
10%
0%
Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3

EHCI group: Research axes
• Engineering for HCI
– Models and processes establishing links between
• EHCI group
psychology-ergonomics and software engineering • Mobility
– Interaction plasticity

– http://iihm.imag.fr/en/

Mobility: a broad concept Mobility: a broad concept
(1) moving between devices - hot-desking (1) moving between devices - hot-desking
(2) moving within an instrumented (2) moving within an instrumented
environment - intelligent buildings and environment - intelligent buildings and
intelligent appliances intelligent appliances
(3) devices within moving vehicles- (3) devices within moving vehicles-
computers in cars computers in cars
(4) small devices that move with you - (4) small devices that move with you -
PDAs, laptop computers etc PDAs, laptop computers etc

Alan Dix (Lancaster University) Alan Dix (Lancaster University)



Mobility: Mobility:
moving between devices moving between devices

• Nomadic computing

Mobility:
moving between devices Mobility: a broad concept
• In the 1990s, the Internet (1) moving between devices - hot-desking
– a global transport for e-mail (2) moving within an instrumented
and data exchange, environment - intelligent buildings and
– but mostly for people intelligent appliances
at fixed locations (3) devices within moving vehicles-
computers in cars
• More recently (4) small devices that move with you -
– the office extended to employees on-the-go
PDAs, laptop computers etc
("nomadic workers")
=> similar to the case 4 « small devices that move Alan Dix (Lancaster University)
with you »

Moving within an instrumented Moving within an instrumented
environment environment
Smart spaces
How many
people are in
What is the the gym? Is
nearest available the treadmill
conference room? free?

• Examples: Motion Sensor:
– Smart spaces Room 3201 is
Empty What is the
– Augmented classroom / Augmented museum average
temperature
on the second
– Wireless supermarket floor?



• Active Badge
• Olivetti Research Laboratory in
– Experimented with 32 staffs in ORL company in 2 weeks
Cambridge, UK, in 1990-92 – All staff wearing badge that emits IR signal every 15 seconds
• Active Badge: location-aware systems – Intended to aid telephone receptionist
– FIND(name)
» Provides current location of the named badge
wireless
IR network – WITH(name)
Active Badge Badge Sensor Server/Application » Locates a named badge and other badges around the badge
– LOOK(location)
Active Badge » Provides badges near the specified location
Badge Sensor – HISTORY(name)
... » Generates report of the location history for the named badge
Active Badge

• Server • ATT Lab Cambridge: Active Bat
• Network Control
– Polling all sensors on the network – Sentient Computing: a form of ubiquitous
• Representation computing which uses sensors to perceive its
– Builds linked list of (ID, location, time) in time order
• Data Processing environment
– Process large amount of data from the active badge network - A "follow-me phone" which would cause the
• Display Interface telephone nearest the recipient to ring.
– Showing textual information
- Teleporting desktops via VNC just by clicking their
Active Bat near the computer.

Video Sentient.mpg

• ParcTab (Rank Xerox)1995
– http://sandbox.parc.xerox.com/parctab/ • Projet EasyLiving Microsoft

• Video
EasyLiving.mpeg
7 ,8
cm

cm
2 ,4
5 cm
10,



• Location-aware system (within a buidling)
• Context-triggered actions are simple IF-
THEN rules used to specify how context-
aware systems should adapt

Like living in a rule-based expert system
• Examples:
– Smart spaces
– Augmented classroom / Augmented museum
– Wireless supermarket

• Augmented classroom: AR (video)

• Examples:
– Smart spaces
– Augmented classroom / Augmented museum
geometry and mathematics education – Wireless supermarket

Moving within an instrumented
Wireless supermarket concepts
environment
• Augmented surpermarket • The Shopping Buddy
– Based on RFID for products wireless touch-screen device attached to a
– WLan for communication inside the shopping cart
supermarket – scans in items placed in the cart by shoppers
– Intelligent chariots (with PC tablet) – delivers personalized services and incentives
when activated with a frequent-shopper card
– RFID which triggers certain offers and can
help shoppers find anything in the store and
draw a path to find it.



Wireless supermarket Operation Wireless supermarket Operation
• As you shop, you can scan in each item
– keeps a running total of how much you are spending
– eliminates the need to wait in line at the check-out
– you can also check the price of each item before you buy
• As you walk down the aisles
– promotions and paperless coupons “pop-up” on the screen
– Reminders of articles you search
– Directions to articles
• You can place an order at the deli (ex. Half a kilo of ham)
from anywhere
– notification to pick it up (No more taking a number and
standing in line)

Wireless supermarket Operation Wireless supermarket Operation
• The Everywhere Display (IBM) transforms any surface
• The Everywhere Display (IBM) into an interactive computer

• Beamed • A recent MIT project: Sixthsense
from the supermarket ceiling – wearable gestural interface that augments the
• transforms any surface physical world around us with digital infor
into an interactive computer
• On the floor
• On the items

Moving within an instrumented
Mobility: a broad concept
environment
• Technological approach (1) moving between devices - hot-desking
– 1. Augment the user (2) moving within an instrumented
• The user wears or carries a device to obtain environment - intelligent buildings and
information about physical objects. intelligent appliances
• => similar to the case 4 « small devices that move (3) devices within moving vehicles-
with you »
computers in cars
– 2. Augment the physical object
(4) small devices that move with you -
• The physical object is changed by embedding
input, output or computational devices on or within PDAs, laptop computers etc
it.
– 3. Augment the environment surrounding the Alan Dix (Lancaster University)
user



Devices within moving vehicles Devices within moving vehicles

micro

Tactile Surface

Keyboard

Devices within moving vehicles Devices within moving vehicles
• FutureLab of Ars Electronica (Autria)
• INTUITION DGA project
• Augmented Reality
• Augmented cockpit (Rafale)
• Instar.mov video

Devices within moving vehicles Mobility: a broad concept
• INRETS-Lille
(1) moving between devices - hot-desking
(2) moving within an instrumented
environment - intelligent buildings and
intelligent appliances
(3) devices within moving vehicles-
computers in cars

Alan Dix (Lancaster University)



Evolution of personal computers:
Mobility: small devices
a continuum

Desktops Laptops Palmtops Handhelds Wearables

Eye-pod, projection on contact lenses (univ. Washington)

Evolution of personal computers: Evolution of personal computers:
a continuum a continuum
External support + hand(s) 1 hand (at least) Hands-free

Desktops Laptops Palmtops Handhelds Wearables Desktops Laptops Palmtops Handhelds Wearables

Weight
portability/mobility/maniability

Wearable Computer Aim of Wearable Computing
• A small portable computer that is designed to be worn on •To develop new interfaces that
the body during use. mediate (augment, deliberately
diminish, or otherwise modify) non-
• In this, wearable computers differ from PDAs, which are computer activities
designed for hand-held use.
• Wearable computers are usually either integrated into
the user's clothing or can be attached to the body • Without interfering with the user's
through some other means, like a wristband. everyday tasks
• They may also be integrated into everyday objects that
are constantly worn on the body, like a wrist watch or a • The design of wearable computers is
hands-free cell phone. still a topic of research, and a variety
of user interfaces are being proposed

University of Waterloo CA University of Waterloo CA



Evolution of
Application domains
Wearable Computers
• Medical domain
– Medical
monitoring
device
– (blood pressure,
temperature)

• Military domain Amon: ICT project

GlucoWatch

Application domains Wearable Computing
• Mobile workers
• Key characteristics
– Hands free
• Constant: Always ready. It is always on and always
running.
• Personal: Human and computer are inextricably
intertwined.

Mobility: Huge variability of
devices Mobility: small devices
• Usage spaces
of wireless devices
Desktops Laptops Palmtops Handhelds Wearables

Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002



Mobility: small devices Mobility: small devices
• Information • M-Commerce
– Personal Information Management – Electronic money, e-coupons
– Business information: specific professionals, such as a drug
interaction reference for doctors
• Self-Enhancement
– Active assistance
• Ways in which the device extends one’s normal capabilities.
(Finland: cell-phones "Kanny » extension of the hand »)
– Health and Safety
• Contact with emergency response teams or family
– Education

Mobility: small devices Mobility: small devices
• M-Commerce • Usage spaces
– Electronic money, e-coupons of wireless devices
• Relationships
– Core competence of phones to connect people • Huge variety
• Entertainment of applications
– Music, games
– The mobile game market tends to follow the more
mature console/PC game market
– Game performance must match or exceed the users’
expectations

Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002

Variety of applications Variety of applications
• Device: designed for a certain user task
• Device: designed for a certain user task
– impractical to have several different information
Sony – Reader-ebook PRS-505
Screen has no backlight or flicker
appliances for different purposes
to give the experience of reading a paper page
• Device: « Swiss-army-knife » useful in many
different situations
• + Different devices are appropriate for – HCI complicated / non intuituive
different tasks and this should also guide the
application design • the question of information appliance or
• - Impractical to have several different information Swiss-army knife still remains unanswered
appliances for different purposes



Mobility: Huge variability of
Mobility: a broad concept devices
(1) moving between devices - hot-desking
(2) moving within an instrumented Desktops Laptops Palmtops Handhelds Wearables
environment - intelligent buildings and
intelligent appliances
(3) devices within moving vehicles-
computers in cars

Alan Dix (Lancaster University)

Objectives
• EHCI group
• Mobility
• Interaction with handheld devices

From: MobileHCI 07 N. Savio J. Braiterman
Design Sketch: The context of Mobile Interaction

Challenges Variability of usage context
• Key issue: Transparency/Usability of
mobile devices
• Challenges for HCI
– Variability of platforms
– Variability of usage context
– Limited interactionnal resources

From E. Kaasinen: User acceptance of mobile services - value, ease of use, trust
and ease of adoption



Challenges Challenges
• In mobile computing: the problem space
• In conventional HCI: the problem space is is unbounded
bounded
– The target user (e.g., an office clerk)
– accomplishes a well-defined set of tasks (e.g., writing a report)
– using a fixed class of devices (e.g., a workstation)
– in a predefined set of environments (e.g., in the office) Task User

Env. Platform

Task User

Env. Platform

Introduction: how to address this Handhelds: diversity, but in
problem? common …
• Approach1: to develop the systems on a case per case basis (ad- • Small screens
hoc manner)
• Limited input capabilities
– Development cost and maintenance: very high!
– Consistency problem between the various versions
• No standards yet

• Approach2: to improve the methods and tools in order to support • Recommandation
– Portability • Apply the foundations of HCI - e.g., the method
– Reusability
• Create without re-inventing the wheel: draw your
– Modifiability
design on existing solutions (in order to support
– Scalability in terms of
consistency across applications/brands) and invent
• Computing resources new design solutions to overcome current limitations
• Services
• Interaction techniques (multimodality, UI for small and
large devices)
=> Plastic UI

Objectives Focus on HCI

Gulf of Evaluation
Mobile Device Goals

Gulf of Execution

From: MobileHCI 07 N. Savio J. Braiterman
Design Sketch: The context of Mobile Interaction



The design principle of
The Design of Everyday Things
transparency
• How to design things that real people can
a design principle is to try to make systems transparent so
that people can understand them better and know what to
use: successfully, easily, pleasantly
do • 7 stages of performing an action
• NOT to be understood as literal
• Conceptual models – the device should
• useful feedback explain itself
• easy to understand • Visibility and Affordances
• intuitive to use
• clear & easy to follow instructions • Mappings
• context sensitive guidance of • Feedback
how to proceed when stuck

From Prof. Steve Whittaker From Prof. Steve Whittaker

Seven Stages of Action Gulfs of Execution & Evaluation
Goals
Goals
Evaluation of
Evaluation of Intention to act the interpretations
Intention to act the interpretations
GULF OF EXECUTION
Interpreting
Interpreting the perception
Sequence of actions
the perception
Sequence of actions GULF OF EVALUATION

Execution of the Perceiving the state Execution of the Perceiving the state
action sequence of the world action sequence of the world

The World The World


Focus on HCI The Gulf of Execution
• Does the system provide actions that
correspond to the user’s intentions?
Gulf of Evaluation
• The difference between intentions and
Mobile Device Goals allowable actions is the Gulf of Execution

Gulf of Execution

From Prof. Steve Whittaker



The Gulf of Evaluation The Seven Stages as Design aids
• Does the system provide a physical How easily can the user:

representation that can be readily Determine the function
perceived and interpreted in terms of the of the system?

user’s intentions and expectations? Tell what actions Tell if the system is
in the desired state?
• The Gulf of Evaluation reflects the amount are possible?

of effort that the person must exert to Determine a mapping from
Determine a mapping from
interpret the physical state of the system intention to physical action? system state to interpretation

and determine how well the intentions
have been met. Perform the action? Tell what state the
system is in?


… principles for good design Mobile HCI: inputs/outputs
• Visibility – by looking, users can tell the state of the device and
possible actions
• Conceptual model – operations are presented consistently; users
can easily understand “how the system works”
• Mappings – it is possible (easy) to determine the relationship
between actions and results, between controls and their effects, and
between the system state, and visible indicators
• Feedback – users get full and continuous feedback about the
results of actions

From Prof. Steve Whittaker

Mobile HCI Mobile HCI
• A very active domain
• Approach1: to develop the systems on a case per case basis

• Approach2: to improve the methods and tools in order to support
– Portability • Plasticity (approach 2)
– Reusability
– Modifiability • Innovative interaction techniques
– Scalability in terms of (approach 1)
• Computing resources
• Services – Innovative interaction techniques
• Interaction techniques (multimodality, UI for small and – Multimodality
large devices)
=> Plastic UI



Plasticity of an interactive
Plasticity
system
• Capacity of the system to adapt to the
• Materials: capability to change their own context of use while preserving their own
shape while preserving usage, they do not utility and usability
break (within some limits, of course)
• Animals, plants: they adapt to the • Context of use …
environment to survive • Adaptation …
• Utility and Usability …
• By analogy in HCI …

Context of use Plasticity of an interactive
system
• Context: it’s complex! Generally speaking, it is an information space
– that is shared between actors, dynamic, structured, and
– that serves interpretation (by the actors)

• There is “no context without context” -> context of use
• Capacity of the system to adapt to the context
• Context of use: an information space of use while preserving their own utility and
– that is shared (between the software components of the interactive system and
between the system and the users), dynamic, and structured into 3 sub-spaces: usability
• The users
• The platform (from an elementary platform such as a mobile phone to a
cluster of elementary devices)
• The physical environment (location, light-sound-heat and social conditions)
– that supports the adaptation process to preserve the system utility and usability • Context of use …

• Adaptation …


Two means for UI adaptation UI remolding (reshaping)
• Example: Plastic Clock

• Remolding (reshaping)

• Redistribution



UI remolding (reshaping)
UI remolding (reshaping)
• Physical Presentation level (PP)
NSButton :
Contrôleur de

Nabaztag dialogue

Button :
ANF Présentation

Interaction

JButton :
NF Interaction
Interaction

The classes of widgets are maintained but their
behaviour (possibly, their spatial and temporal
layout) has changed
PC

PDA

UI remolding (reshaping): may UI remolding (reshaping)
cover multiple levels of abstraction
• Logical presentation level (LP) • Dialogue Control level (DC)
Label + ComboBox : Contrôleur de
dialogue
Tasks are maintained but their ordering has changed
DC Modification -> LP modification -> PP
Contrôleur de
dialogue
Contrôleur de
Contrôleur de

modification
dialogue ANF dialogue Présentation
Présentation

ANF
Label + T extField :
Présentation NF Interaction
Interaction

NF Interaction
Interaction

Substitution of widgets
LP Modification -> PP Modification

UI remolding (reshaping) UI remolding (reshaping)

• Functional Core Adaptator level (FCA) • Functional Core level (FC)
Contrôleur de
Contrôleur de
Book (workstation) Contrôleur de
Contrôleur de

ANF
dialogue
dialogue
- title ANF
dialogue
dialogue
The nature of the services of the functional core has
changed
ANF ANF
ANF
Présentation
- authors ANF
Présentation

- summary
FC modification ->FCA Modification -> DC modification
NF Interaction NF Interaction

- keywords
… -> LP modification > PP modification

Book(Handheld)
- title
- ﬁrst author
-keyword

Tasks and domain-dependent concepts may be added/suppressed
FCA Modification -> DC modification -> LP modification > PP modification



UI Redistribution
UI Redistribution
• Rekimoto’s “pick
and drop”
• The UI uses interaction resources that are distributed over multiple elementary
platform (distributed UI) – Distribution is static
• UI redistribution may be static or dynamic
– Distribution at the
workspace level
• Dynamic redistribution => on-the-fly migration of the UI
– No remolding
– Migration may be total: the UI releases (frees) all of the interaction resources
currently used and migrates entirely to other interaction resources

– Migration may be partial: only portions of the UI migrate -> what is the
• i-LAND
granularity for UI migration? – Distribution is dynamic
– Distribution at the pixel
level
– No remolding
http://www.ipsi.fraunhofer.de/ambiente/english/projekte/projekte/i_land.html

UI Redistribution Plasticity of an interactive
2. Arrival of the PDA: UI partial migration
system
• CamNote
• Capacity of the system to adapt to the context
of use while preserving their own utility and
usability

Remolding of the
control panel • Context of use …
at the FCA level
1. PC only: centralized UI
• Adaptation …

3. Distributed UI (PC and PDA) +
remolding

Usability of an interactive
Utility of an interactive system
system: many faces!
• Capacity of the system to provide the target users with
the appropriate functions/services (not less, not more) • To make it simple: Capacity of the system to allow the
target users to access and manipulate the services
• Appropriate = in conformity with users’need (not with provided by the interactive system in conformity with
designers’ and developers’ needs!) their cognitive, motor and perceptual capabilities
– Cognitive is the hard part ! Knowledge, pleasure,
• How to elicit the appropriate services? emotion, mood come into play!
– Talking to users is not a luxury, it is mandatory!
– Questionnaires, focus group, evaluation, rapid • How to develop a usable system?
prototyping (low-fidelity), observation of users in – Use theory-based knowledge (e.g., Theory of action,
their daily activities, etc. Fitts law)
– In short: user-centered design – Apply ergonomics rules
– Design, evaluate with the end-user and iterate



Utility and Usability are not intrinsic to a product!
It depends on the context of use Mobile HCI
• Utility and usability depend on the “values”
that the user associates to the system • A very active domain

• The “values” depend on the context of use • Plasticity
• Innovative interaction techniques
– Innovative interaction techniques
• Example: possible values for a heating control
system that may depend on the context of use – Multimodality
– Money saving
– Comfort

Interaction techniques:
Interaction techniques
small screen
• Small screens • Fisheye technique
• Limited input capabilities – Allows the user to see
the details
without loosing
the context of
the details
(Furnas:
seminal work)

– Video
– datelens-video-web-server.wmv

Interaction techniques: Interaction techniques:
small screen small screen
• A peep hole on a large information space [Ka-Ping Yee, CHI2003]
• Problem: loss of the context - • Microsoft Research Lab
• Halo tends to alleviate this problem
• Projet Halo
Halo to make observable the information that
is of interest to the user

• Problem +



• Hybrid technique: thumbnail+editing
Video Halo.avi • Collapse-to-Zoom
– Pages that are to big to fit the screen are presented as a “summary
thumbnail”
– Thumbnails are editable (can be suppressed/enlarged, etc.)

Solution expand

collapse

[Baudisch CHI 2004]

• Hybrid technique: thumbnail+editing • Menu techniques
• Collapse-to-Zoom
– Space on screen
– Pages that are to big to fit the screen are presented as a “summary
thumbnail” – No keyboard for shortcuts (novice mode)
– Thumbnails are editable (can be suppressed/enlarged, etc.) – The best way to interact is to use only one-hand

Grey lines indicate the
existence of collapsed areas Circular menus

(advertisement)

video collapsetozoom

• Marking menus • Marking menus
• Multi-stroke marking menus
• Advantages • Instead of considering a spatial compound stroke,
– Circular design • Multi-Stroke menus introduce a serie of simple strokes
– Fluid transition
– Scale independance
• Require less physical input space
in novice & expert modes
– A submenu is displayed on top of
• Limitations its parent menu
– Screen space requirement
– Overlapped marks
– Number of commands
• Error rate in expert mode • Increase accuracy in expert mode
• Ambiguous marks in expert mode
• Increase the number of items
– No ambiguous gestures in expert mode

video `Multistroke



Wave menus Wave menus
• Few physical input space
• A submenu is displayed at the
center of the menu system. In
order to remain visible, its parent
menu is then enlarged to surround Novice Mode
this submenu
• In expert mode, Wave Menus (1)
work exactly the same way as
Multi-Stroke Menus (2)
Expert Mode
video WaveMenu

Wavelet menus: extension on Iphone
Wavelet menus: extension on Iphone

•The Wavelet menu appears centered around the
contact point.
•By drawing a stroke towards the desired item, the
first level is enlarged permitting progressive
appearance of the submenu.
•A second stroke selects an item in the submenu.

• Menu techniques
Wavelet menus: extension on Iphone – Space on screen
– The best way to interact is to use only one-hand

Long lists management: the linear list appears in
the center of the Wavelet menu and is surrounded video Leafmenunovice Leafmenuexpert
by its parent menus. ( video Wavelet.mov)



• Menu techniques
• Non graphical techniques => Eye-Free interaction
– Space on screen
– Tactile & Sound feedback
– The best way to interact is to use only one-hand
– ArchMenu, ThumbMenu • Spatialized sound

Meeting at 10

Soundbeam
Neckset

Vibro-Tactile feedback
Shoogle senses data (and makes sense of them!) [ Univ. Vibro-tactile feedback
Glasgow 07] Shape-change feedback: the dynamic knob [Deutsche Telecom
Sensors: the Shake includes an accelerometer, gyroscope, et Univ. Postdam]
magnetometer, vibro-tactile display communicating via
bluetooth
Expresses the existence of messages, remaining resources
(e.g., the battery level) using a dynamic model inspired from
mechanics (balls anchored with springs bouncing against walls)

Interaction techniques Text entry
• Small screens
• Limited input capabilities



Text entry Keyboard: Text entry

Text entry Text entry
• LucidTouch: • LucidTouch:
a see-through mobile device, pseudo-tranparency a see-through mobile device, pseudo-tranparency

Keyboard: entry Text entry
• Oulu University (Finland) • Hand writting + Movements



Pointing Techniques Pointing Techniques
• The best way to interact is to use only • “Offset Cursor” [Potter, Shneiderman 1988)]: how to replace the stylus
one-hand •
with the finger without hiding the target
Selection technique: finger touches the screen, display the cursor at 1/2
• Using the thumb inch over the finger, move the finger to bring the cursor over the target,
remove finger

Target Finger hides the target Finger moves to bring the cursor
Gestures of the thumb Cursor is offset over the target
MicroRolls from the target

Pointing Techniques Pointing Techniques
• Shift [P. Baudisch 07, MicroSoft]: improves the “offset cursor” technique
Occlusion

Occlusion of targets Occlusion of the interface

Border accessibility
Thickness of screen
Areas more difficult to reach [Parhi 06] edges

Accuracy

Video: shiftMini.m4v Larger than the stylus Prediction hard

Pointing Techniques New input Techniques
• Tap-Tap Anne Roudaut (AVI 2008) Video: taptapMini.m4v
• Temporal multiplexing strategy
• Camera phone based motion sensing
• 1ST tap: Specify the focus area
Uni. of California, Berkeley
• 2nd tap: Select the target at a larger scale & IBM
Video: Phone3DMvt-UIST06.mov

Target to select 1st tap Magnified popup 2nd tap



New input Techniques New input Techniques
• Movement-based UI – 3D gesture
• Tilting using accelerometers [Rekimoto 1996, Harrison&Fishkin 1998] • Movement-based UI – 3D gesture
• Embodied UI: Rank Xerox

Photos album

Video: RocknScroll.mov

• Embodied UI: Rank Xerox
• Embodied UI
• Prototyping tool
for embodied UI
– Stanford HCI group

–Video: MapPDADesignplatform-UIST06.mov

• RodDirect: the stylus as a peripheral device (2 degrees of
freedom) + the second hand • Distorsion: bendable display
• Two types of stylus movement in the stylus holder are • The Gummi interface: just bend it! (Sony, Schwesig CHI 2004)
associated with parameters
– (1) Twist (Rotation) (2) Push/Pull (Sliding)

Map Browser Application Switching Scheduler
JAIST: Japan Advanced Institute of Science and Technology, JAPAN
Video: RodDirectDemo.mpg



New input Techniques Mobile HCI
• Gummi: zoom-in a subway station


• Plasticity
• Innovative interaction techniques
• Alpha-blending of 2 graphics layers – Innovative interaction techniques
– Multimodality

Which actor performs modalities
Multimodal Interaction
selection ?
• Multimodal systems
– Multi-Sensori-Motor Systems
– extend the sensori-motor capabilities of
computer systems = mobile devices
Designer User System
• Multimodal interaction makes use of several
input and/or output (feedback) modalities in Information Selection of Multimodal
to be one or several modalities
interacting with a computer system. conveyed
Expression
– Examples of modalities: manual gestures, gaze,
touch, speech, head & body movements Actor of the selection
– Modality: human sensory channel, different
representation modality, or different input method

Multimodality selection Multimodality Adaptability
Go to the middle
of the message

No adaptation
Gestural modality Speech
Adaptability

Adaptivity Selection by Selection by Selection by
the designer the user the system

Embodied modality Direct manipulation


Mobile Hci L Nigay

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Mobile Hci L Nigay

Semelhante a Mobile Hci L Nigay (20)

Último

Último (20)

Mobile Hci L Nigay