Presentation of "Distributing User Interfaces" as invited talk by Prof. Dr. Ricardo Tesoriero on the 4th Workshop on Distributed User Interfaces in the 14th International Conference on Web Engineering ICWE 2014
Distributing user interfaces. 4th Distributed User Interfaces Workshop on 14th International Conference on Web Engineering ICWE 2014
1. Distributing User
Interfaces
4th Workshop on Distributed User Interfaces and Multimodal Interaction
14th International Conference on Web Engineering ICWE 2014
Prof. Dr. Ricardo Tesoriero
ISE Research Group
University of Castilla-La Mancha
2. Index
Introduction
Motivation
Characterizing the Distribution of User Interfaces.
User Interface Distribution Capabilities
User Interface Distribution States
Proxywork
Proxywork Distribution Primitives
Conclusions
Future Work. Open Issues
7. User Interface EcoSystems
See on deviceNotification
User interface
L. Terrenghi, A. Quigley y A. Dix, «A taxonomy for and analysis of multi-person-display
ecosystems,» Personal Ubiquitous Comput., vol. 13, nº 8, pp. 583-598, 2009.
9. Case 1: Distributing the article section
of the Web Page
Imagine this scenario…
Suppose that you are browsing information on the Internet using your
Smartphone
You go to your favorite Web Site and you start reading an interesting article
while you are on the bus, train or metro on your way home
10. Case 1: Distributing the article section
of the Web Page
When you arrive home you find your flatmates watching the Smart TV
As you tell them about the article you have read, they suddenly got
interested in it.
11. Case 1: Distributing the article section
of the Web Page
To quickly share this information with them, you “transfer” the article from
the Smartphone to the SmartTV
You DISTRIBUTE the article section of
the Web page from the Smartphone to
the Smart TV
12. Case 2: Distributing a menu section of a
Web Page
You are projecting a Web page on the Wall to show event participants
information
You use the laptop keyboard and pad to get the information
13. Case 2: Distributing a menu section of a
Web Page
As you want to control pages to be displayed by the projector remotely, you
“transfer” the menu section to the Smartphone
14. Case 3: Continuity
You are filling a Web form using your Laptop, but you have to catch the train!
“Transfer” (Distribute) the Web form of the Web Page to the Smartphone
(including the information you have already entered when using the Laptop)
15. Characterization of the User Interface
Distribution
How do we describe these characteristics on an Ecosystem of User Interfaces?
There are many models to describe the User Interface. However, most of them (i.e.
AUI Model of the CRF or IFML) employ a Tree based structure where Interaction
Objects have a single parent, if any (root).
How can we characterize the user interface distribution if a component can
be hosted by more than one container?
G. Calvary, J. Coutaz, L. Bouillon, M. Florins, Q. Limbourg, L. Marucci, F. Paternò, C.
Santoro, N. Souchon, D. Thevenin y J. Vanderdonckt, «The CAMELEON reference
framework,» Deliverable 1.1, CAMELEON Project, 03 Septiembre 2002. [On Line].
Available: http://www.w3.org/2005/Incubator/model-based-
ui/wiki/Cameleon_reference_framework. [last access: 12/06/13].
OMG. IFML: The Interaction Flow Modeling Language. URL= http://www.ifml.org/
16. How can we characterize the distribution
capabilities of user interfaces?
Metamodel defined in ECORE
enriched with OCL
Pedro G. Villanueva, Ricardo Tesoriero, José A. Gallud. Revisiting
the Concept of Distributed User Interfaces. Distributed User
Interfaces: Usability and Collaboration. Springer.
Human–Computer Interaction Series 2013, ISBN 978-1-4471-5499-
0, pp 1-15.
17. Graphical UI Distribution Model Editor
• Technology
• Eclipse Plugin
• EMF
• GMF
• Characteristics
• Graphical DSL
• Model Validation
• UI Distribution
Capabilities
Video
18. Platform
The combination of Hardware and OS that supports the user interface
Tables / Smart phone running Android or iOS
Laptops running Windows or Linux
Etc.
What about the Web?
The Web Browser is considered as part of the platform
Two Web Browsers running in the same machine are two different platforms
Two Web Browser Tabs (even in separate Windows) are the same platform
What about multiple monitors
Monitors are considered devices that are connected to the same platform
N monitors connected to the same computer belong to a single platform
19. Interaction object
The Interaction Object plays the same role as the Abstract Interaction Object
defined by the Cameleon framework.
On concrete implementations they may play the role of Windows, Panels, Text
Fields, Buttons, etc.
J. M. Vanderdonckt y F. Bodart, «Encapsulating knowledge for intelligent automatic
interaction objects selection,» de INTERACT '93 and CHI '93 Conference on Human
Factors in Computing Systems, Amsterdam, 1993.
G. Calvary, J. Coutaz, L. Bouillon, M. Florins, Q. Limbourg, L. Marucci, F. Paternò, C.
Santoro, N. Souchon, D. Thevenin y J. Vanderdonckt, «The CAMELEON reference
framework,» Deliverable 1.1, CAMELEON Project, 03 Septiembre 2002. [On Line].
Available: http://www.w3.org/2005/Incubator/model-based-
ui/wiki/Cameleon_reference_framework. [last access: 12/06/13].
20. Hosting
Defines a relationship between two Interaction Objects (Host and Guest)
It represents that the Guest Interaction Object can be hosted in the Host
Interaction Object during the execution of the User Interface
A Guest Interaction Object can be hosted in more than one Host Interaction
Object
All Guest Interaction Objects must be hosted in at least one Host Interaction
Object during the User Interface lifetime.
The host may change during the User Interface lifetime
21. Interaction Container
Interaction Component
The Hosting relationship defines Interaction Objects as
Interaction Containers
Panels, Layouts, Menus, Submenus, Tables, etc.
Interaction Components
Buttons, Labels, Entry Fields, Menu Items, RFID Tags
If an Interaction Object does not host any Interaction Object then it is an
Interaction Component
If an Interaction Object hosts another Interaction Object then it “mutates” into an
Interaction Container
Both Interaction Components and Interaction Container must be contained in
another Interaction Object
22. Implementation
Interaction Surface
Defines a relationship between an Interaction Container and a Platform
It represents that the Interaction Container is supported by a Platform
An Interaction Container is implemented by at most one Platform
An Interaction Container which is implemented by Platform “mutates” into an
Interaction Surface
An Interaction Surface might not be hosted on any other Interaction Object.
Windows, Activities, RFID Panels, Pages, Views
23. Interaction dependency
Defines a relationship between two Interaction Surfaces (Master and Slave)
It represents that the lifetime of the Slave Interaction Surface depends on the
lifetime of the Master Interaction Surface
Floating Toolbars depend on the Main Window
27. User Interface Distribution Capabilities
Divisible
A UI System is Divisible iff exists an Interaction Object that can be hosted on more
than one Interaction Surface
Distributable
A UI System is Distributable iff exists at least one Interaction Object that can be
hosted on at least two Interaction Surfaces implemented on different Platforms
28. User Interface Distribution States
We define the User Interface Distribution State as the organization of all the
Interaction Objects that are part of a UI System at a given instant in time.
Unified State: A UISystem reaches the Unified State iff all Interaction Objects are
hosted on the same Interaction Surface at a given time
Divided State: A UISystem reaches the Divided State iff it has at least two
Interaction Surfaces which host at least one Interaction Object each at a given
time.
Distributed State: A UI System reaches the Distributed State iff it defines at least
two Interaction Surfaces that are hosted on different Platforms. As Interaction
Surfaces are Interaction Contaners, they host at least one Interaction Component
each at a given time.
Single Platform State: A UI System reaches the Single Platform State iff it all
Interaction Objects that are part of the UI System are hosted on the a set of
Interaction Surfaces that share the same Platform.
30. Quiz
Capabilities
Divisible
Distributable
Distribution States
Unified
Divided
Single Platform
Distributed
M. Manca y F. Paternò, «Distributed User Interfaces with MARIA,» de Distributed User Interfaces 2011
(DUI 2011), CHI 2011 Workshop, Vancouver, BC, Canada, 2011.
33. WallShare
Capabilities
Divisible
Distributable
Distribution States
Unified
Divided
Single Platform
Distributed
!
Pedro González Villanueva, Ricardo Tesoriero, José A. Gallud: Multi-pointer and collaborative system
for mobile devices. Mobile HCI 2010: 435-438.
Pedro González Villanueva, José A. Gallud, Ricardo Tesoriero: WallShare: a multi-pointer system for
portable devices. AVI 2010: 416
34. RFID Panels
Capabilities
Divisible
Distributable
Distribution States
Unified
Divided
Single Platform
Distributed
Ricardo Tesoriero, Pedro G. Villanueva, Habib Moussa Fardoun, Gabriel Sebastian. “Distributed User Interfaces in Public
Spaces using RFID-based Panels”. International Journal of Human-Computer Studies. ISSN: 1071-5819. Volume 72. Issue 1.
Pages: 111–125. January 2014.
Pedro G. Villanueva, Ricardo Tesoriero, José A. Gallud, Abdulrahman H. Altalhi. “A Framework to Develop Web Applications Based
on RFID Panels”. International Journal of Universal Computer Science. ISSN: 0948-695x. Volume 19. Issue 12. Pages: 1792-1807. June
2013.
!
37. IF Distributed UI is an State
AND
IF Distributable is a Capability
THEN
DUI ≠ DeUI
Therefore, when we talk about Distributed User Interfaces, we are really talking
about Distributable User Interfaces!
38. Proxywork : Distributable User
Interfaces for the Web
Proxywork allows users to distribute the user interface components of Web
applications among a set of displays.
The distribution is controlled by the user through a set of primitives (i.e. show,
hide, copy, move, etc.) attached to Web page components.
These primitives are automatically attached to Web page components on runtime
by the Proxywork Web proxy.
Therefore, Web pages do not require any extra information to be distributed
among different displays.
Pedro González Villanueva, Ricardo Tesoriero and José. A. Gallud.
Distributing web components in a display ecosystem using
Proxywork. BCS-HCI '13. Proceedings of the 27th International BCS
Human Computer Interaction Conference, Brunel University,
London, UK, 9-13 September 2013, art. 28.
URL=http://dl.acm.org/citation.cfm?id=2578048
Pedro González Villanueva. Distributable User Interfaces. PhD
Thesis. University of Castilla-La Mancha. 2014
40. Proxywork Limitations
The Web page should be “well formed”
Although Proxywork is able to distribute any component; the distribution is
limited to specific Tags (DIVs)
HTTPS
User Interface Adaptation
Cross domain communication
Etc.
41. Conclusions
Defines a metamodel to characterize the distribution of User Interfaces
Graphical Model Editor to build and validate User Interface Distribution
models
Reflexive Model Editor that characterizes User Interfaces
Set the difference between User Interface capabilities and states to redefine
the DUI concept as DeUI
Proxywork transforms Web Pages into Distributable User Interfaces
42. Future Work
Distributable User Interfaces and Task modeling
Adapting Distributable User Interfaces
Controlling Distributable User Interfaces
Etc.
43. Thank you!
Ricardo Tesoriero ricardo.Tesoriero@uclm.es
José A. Gallud jose.gallud@uclm.es
Pedro González Villanueva pedrogovi@uclm.es
Notas do Editor
Good Morning,
José Antonio thank you very much for the introduction
The subject of this talk is about distributing user interfaces where we expose a classification model that allows us to
The first question we ask is
Why distributing user interfaces is important?
Actually, users interact with multiple devices at the same time.
Users interact with user interfaces running on multi-purpose mobile devices.
For intance, smartphones, tablets, laptops, netbooks, notebooks, smartwatches, etc.
Using touch screens, key pads, mouses, keyboards, etc.
Besides, users are also able to interact with user interfaces that run on multi-purpose stationary devices.
For instance, users can interact with Smart TVs, Projectors displays using different devices such as the Kinect, Play station move or the wii-mote
Finally, users interact with user interfaces designed for specific purposes.
For instance,
Airplane cockpits.,
RFID Interactive Panels,
Advertisement Displays,
Etc.
The concept of a user interface environment is based on the idea developed by Lucia Terrenghi of Ecosystems of coupled displays
For instance, a smartwatch and a Smartphone define a coupled display ecosystem where users get notified on the smartwatch when a message is received by the Smartphone.
The smartwatch provides users with information regarding the message (i.e. the author as well the message).
Due to smartwatch display limitations, sometimes is not posible to display the whole information or to replay the message.
By pressing a button on the Smartwatch, users synchronize the view of the information on the Smartwatch to the Smartphone view.
So, they can are able to interact with the full fleshed versión of the user interface.
THIS IS AN EXAMPLE OF A COUPLED DISPLAY
Another interting example of a user interface environment is the capability of smartphones to control other devices (i.e. SmartTVs) acting like a remote control.
THIS IS ANOTHER EXAMPLE OF A COUPLED DISPLAY
To sum up, we have two pairs of independent user interfaces running on their own platforms where one device controls the other one.
Therefore, If we consider both ecosystems as user interfaces, we can conclude that we have two distributed user interfaces.
The first question we ask is
Why distributing user interfaces is important?
Actually, users interact with multiple devices at the same time.
You DISTRIBUTE the article section of the Web page from the Smartphone to the Smart TV
How to represent the distribution characteristics of the user interface
We consider the browser as a virtual machine; therefore the browser is part of the platform
The concept of a user interface environment is based on the idea developed by Lucia Terrenghi of Ecosystems of coupled displays
For instance, a smartwatch and a Smartphone define a coupled display ecosystem where users get notified on the smartwatch when a message is received by the Smartphone.
The smartwatch provides users with information regarding the message (i.e. the author as well the message).
Due to smartwatch display limitations, sometimes is not posible to display the whole information or to replay the message.
By pressing a button on the Smartwatch, users synchronize the view of the information on the Smartwatch to the Smartphone view.
So, they can are able to interact with the full fleshed versión of the user interface.
THIS IS AN EXAMPLE OF A COUPLED DISPLAY
Another interting example of a user interface environment is the capability of smartphones to control other devices (i.e. SmartTVs) acting like a remote control.
THIS IS ANOTHER EXAMPLE OF A COUPLED DISPLAY
To sum up, we have two pairs of independent user interfaces running on their own platforms where one device controls the other one.
Therefore, If we consider both ecosystems as user interfaces, we can conclude that we have two distributed user interfaces.