The world of today and its new technologies like smartphones, tablets, or any flat interaction surface has increasing the need for graphical user interfaces integrating gestural interaction in which 2D pen-based gestures are properly used. Integrating this interaction modality in streamlined software development represents a significant challenge for designers or developers: it requires important knowledge in gestures management, in deciding which gesture recognition algorithm should be used or refined for which types of gestures, or which usability knowledge should be used for supporting the development. These skills usually belong to experts for gesture interaction and not actors usually involved in user interface design process. In this paper, we present a structured method for facilitating the integration of gestures in graphical user interfaces by describing the roles of the gesture specialist and other stakeholders involved in the development life cycle, and the process of cooperation leading to the creation of a gesture-based user interface. The method consists of three pillars: a conceptual model for describing gestures on top of graphical user interfaces and its associated language, a step-wise approach for defining gestures depending on the end user’s task, and a software that supports this approach. This method is exemplified with a running example in the area of document navigation.

1. ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012) 1
François Beuvens and JeanVanderdonckt
francois.beuvens@uclouvain.be
Researcher at LiLab, http://www.lilab.be
Université catholique de Louvain (Belgium)
http://www.slideshare.net/jeanvdd/designing-graphical-user-interfaces-integrating-gestures

2. - Context
- A method for designing graphical user interfaces
integrating gestures
- Underlying model
- Supporting tool
- Ongoing and future works
2
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

3. - Context
- A method for designing graphical user interfaces
integrating gestures
- Underlying model
- Supporting tool
- Ongoing and future works
3
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

4. - Gestures are quick to learn and easy to reproduce
- Stylus amplifies the hand movement and is
comfortable to use
- Offers high precision
- Gives a direct output and visual feedback
- Allows recording meta-data
- Pressure
- Inclination
- Orientation
- ...
4
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

5. - Many existing datasets and algorithms
- Developers have little or no time to investigate all these
different algorithms, datasets and variations
- Some platforms for gesture recognition are
helpful, yet hard for algorithm selection
- Lack of availability
- Lack of extensibility and flexibility
5
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

6. - Context
- A method for designing graphical user
interfaces integrating gestures
- Underlying model
- Supporting tool
- Ongoing and future works
6
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

7. - 4 categories of contributors
- Engineer/architect: requirements analyser
- Designer: in charge of aesthetics
- Gesture specialist: devoted to recognition mechanisms
- Programmer: coder of the UI
- Work together around a tool/system
7
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

8. - Interface Users (IU)
- End users of the interface
- System Users (SU)
- Use the system to produce the user interface
- System Feeders (SF)
- Feed the system with knowledge
8
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

9. 9
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
Interface Users define user
interface requirements.

10. 10
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
Based on the UI
requirements, System Users
define system requirements.

11. 11
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
If system requirements are
met, System Users use the
system to produce the user
interface.

12. 12
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
If system requirements are
not met, based on the system
requirements, System
Feeders feed the system.

13. 13
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
If UI requirements can be
met, System Users can
produce the interface so that
Interface Users can use it.

14. 14
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
If UI requirements cannot be
met, System Users refine
system requirements, then go
back to system feeding.

15. 15
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)
If Interface Users are not
satisfied, Interface Users
refine their requirements, then
go back to system use.
Otherwise, the process ends
up

16. - Context
- A method for designing graphical user interfaces
integrating gestures
- Underlying model
- Supporting tool
- Ongoing and future works
16
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

17. 17
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

18. 18
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

19. 19
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

20. 20
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

21. 21
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

22. 22
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

23. - Context
- A method for designing graphical user interfaces
integrating gestures
- Underlying model
- Supporting tool
- Ongoing and future works
23
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

24. 24
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

25. - Theoretical
- Conceptual modeling of 2D pen-based gestures for GUIs
- Methodological
- Integration into UsiXML, a XML User Interface Description
Language (www.usixml.org)
- Define the role of the System Users, etc.
- Empirical
- User study on user satisfaction with gesture integration
- Development benefits estimation
25
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

26. - Empirical: with respect to
- System Feeders: to what extent is it expensive to
incorporate new knowledge in UsiGesture
- A new algorithm (recognizer)
- A calibration of an algorithm
- A new variant of an existing algorithm
- System Users: to what extent is it straightforward to design
a GUI based on UsiGesture
- Interface users: do they perceive a better recognition when
the right algorithm is used
26
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)

27. 27
Thank you very much for your attention!
Any question?
See more at:
www.usixml.org, www.usixml.eu
www.lilab.eu, www.lilab.be
ACM Int. Conf. on Design of Communication SIGDOC'2012 (Seattle, October 3-5, 2012)