Exploring complex spatiotemporal data can be very challenging for non-experts. Recently, gestural interaction has emerged as a promising option, which has been successfully applied to various domains, including simple map control. In this paper, we investigate whether gestures can be used to enable non-experts to explore and understand complex spatiotemporal phenomena. In this case study we made use of large amounts of Linked Open Data about the deforestation of the Brazilian Amazon Rainforest and related ecological, economical and social factors. The results of our study indicate that people of all ages can easily learn gestures and successfully use them to explore the visualized and aggregated spatiotemporal data about the Brazilian Amazon Rainforest.

1. Team: Thomas Bartoschek, Gerald Pape,
Jim Jones, Christian Kray, Tomi Kauppinen
Gestural Interaction with
Spatiotemporal Linked Open Data
FOSS4G. Nottingham, England.
jim.jones@uni-muenster.de

2. Name: Jim Jones
The Author
Thomas Bartoschek
Gestural Interaction with Spatiotemporal Linked Open Data 2
“Dear Jim,
You lucky son of * *****! You’re going to substitute
me in Nottingham at FOSS4G.
Best,
Thomas ”

3. Name: Jim Jones
Agenda
• Motivation
• Quick Introduction to Linked Open Data
• Application
• Dataset Creation
• Gesture-based Approach
• Approach Evaluation
• Conclusion
• Future Work
Gestural Interaction with Spatiotemporal Linked Open Data 3

4. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 4

5. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 5

6. Name: Jim Jones
Deforestation decreasing...
2004: 27.423 km²
2012: 4.571 km²
(Awesome!)
Gestural Interaction with SpatiotemporalLinked Open Data 6
Known Issues
 Data produced queued for analysis (Huge
amounts)
 Hard to visualize
 Hard to correlate the produced data with
external variables.

7. Name: Jim Jones
Motivation
Gestural Interaction with Spatiotemporal Linked Open Data 7
•How to efficiently correlate the deforestation phenomena
with external variables?
•How to effectively communicate it to the population?

8. Name: Jim Jones
Triangle of Sustainability
● Wissenschaft Interaktiv 2012
(Interactive Science)
● Awarded with €10k for
developing the project
● 4 weeks for development!
Gestural Interaction with Spatiotemporal Linked Open Data 8
Linked Open Data!

9. Name: Jim Jones
What is Linked Data?
(In 2 minutes)
Gestural Interaction with Spatiotemporal Linked Open Data 9

10. Name: Jim Jones
The Web
Gestural Interaction with Spatiotemporal Linked Open Data 10
• Awesome Discoverability!
• But just for humans :(

11. Name: Jim Jones
Semantics
Gestural Interaction with Spatiotemporal Linked Open Data 11
?
The forbidden fruit
Xxx xxxx x xxxx xxx xxxx
Xxx xxxxxxx xx x x xxxxxxxx.
Cheers,
-God
● What is the article about?
● Who is the author?
● What is inside the picture?
● Where, when and by whom was the picture taken?
● ...

12. Name: Jim Jones
Linked Open Data
Gestural Interaction with Spatiotemporal Linked Open Data 12
“linked open data describes a
method of publishing structured
data so that it can be interlinked
and become more useful.”
Raw Data
now!

13. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 13
ID 1
(Uniquely Identified)
ID 5
ID 3
ID 4
ID 2
ID 6

14. Name: Jim Jones
Some benefits of Linked Open Data
Gestural Interaction with Spatiotemporal Linked Open Data 14
•Attaches meaning to data
•Allows efficient thematic searches (e.g. person, company, city, book, etc.)
•Links different kinds of datasets making the WORLD a single big repository!
•Private data and public data can be mixed, enabling companies and individuals to
make better decisions and generate innovations.
My Repository

15. Name: Jim Jones
Creating Dataset...
Gestural Interaction with Spatiotemporal Linked Open Data 15
Deforestation
•Soybeans Crops
•Cattle
•GDP
•Population
•Brazilian Municipalities' Area
•Download data from IBGE Servers
•Create Triples

16. Name: Jim Jones
Dataset
Vocabularies
• Time and Space Core Vocabulary (TISC)
• Open Linked Amazon Vocabulary (OLA)
Gestural Interaction with Spatiotemporal Linked Open Data 16
Spatiotemporal Dataset
● Aggregated in grid cells of 25km x 25km
● Time-series 2004 - 2009

17. Name: Jim Jones
Motivation (Recap...)
Gestural Interaction with Spatiotemporal Linked Open Data 17
•How to efficiently correlate the deforestation phenomena
with external variables?
•How to effectively communicate it to the population?
Gestural Interaction with Virtual Globes!

18. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 18

19. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 19

20. Name: Jim Jones
Gestures
Gestural Interaction with Spatiotemporal Linked Open Data 20
pan zoom in / out time travel

21. Name: Jim Jones
Video ( +1 minute)
http://www.youtube.com/watch?v=Jjzj84R1Tgc
Gestural Interaction with Spatiotemporal Linked Open Data 21

22. Name: Jim Jones
Evaluating the Approach
Gestural Interaction with Spatiotemporal Linked Open Data 22

23. Name: Jim Jones
Evaluation Procedure and Material
• 5- 8 Minutes usage
• Questionnaire with 28 questions in two groups:
• 1 group: demographic information
• e.g. gender, age, primary hand, familiarity with gesture control interfaces
• 2 group: Task Load (NASA TLX – Task Load Index). Rating how well gestures
and map actions fit together, from 0 to 20.
• e.g. In which scale you define the mental or physical workload for the gestures?
• No reward!
Gestural Interaction with Spatiotemporal Linked Open Data 23

24. Name: Jim Jones
Test Group
• Participants: 43
• 28 male
• 15 female
• Youngest : 10
• Oldest: 59
• Left handed: 7
• Right handed: 36
• Familiarity with gesture-based interfaces:
• Familiar: 19
• Unfamiliar: 24
Gestural Interaction with Spatiotemporal Linked Open Data 24

25. Name: Jim Jones
Results
Gestural Interaction with Spatiotemporal Linked Open Data 25

26. Name: Jim Jones
Observations
• Most participants reported the gestures as mentally challenging in the first minutes
• Older people found the system very accessible, particularly when compared with their
first time using a mouse.
• User recognition gets considerably affected when bystanders are ‘seen’ by the Kinect
sensor.
Gestural Interaction with Spatiotemporal Linked Open Data 26

27. Name: Jim Jones
Software
Gestural Interaction with Spatiotemporal Linked Open Data 27
Virtual Globe (UI)
LOD Processing
Gestures (Kinect)
https://github.com/giatschool/triangle-of-sustainability

28. Name: Jim Jones
Hardware
Gestural Interaction with Spatiotemporal Linked Open Data 28

29. Name: Jim Jones
Triangle on Tour!
• SBPC Science Fair
• Recife (August, 2013)
• KIT-Alumini Club Seminar
• Recife (September,2013)
• GeoCamp
• Campos do Jordao (November, 2013)
• Ecogerma
• Belém (November, 2013)
Gestural Interaction with Spatiotemporal Linked Open Data 29

30. Name: Jim Jones
Conclusion
• The triangle provides an attractive and easy to use solution for displaying
spatiotemporal LOD.
• It has a high acceptance in class-rooms (Already presented in many German
schools)
• Participants were able to learn how to use 3 gestures without excessive workload.
• Linked Data offers a good bases for efficiently connecting deforestation data with
external variables.
Gestural Interaction with Spatiotemporal Linked Open Data 30

31. Name: Jim Jones
Future Work
• Improve registration gestures
• Inefficient when more than one person is in front of the sensor
• Try different Kinect frameworks.
• Enable registration by children (short people)
• Minimum arm length problem.
Gestural Interaction with Spatiotemporal Linked Open Data 31

32. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 32

33. Name: Jim Jones
Gestural Interaction with Spatiotemporal Linked Open Data 33
Muito obrigado.
Danke Schön.
Kiitos.
Dziękuję bardzo.
Thank you!