Presented at the fourth ACM SIGCHI Symposium on Engineering Interactive Computing Systems, IT University of Copenhagen, Denmark, June 25–28, 2012.

1. 2 December 2005
Fusion in Multimodal Interactive Systems:
An HMM-Based Algorithm for User-Induced
Adaptation
Bruno Dumas1, Beat Signer1 and Denis Lalanne2
1 WISE Research Lab, Vrije Universiteit Brussel, Belgium
2 DIVA Research Group, University of Fribourg, Switzerland

2. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2
June 26, 2012
Outline
• Fusion of multimodal input
• Multimodal interaction: from design to implementation
• Algorithms for fusion of multimodal input
• A Hidden Markov Model-based fusion algorithm
• Instantiation of the algorithm
• Evaluation of the algorithm
• Qualitative test
• Quantitative evaluation
• Performance assessment
• Conclusion

3. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 3
June 26, 2012
Fusion of Multimodal Input
 Challenges in multimodal interaction engineering:
continous and probabilistic inputs, real-time…

4. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 4
June 26, 2012
Challenging/Ambiguous Fusion Cases
• “Put-that-there”
 Complementarity case
• “Play next track”
 Different meanings based on order and context

5. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 5
June 26, 2012
CARE Properties at the Design Level
 Describe how modalities can be combined
 Complementarity
 All modalities are necessary
 Assignment
 Absence of choice
 Redundancy
 Same expressive power between modalities
 Equivalence
 Any modality is sufficient

6. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 6
June 26, 2012
HephaisTK Multimodal Framework

7. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 7
June 26, 2012
Description of Multimodal Dialogues
• Syntactic-level description
• For more details: check
SMUIML language

8. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 8
June 26, 2012
Fusion Algorithms
• Frame-based algorithms & Unification-
based algorithms
• Symbolic approaches
• Both are a de facto standard
• Symbolic-statistical algorithms

9. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 9
June 26, 2012
Symbolic-Statistical Approaches
 Symbolic data
 Interpretation of low-level data
 Statistical processing
 Machine learning algorithms

10. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 10
June 26, 2012
Our HMM-Based Fusion Algorithm
 Why hidden Markov models?
 Very good modelling of time-related events
 Takes advantage of input data coming from
probabilistic input modalities
 On-the-fly adaptation of the model
- e.g. based on user feedback

11. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 11
June 26, 2012
Put-That-There Example

12. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 12
June 26, 2012
Put-That-There Example

13. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 13
June 26, 2012
Put-That-There Example

14. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 14
June 26, 2012
Put-That-There Example

15. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 15
June 26, 2012
Put-That-There Example

16. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 16
June 26, 2012
Put-That-There Example

17. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 17
June 26, 2012
Instantiation of the Algorithm

18. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 18
June 26, 2012
Evaluation of the Algorithm
 Evaluation on three levels
 Qualitative “gut check” test
 Benchmarks on real-world examples
 Performance assessment

19. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 19
June 26, 2012
Evaluation: Qualitative Test
• Three expert users were asked to assess the
behaviour of the algorithm
 frames-based and HMM-based fusion
 5 minutes per condition + interview
• Coherent behaviour between the algorithms
• No noticeable difference
in responsiveness

20. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 20
June 26, 2012
Evaluation: Benchmark

21. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 21
June 26, 2012
Evaluation: Benchmark

22. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 22
June 26, 2012
Benchmark Results (1)
• CARE properties: equivalence and redundancy tests

23. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 23
June 26, 2012
Benchmark Results (2)
• Sequential and non-sequential complementarity tests

24. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 24
June 26, 2012
Benchmark Results (3)
• “Play next track” example
Meaning
frames
HMM-based

25. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 25
June 26, 2012
Evaluation: Performance
• Per fusion algorithm:
• 5 runs of 40 input pieces
• 5 x 20 expected fusion results
• Frame-based: 18.2 ms
• Standard deviation: 12.7 ms
• HMM-based: 16.6 ms
• Standard deviation: 11.6 ms

26. Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 26
June 26, 2012
Conclusion
• A new symbolic-statistical multimodal fusion
algorithm based on Hidden Markov Models
• Integrated into the HephaisTK framework
• Superior recognition rates compared to existing
algorithms
• Handling of ambiguous cases
• Processing of real-time user feedback
http://wise.vub.ac.be/bruno-dumas