The document discusses multimodal man-machine interaction and human-computer interaction. It defines multimodal interaction as using multiple input and output modalities simultaneously, like vision, speech and touch. This allows for more natural interaction compared to unimodal systems. Some challenges with multimodal interaction are integrating information from different modes and designing universal interfaces. Examples of multimodal systems discussed are Bharati, an Indian initiative for internet access, and ITR, a project at the University of Illinois. The future of HCI is predicted to involve more natural interfaces using combinations of modalities.

1. Multimodal Man-Machine Interaction By: Rajesh P. Barnwal School of Information Technology Indian Institute of Technology, Kharagpur Sociology Language Design Engineering Ethnography Psychology Human Factors Computer Science

22. Multimodal Framework of Bharati Text Analyzer Speech Recognizer Visual Language Manager Speech to text converter Hindi/ Bengali to English Language Translator Keywords Extractor Information Visualization Text to Speech Converter English to Hindi/ Bengali Language Translator Content Receiver Search Engine Internet Strings of text Strings of text Query Strings HTML page Query Strings Text Speech WIMP WIMP : Window, Icon, Menu, Pointing Device (Picture adapted from NID (IITKgp) Website) Next

24. ITR – Multimodal Framework (Picture adapted from ITR Website)

25. ITR Project – Demo Video (Video from ITR Website)

30. Thank You! Questions, if any?

31. Thank You!

32. Typical Information Flow in a Basic Multimodal Interaction Gesture Recognition Gesture Understanding Speech Recognition Natural Language Processing Camera Glove Laser Touch Microphone Context Management Multimodal Integration Dialogue Manager Application Invocation and Coordination Response Planning App1 App2 App3 Graphics VR TTS Feature/ Frame Structures Feature/ Frame Structures

33. Human Computer Interaction (Classical Model) (Picture adapted ACM-SIGCHI)