An Android Communication Platform between Hearing Impaired and General People

An Android Communication Platform Between Hearing
Impaired and General People.
Presented By:
Afif Bin Kamrul
Roll No: 1404065
Supervised By
Shayla Sharmin
Assistant Professor
Department of CSE
CUET

2 Department of CSE, CUET
Contents
 Introduction
 Motivation
 Previous Work
 Objectives
 Methodology
 Implementation
 Experimental Result
 Future work and conclusion
 References

Introduction
Communication:
 Verbal action
 Facial Expression
 Body gesture and posture
(Sign Language)
Deaf and Mute People
Sign language

Introduction (Cont.)
Disable to Hear: 466 million
people
Suffering from variable degree of hearing
loss =13 million [1]
Suffering from severe hearing loss =3 million
[1]

Motivation
Difficult to Understand
Sign Language
Few works in Bangla
based on mobile platform

Motivation (Cont.)
Need Friendly Medium to
understand Sign and Natural
Language

Previous Work
 [Sarkar et al., 2009]
 Worked on converting Bangla text to Bangla sign language.
 Due to matched input text, video clips are played.
 Desktop or laptop based application.
 [Shahriar et al., 2017]
 Used CMU Sphinx speech recognition to convert Bangla
speech to Bangla sign language.
 Data to train the acoustic model for Bangla are limited.
 Displays only still sign images.

Previous Work (Cont.)
 [Gayyar et al., 2016]
 Worked on Arabic sign language keyboard.
 Developed application which conducts local database based on
signs’ Unicode.
 [Tanvir, 2018]
 Worked on a procedure of converting Bangla Text to Bangla
Sign Language.
 No medium for deaf people to communicate.

Objectives
An Android App to Bridge the Gap between General
and Hearing Impaired People.
Recognizing Bangla Speech and Converting to Sign
Language via Virtual Agent for more than 200 Bangla
words for Deaf People.
Sign Language Keyboard that Converts the Sign into
Bangla Text.
আজ

Methodology
Start
Speech to Sign Language
Conversion with the Help of
Virtual Agent
Launch Speech to Text by Google
API
Launch Sign Language Keyboard
End
Convert Bangla Text

Methodology (Cont.)
Draw character in Adobe Photoshop
Adding puppet controllers in Adobe
character animator rig mode
Design and recording animation in
Adobe Character Animator scene
mode
Render using Adobe Media Encoder

Methodology (Cont.)
Speaking
Bangla by
Google API
Fetching Sign
Sign Language
Output via
MediaPlayer
Raw folder
containing
animation
Splitting Suffixes
using recursion
Splitting Words by
detecting space in
strings

Methodology (Cont.)
Start
Button with
sign
preview
using
InputMetho
dService
Select
Sign
Match with
Corresponding
Bangla
Character using
conditional
logic
Commit
Character
SentenceEnd

Implementation
Option for general
people and deaf/mute
people
Speech recognition
begins by pressing the
mic button

Implementation (Cont.)
Speech
recognition
Showing
signs

Keyboard setup
screen
Enabling the
keyboard
Selecting the
keyboard

Typing on
keyboard
Typed
text

Experimental Result
Teacher explaining sign
to studentStudent explaining sign
after using application
Bak-Sraban Pratibandhi Biddalay, PHT
Center, Muradpur, Chittagong

Experimental Result (Contd.)
Deaf/mute
people
Teachers and general people (5 word trials between each
student and teacher/general people)
1 2 3 4 5
1 5 5 3 4 5
2 5 3 4 5 5
3 4 5 5 5 4
4 5 4 5 4 5
5 4 5 4 4 5
6 5 4 5 5 5
Total 28 26 26 27 29
Total success = 136
Accuracy of converting signs=
𝑡𝑜𝑡𝑎𝑙 𝑠𝑢𝑐𝑐𝑒𝑠𝑠
𝑡𝑜𝑡𝑎𝑙 𝑡𝑟𝑖𝑎𝑙
= (136/150)% = 90.6%
Table-01: No of times application could convert signs from voice

Deaf/mute
people
Teachers and general people (5 word trials between each
student and teacher/general people) Total
1 2 3 4 5
1 5 4 3 3 3 18
2 5 2 3 4 2 16
3 4 3 3 3 3 16
4 5 3 3 4 4 19
5 4 3 3 4 3 17
6 4 2 4 3 3 16
Table-02: No of times students understood the converted signs
Total success = 102
Accuracy of understood signs=
𝑡𝑜𝑡𝑎𝑙 𝑠𝑢𝑐𝑐𝑒𝑠𝑠 𝑜𝑓 𝑡ℎ𝑖𝑠 𝑡𝑎𝑏𝑙𝑒
𝑡𝑜𝑡𝑎𝑙 𝑠𝑢𝑐𝑐𝑒𝑠𝑠 𝑜𝑓 𝑝𝑟𝑒𝑣𝑖𝑜𝑢𝑠 𝑡𝑎𝑏𝑙𝑒
= (102/136)% = 75%

Participants
Perfectly Typed Word (each
having 5 trials)
1 5
2 3
3 5
4 4
5 4
6 3
Table-03: No of times students typed the words perfectly using
keyboard
Total= 24
Accuracy of converting signs=
𝑡𝑜𝑡𝑎𝑙 𝑠𝑢𝑐𝑐𝑒𝑠𝑠
𝑡𝑜𝑡𝑎𝑙 𝑡𝑟𝑖𝑎𝑙
= (24/30)% = 80%

Questionnaires for deaf/mute students (1=lowest,5=highest):
 Are the animated signs understandable?
 Is the application boring?
 Is the keyboard flexible to type?
Question
Participants (deaf and mute user)
1 2 3 4 5 6
Q1 4 5 5 4 3 5
Q2 2 1 2 2 1 2
Q3 5 4 4 4 5 4
4.333333333
1.666666667
4.333333333
0
1
2
3
4
5
6
Q1 Q2 Q3
Mean

Questionnaires for teachers and general people
(1=lowest,5=highest):
 Are you facing any problem while using the application?
 Do you find this application easy to use?
 How interactive the application is?
Question
Participants (deaf and mute user)
1 2 3 4 5
Q1 3 4 2 2 3
Q2 4 4 3 5 5
Q3 4 5 1 4 3
2.8
4.2
3.4
0
1
2
3
4
5
6
Q4 Q5 Q6
Mean

Future Work
Log-in system can be introduced for the users.
Word list can be increased more to support a broad range
of Bangla words.
Messaging option can be introduced.
Facebook auto-share option can be introduced.
Keyboard layer might be modified.

Conclusion
The platform can be helpful as smartphone is portable
by all of the people.
Enables the deaf/mute students to participate with us in e
very aspect of life.
The application can remove communication barrier in be
tween general and deaf/mute communities.

References
[1] Deafness in Bangladesh Mohammad Alauddin FRCS; FCPS; DLO and Abul Hasnat Joarder, MBBS, FCPS D
ept. ofOto!aryngo!ogy, BSM Medical University, Shahbag, Dhaka, Bangladesh
[2] Ahmed Tanvir, “A Small Initiative to Convert Bangla Text to Bangla Sign Language”, American-Internationa
l University, Bangladesh. Available: https://www.academia.edu/30768422/A_Small_Initiative_to_Convert_Bangl
a_Text_to_Bangla_Sign_Language
[3] B. Sarkar, K. Datta, C. Datta, D. Sarkar, S. J. Dutta, I. D. Roy, A. Paul, J. U. Molla, and A. Paul, “A translator
for bangla text to sign language,” in 2009 Annual IEEE India Conference. IEEE, 2009, pp. 1–4.
[4] Mohammed Safayet Arefin, Lamia Alam, Shayla Sharmin, Mohammed Moshiul Hoque, “An Empirical Fram
ework for Parsing Bangia Assertive, Interrogative and Imperative Sentences”, 1st International Conference on
Computer & Information Engineering, 26-27 November, 2015 Organizer: Dept. of CSE, Rajshahi University
of Engineering & Technology, Rajshahi, Bangladesh.
[5] A. Kowshal, S. Sharmin and M. M. Hoque (2017). Development of an Interactive Game to Increase Speech A
bility for Language Impairment Children. 2017 International Conference on Engineering Research, Innovation a
nd Education (ICERIE).
[6] Rhythm Shahriar, A.G.M. Zaman, Tanvir Ahmed, Saqib Mahtab Khan, H.M. Maruf, “A Communication Platf
orm Between Bangla and Sign Language”, 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HT
C) 21 - 23 Dec 2017, Dhaka, Bangladesh.

References
[7] Kaidul Islam and Bishnu Sarker, “Designing A Press and Swipe Type Single Layered Bangla Soft Keyboard f
or Android Devices”, 16th Int'l Conf. Computer and Information Technology, 8-10 March 2014, Khulna, Bangla
desh.
[8] Mahmoud EI-Gayyar, Amira Ibrahim and Ahmed Sallam, “The ArSL Keyboard for Android”, 2015 IEEE Sev
enth International Conference on Intelligent Computing and Information Systems (ICICIS'15).
[9] Lance A. Allison, Mohammad Muztaba Fuad, “Inter-App Communication between Android Apps Developed
in App-Inventor and Android Studio”, 2016 IEEE/ACM International Conference on Mobile Software Engineerin
g and Systems.
[10] Md. Shahnur Azad Chowdhury, Nahid Mohammad Minhaz Uddin, Mohammad Imran, Mohammad Mahadi
Hassan and Md. Emdadul Haque, “Parts of Speech Tagging of Bangla Sentence”, Available: https://www.researc
hgate.net/publication/229038426_Parts_of_Speech_Tagging_of_Bangla_Sentence
[11] Sabir Ismail, M. Shahidur Rahman, Md. Abdullah Al Mumin, “Developing an Automated Bangla Parts of
Speech Tagged Dictionary”, 16th Int'l Conf. Computer and Information Technology, 8-10 March 2014, Khulna,
Bangladesh.
[12] Mohamed JEMNI, Oussama ELGHOUL, “Using ICT to teach sign language”, Eighth IEEE International Co
nference on Advanced Learning Technologies.

An Android Communication Platform between Hearing Impaired and General People

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to An Android Communication Platform between Hearing Impaired and General People

Similar to An Android Communication Platform between Hearing Impaired and General People (20)

Recently uploaded

Recently uploaded (20)

An Android Communication Platform between Hearing Impaired and General People