csun2010

AAC for Elderly People with Hearing,
Speech or Memory Loss due to
Aging
Lawrence Chung, Shinyoung Lim,
Yoosun Chung, Rutvij Y. Mehta, Amit B. Chembra
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
1© 2010

Acknowledgements
• ‘SymbolStix Inc.’
• Students that participated in the prototype implementation
Abdullah Azzouni, Bharat Paturi, Manoj Kumar, Michael Fashola, Nisha Mathew,
Rakesh Sanapala, Rohit Sharma, Sandeep Sistla , Sritej Vepa, Vamshi Bandaru,
Vidya Vijayan , Vinay Kumar
• For their valuable comments & feedback (Research group)
Ebenezer Oladimeji, Sam Supakkul and Tom Hill (a special thanks for the Nexus!)
2© 2010

Outline
Introduction
Goal
Motivation
User Needs of AAC Service
Design of a prototype system
Our models – Olive* , H.O.P.E
Features and Demo
Enhancements
Project in a Nutshell
Conclusion
3© 2010

Introduction
• ‘Science Blog’ – Reduced Hearing more common
and hearing aids inadequate for elderly
• AAC focused on Autism, rarely on elderly people
• Treatment focused on medical/surgical procedure
not rehabilitative approaches
• Communication barriers of elderly not researched
and existing measures not tailored to their needs
• Current technology – No concept exploration
• Population of the elderly to be significant soon
• Need for an assistive system ever more than before
4© 2010

Goal
• Identify – Communication difficulty is a problem
• Understand user needs for an assistive system
• Eradicate the need for bulky, costly instruments
• Develop a communication system to overcome
Hearing loss, Memory loss, Speech Disorder,
Weak Vision
• Use existing hardware (e.g. Cell phones) - cut cost
• System that can be ‘personalized’ for user needs
• A ‘mobile’, ‘assistive’ and ‘cost-effective’ system
that helps reduce the user’s communication
barriers
5
A
S
-
I
S
T
O
-
B
E
© 2010

Communication difficulty - A problem ?
• Elderly are concerned over perception of others
• Lack of communication decreases their self reliance
• Due to communication barriers, they limit their interactions
• Lose confidence and remain confined
• Fear getting embarrassed, so do not communicate
• Give up their interests and live a neglected life
• Psychological problems may arise
• Problems and Behavior can be identified in day to day events
or scenarios
• Such problems are preventable – just have to fill the gaps
6© 2010

Motivation (The Why question)
• Scenario (Family Dinner Outing)
- Elderly parents
- Loss of hearing, Speech impairment and Memory loss
- Not ready to publicly accept their age-related disorders
- Do not welcome use of external medical aids
- Human perception, memory and communication are
visual not textual
- We function best when all senses complement each other
- Similar scenarios repeat in other forms every day
7© 2010

SonOld parents
Son: Mom and Dad, Get
Ready. We are going to
the Chinese restaurant
we went last time for a
family get together.
Parents: Sure
Entire family gets in the
car and leave for the
restaurant
8
A visual scenario (Family Dinner Outing)
© 2010

Mom: Son, where are
we going ?
Son: I told you – the
Chinese restaurant
In the car – On the way
9© 2010

Mom: Son, where are
we going ?
Son: (In an irritated
tone) I have already told
you – the Chinese
restaurant !!
In the car – On the way ( A few miles later …)
Son: (Realizing) Mom –
Are you wearing your
hearing aid ?
Mom: (Does not have it
on) Hmm.. Yes..
Ah..Here I have it….
LOSS OF HEARING !
10© 2010

Mom: (To Dad) – If
Steve comes and
bows, then
acknowledge him. Last
time he met, you did
not talk nor gave a
smile to him
Dad: (STEVE ??? Has a
clueless expression) OK
In the car – ( Almost near the restaurant …)
MEMORY
LOSS !!
11© 2010

Mom: (To Dad) – Why
did you not greet
Steve ? I told you to
greet him !!!
Dad: (Embarrassed) I
could not recognize
him !
Outside the restaurant …
MEMORY
LOSS !!
Steve walks and
greets Dad.
Dad fails to recognize
him !!!
Steve – at
present
Mom: You remember
Steve – our daughter’s
son ! Remember how
cute he was when he
was a small boy !
Dad: (Apparently still
clueless – trying to
recollect) Ah..OK
Steve as a boy
12© 2010

Grandson: Dad – What
did Grandma say? I
cannot understand
what she murmurs….
Son: Mom – Are you
wearing your
dentures?
Inside the restaurant …
Unclear/Incomprehensible
Speech !!
Mom instructs her
grandson to eat
carefully without
spilling
Grandson
13© 2010

• Categories of Difficulties/Communication Barriers for Elderly
- Unclear Utterance , often leading the elderly to repeat and
hard for the listeners to comprehend
- Incomplete/Incomprehensible speech: Speech often times
nonsensical, difficult to express thoughts
- Listening Inability: Hard of hearing especially mobile
communications (Hearing devices – voice-to-text
translation could be introduced)
14© 2010

- Comprehension Inability: Cannot grasp or understand what
others have to say (Cognitive impairment)
- Communication issues due to memory loss: Cannot
remember names of people, places, activities, topic
- Communication in emergency: Should be able to make an
emergency call (say, 911), especially those having medical
problems or living alone (Voice to text/Voice to image relay)
15© 2010

Design of the prototype system
• Target 3 language acquisition mechanisms by icons/photos
1) See what you cannot hear
- Overcome loss of hearing
2) Show what you cannot speak
- Overcome unclear speech (Text-to-speech)
3) See what you cannot remember
- Overcome memory loss
16© 2010

• Key technical concepts of the system:
- Vocabulary : To show visual images, for words heard and to
to be spoken, transformation to address complex scenarios
- Navigation: Identify the scenario/context and allow quick
and easy navigation based on activities of daily living
- Learning: Detecting and building patterns of frequent
vocabulary use in daily living – System adaptability.
17© 2010

• Development Focus:
Ontology: Philosophical study of existence (source: Wikipedia)
Things, relationships and constraints on them
Focus towards ontology-driven context awareness in User
Interface and services
Vocabulary : 5 basic entities
TEXT, ICONS, SOUND
SPEECH & OBJECT RECOGNITION
Target the important sensory activities of SPEECH,
HEARING, VISION and THOUGHT process
All activities available to the user based on the 5 entities
18© 2010

OLIVE – The first attempt
O.L.I.V.E – Older Living Independently
• Started as a independent study project
• AAC research for the elderly – first step
• Implementation on Samsung Instinct
Phone
• Work on ‘Search’ and ‘Teach’ functions
19© 2010

HOPE
H.O.P.E – Helping Older People Easily
• Research ongoing since 2009
• Built on Android Platform
G1, Nexus Phone
• Increased the vocabulary from
Search and Teach to the
5 ontological primitives
20© 2010

HOPE
• Main categories
- Greetings
- Daily Living Activities
- Emergency situations
- Talk (Build sentences)
- Real World Scenarios
- Active Mind (Games)
21© 2010

HOPE
• Menu categories
Speech recognition used in
- Tagging images (assigning a
name to your images for a photo
album – to be extended for all
images in the system)
- Retrieving an image based on
voice/text input
22© 2010

HOPE
• Prominent Points:
- Not feature oriented but scenario oriented
- Use contextual mechanism to minimize
number of clicks
- Speech recognition to make it not only AAC
but also assistive
- Activities revolve around 5 vocabulary entities
- Add on game for Active Mind, can be used when not
actively communicating
26© 2010

HOPE
• Enhancements:
- Add more scenarios to reduce effort of the end user in
creating statements
- Extend the tag/recognize feature to associate a name with
any image of the system instead of the photo album
- Add more user configuration options like zooming of icons
- Add more vocabulary primitives along the 5 organizational
entities
- Cater application to a larger target user population
28© 2010

HOPE
• Enhancements:
- Explore the ‘Contextual’ improvement
- Use the ‘minimum clicks’ approach
- A sample contextual mechanism for Daily Living Activities
inspired from the vocabulary of the Maya civilization
Revolves around the activity with only relevant contextual
icons surrounding it
Final sentence is the combination of the activity and the
context icons e.g. Take Bath + ( Cold or Hot Water)
29© 2010

Project in a Nutshell
• Feasibility Study involved identifying communication
barriers and user needs
• Developed service requirements and design concepts for
our prototypical system
• Still difficult to draw solid tangible requirements from user
needs due to complexity and individuality in communication
barriers
• On going project incremental – evolutionary and agile
methodology
• Many issues – what to represent, how to represent in the
best effective manner.
• Growth depends on more learning, discovery
30© 2010

Conclusion
• Population of elderly to grow – so is the need and urgency
for countering communication difficulties
• Issue has been studied, user needs and concerns identified
• Design of the system based on 5 types of vocabulary
• Tiny AAC software implemented – H.O.P.E
• More investigation on contextual vocabulary and
representing core vocabulary
• Develop a ‘minimum clicks – maximum vocabulary’
approach
• Extend system by adding features for a larger audience
• Make a change – Bridge the gap due to communication
barriers
31© 2010

csun2010

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (15)

Semelhante a csun2010

Semelhante a csun2010 (20)

csun2010