1
HCI Final Report – Team 6
Roberto Medico
Roberto.medico91@gmail.com
Niko Leinonen
Nikoleino91@gmail.com
Krista Vilppola
Krista.Vilppola@student.oulu.fi
Daniel Askeli
Daniel.Askeli@student.oulu.fi
Jukka Pajukangas
Jukka.Pajukangas@student.oulu.fi
Pablo Panero
Ppanero27@gmail.com
ABSTRACT
The goal of this report is to present the design of an
interactive system to aid people who suffer from dementia.
Population in Finland is getting older and there is a need to
help elders keep performing their daily activities and living
independently. The design phase was iterative. In the first
iteration the concept was studied, as well as literature and
the targeted disease, a preliminary interface for the systems
was also made. In the second iteration a new version of the
interface was done by changing, accordingly to the findings
of the evaluation, the interface produced in the first
iteration. The third iteration, as well as the second one,
resulted in an enhanced interfaced. In the first phase of the
evaluation the system was evaluated by targeted users
(although dementia was simulated), for the second
evaluation the same methods were used plus cognitive
walkthrough and heuristic evaluation. Future work would
involve updating the system to handle better memory loss
and a more customizable interface in order to adapt the
colors to patients’ preferences.
Keywords
Dementia, Alzheimer, Human-Computer interaction,
assistive technology, care, interaction.
INTRODUCTION
Assisting people with dementia is a challenging task that
involves both a deep understanding of the disease and its
effects and a well-reasoned evaluation of the tools and
methodologies that could be used to counterbalance or
alleviate those. This is why the main mean through which
patients are helped is still by placing them side by side with
caretakers that can listen to their needs and help them
through their daily activities.
Nonetheless, having some kind of technology to assist the
patients would allow them to be much more independent
and their caretakers to be less concerned about them. As
mentioned above, though, such technology should be
designed and evaluated very accurately, given the particular
users the system should interact with.
Throughout this paper, our system will be described in
detail. The system is supposed to be helpful both for the
patient and for the caretaker. A home-system and a bracelet
are used to monitor and track patient’s movements and
location, what promotes patient’s safety and independence,
embedding sensors to measure patient’s health (i.e. pulse
rate measurement and a falling detection system). The
above-mentioned bracelet is provided with buttons to call
for help and/or to set reminders. The home-system also
consists of 17-inch screens distributed in the house to be
used by the patient to browse useful information and see
reminders or alarms.
The caretaker is also provided with an application, ideally
running on a mobile device such as a 7-inch tablet, to
browse patient’s information such as health conditions,
location history, devices’ state and to set and send
reminders and alarms to the patient’s house or to contact the
patient directly.
This system assumes that one or more caretakers are assign
to one patient. The targeted population of this system is
families (or hired people, such as a nurse) that take care of a
relative with dementia (e.g. daughter and son take care of
his father).
BACKGROUND WORK
Many studies had already been carried out about this
matter, and several attempts to build realistic prototypes
have been made by Human-Computer Interaction (HCI for
short) designers. Beside the classical well-known clinical
symptoms, such as memory loss, it is possible to identify
other effects that could be tackled by the technology:
agnosia, which is the inability to recognize simple objects
or known people, and apraxia, which is the inability to carry
out gestures or movements [2].

2
There is no doubt that there are different stages of the
disease with increasingly effective symptoms, such as
hallucinations, depression, aggressive behavior, etc. From
the literature we learnt that assistive technology should be
designed and used to support the patient in his/her life
instead of taking decision for him/her giving the patient a
feeling of independence. This could be achieved by
supporting patient’s remaining skills and not blaming or
pointing out his/her disabilities [1]. In addition, the
technology should be unobtrusive and calmly integrate in
the existing environment (e.g. the house) in order not to
confuse or agitate the patient. Another key point to take into
account is that most of the patients suffering from dementia
are elders, who are very likely not used to technology and
devices, so it is not possible to assume any kind of
experience when interacting with computer systems.
Assistive technology should improve and support the well-
being of the patient by focusing on providing specific
functionalities: promote reminiscence and memory, support
daily tasks and procedures, promote and suggest physical
and social activities, encourage access to outdoor space and
involvement in creative hobbies and activities [3].
Besides helping the patient to remember, assistive
technology should also assure his/her safety (e.g. by
monitoring and tracking house environment and devices, or
by sensing health issues, such as falling detection or pulse
rate measurement). Concerning the caretaker, technology
should help to monitor patient’s health in a reliable and fast
way, as well as helping to stay in contact with the patient,
allowing direct or indirect communication.
METHODS
The design process of our system went through two main
cycles, involving design, evaluation and prototype testing.
The chosen approach was to build a unique system that
integrates two main modules, to provide assistance and
support both for the patient and the caretaker. The core
functionalities the system should provide to the patient are
to guarantee and look after his health and safety and to
assist him in carrying out daily tasks counterbalancing the
known symptoms of dementia.
In the early phases of the design process, we decided to
make some assumptions about the future users of the
system: patients suffer from Alzheimer’s disease and/or
Vascular dementia, in the early stages and thus are not
affected by physical weaknesses, paralysis, severe problems
with communication, concentration, thinking, or
hallucinations. Furthermore, the patients are assumed able
to walk, cook independently, to ask for help, and to take
daily medicines.
As for the caretaker, the module is designed to operate on a
smartphone or tablet environments. The application helps
the caretaker handle the task of taking care of the patient,
for example by giving the opportunity to call the patient,
add and send reminders and messages to the patient, browse
patient’s information, or manage the household devices.
System requirements
The system uses a server and a database to keep the
information updated between the caretaker app and the
home system app.
The system can also receive information from the following
sensors:
1. Facial recognition.
2. Temperature of the rooms.
3. Electricity and water consumption.
4. Pulse-rate of the patient.
5. GPS Tracking of the patient.
When the system detects unusual rates of water or
electricity consumption, or temperature, it notifies the
patient through sound and visual alerts. If the problem is
not solved after a considerable amount of time, the system
also alerts the caretaker, sending a notification to his
application.
The system can measure pulse-rate of the patient using the
sensors within the bracelet. However the procedure for
unusual rates is slightly different: If measured values are
unusual (too low/ too high), the system automatically alerts
the patient and caretaker/doctors.
The system offers some general-purpose actions such as
browsing objects, contacts, possible activities to prevent
laziness, past events and actions recorded.
The system can recognize approaching visitors using facial
recognition sensors in the house door.
• If it recognizes a known person, it notifies the patient by
showing information on the nearest screen.
• If it is an unknown person, it notifies the caretaker and
the patient as well.
The system allows the patient to store reminders and events.
The patient can notify the system about actions taken by
clicking the reminders button on his bracelet. Once the
button is clicked, the nearest screen turns on by showing
appropriate options (e.g. patient can notify about medicine
taken in the bathroom, or about upcoming events in the
living room etc.)
• The system allows the patient to store all the relevant
information about a reminder (time, place, type, alert,
etc.) through the interaction with the touch-screen.
• If the reminder is set, it will be shown in the nearest
screen at the set date and time and notified by sound and
a displayed message.

3
• The patient can delete alerts by interacting with the
screen.
• The system notifies the patient about upcoming
reminders stored in the nearest screen and at fixed times
• The patient can view the reminder by touching the
screen and delete it or postpone it.
The system allows the patient to alert caretaker and
emergency by clicking the red button of the bracelet.
The system allows the caretaker to check information about
the patient using an application in his/her mobile device.
• Patient’s health information (medicine needed, recent
visits, etc.).
• Patient’s location history.
• Manage the status of the devices in the house (e.g. turn
off the oven).
• Browse past events and reminders.
• Add new reminders for the patient.
• Browse patient’s house information (id, address, etc.).
• Check sensors’ measurements.
Use Cases
The following diagram shows the use cases from the patient
point of view:
Figure 1: Patient’s Use Cases
The following diagram shows the use cases from the
caretaker point of view:
Figure 2: Caretaker’s Use Cases
The following diagram shows the use cases from the
automated home-system point of view:
Figure 3: Home-system Use Cases
Personas
Brian, 50 years old man with the most common type of
Alzheimer's disease. The disease is in the early stages and
manifests itself through occasional memory loss. Even
though Brian sometimes has problems with his memory, he
manages to live by himself in his house. Brian is aware of
his condition, but sometimes gets confused about basic
memorable things, like location of certain objects. Because
of the disease, Brian is on medication and must be aided to
remember to take his daily dose.
Gretchen, 30 years old nurse. Caretaker of Brian. Gretchen
has a tablet and is capable of using it every day. She keeps
it with her while she is visiting her patient. Gretchen is
familiar with using different applications on her tablet
embraces easily new apps. Gretchen is also used to all kinds
of fast messaging platforms and she is constantly in the
swim. (up-to-date). For her, using different applications and
devices to help her with her job is not frightening nor
challenging.

4
General assumptions:
General assumptions and characteristics of the targeted
users.
Patient with Alzheimer (most common 75%) is capable of:
• Walking.
• Cook independently.
• Asking for help.
• Take her/his medicines.
Patient with Vascular dementia (second most common
15%), in the early stages:
• Patient do not has physical weaknesses or paralyses.
• Not severe problems with communication,
concentration or thinking.
• No hallucinations.
Scenarios
The patient: Brian is a 50 years old man suffering from
dementia, still at its early stage. While sitting on his couch
and watching TV, his phone rings: his daughter is coming
over tomorrow for dinner; Brian presses a yellow button on
his bracelet. After that, Brian stands up and reaches a screen
behind him: while approaching the screen turns yellow and
a virtual assistant comes up, asking for the detail of the
event Brian wants to be remembered about: “Which event
would you like to remind?” “Day and time?”, “When/How
many reminders?”.
Sometime later Brian wants to eat something: as he steps
into the kitchen, the screen right next to the fridge turns on
and displays an alert: “The oven has been used 30 minutes
ago”. Brian now recalls that he had already decided to bake
some food an hour ago and realizes that IT IS now ready in
the oven: so Brian can eat the food, happily noticing that it
is still hot.
It is now time to take his medicine: Brian cannot remember
if he already took them. However, he knows that if he did,
he would have pressed the pink button on his bracelet.
Therefore, he enters the bathroom and approaches the
medicine shelf: the nearby screen turns on and displays a
message: “You last took your medicine today at 4pm”.
Some days later, as he step into the bathroom a message
pops up on the screen nearby the medicine shelf: “Hi Brian,
this is a reminder about the doctor’s visit tomorrow at
14pm, remember to be home at that time. Have a good day,
Gretchen!”
Brian hides the message, and decides to go for a walk to the
nearby shops: since he know it’s possible he will forget the
shop’s position or the destination he wants to reach, he uses
the screen to set the route to be followed, so that whenever
he gets lost the GPS sensor inside his bracelet will alert him
using vocal directions.
It’s Friday morning and the bell rings; Brian is not
expecting any visit so he checks the nearby screen in the
living room where he’s sitting and the system automatically
recognizes the visitor as his nephew Kevin; Brian can
welcome him and enjoy his visit.
The caretaker: Gretchen is a 30-years-old nurse; since some
months, she is taking care of Brian.
It is 4 pm and she wants to check how Brian is doing: she
takes her tablet and opens the application.
Some notifications let her know that Brian took his pills and
that he turned on the oven; she then wants to check if
everything is ok with the devices in Brian’s house, for
instance if the water is not running and if the oven has been
also turned off after the use. After those checks, she post a
message for Brian to make him aware of the doctor’s visit
next Friday: she enters the Messages section of the
application and leave a geo-located message by uploading it
in the access point close to the medicine shelf, so that it will
pop-up in the screen next time Brian approaches it.
She can also set the range of time the message should be
displayed to make sure that Brian would not forget it even
after reading it once. Gretchen is happy about the system
and feels confident that this kind of monitoring can be
really effective even in case of a sudden crisis, because
Brian can always use the buttons on his bracelet and on the
screens to ask immediate help both from her and the doctor.

5
Possible mislead use
1. Issues with the patient
• Alerts and notifications are not heard / ignored by
the patient
• Alerts and notifications get annoying and
repetitive, making the system useless
• Tracking and surveillance makes the patient
uncomfortable: he removes his bracelet and/or
never actively uses it
• Patient encounters difficulties using the screens /
doesn’t think they’re helpful in any way
2. Issues with the caretaker
• Caretaker does not fully understand the interface
provided: he/she switches on/off the devices
without any purpose, causing issues to the patient
(e.g. suddenly the lights turn off etc.)
• Caretaker doesn’t get notifications or tracking info
because of the lack of interaction with the bracelet
by the patient
• Caretaker leaves too many messages and
reminders, annoying the patient
3. Issues with the system
• System is full of alerts and messages: house
becomes a rainbow mess!
• Proximity recognition failure: screen doesn’t turn
on or turn on when the patient is far away
Evaluation Techniques
First Phase
The first evaluation phase was carried out using observation
methods and query techniques. The first experiments
involved a think aloud technique, where participants were
asked to perform tasks using paper prototypes of the Patient
module.
Since the systems has many functionalities (some of them
very similar to use) and the time required from a volunteer
to test them all would be to much, tasks (from both systems,
patient’s and caretaker’s) were split in two groups.
Since getting volunteers who suffer from dementia is out of
our reach, the experiment tried to simulate the dementia
condition (memory loss, disorientation, etc.) in the
following way:
1. The volunteer was introduced a possible scenario
and given a task to perform.
2. While performing that task, the volunteer was
changed without explaining nor introduce the
scenario to the new volunteer.
3. The new volunteer was asked to try to understand
and finish the task that was being performed by the
previous volunteer.
This experiment was carried out using 8 volunteers.
Think aloud observation was also used to evaluate the
Caretaker module. In this case there was no need to perform
any kind of simulation of special conditions (we assume
caretakers are standard users).
After the experiments, each participant was asked to fill in a
questionnaire regarding the user experience with the
system. The answers gave us some insights on the user’s
perception of the system and some implications for design:
• The caretaker application only targets one patient.
• Patient location tracking was not well explained.
• There is no login system.
• Objects view in patient system was not clearly
explained. One volunteer did not understand what it
was for (button).
• The possibility to turn the home-system screens, from
both caretaker app and the own system, was missing.
• The possibility to turn on devices (while managing
them) can mistakenly lead to dangerous situations (e.g.
turning on the oven)
• They patient might not remember the contact even
when looking at the photo.
As a result to this feedback some changes were made to
both interfaces:
• Add labels to every button just in case the patient
forgets what it is for.
• Add an information button in each screen, in that way
the patient can check what was s/he doing if s/he loses
her/his memory in the middle of a task.
• Add a label to the contacts that are relatives, so the
patient always is reminded of who are her/his relatives.
• The possibility of turning on devices from the caretaker
app will be removed. In this way, generation of
dangerous (by mistake) situations is avoided.
• The home-screens will have the possibility to be turned
off by the patient.
Second Phase
After the first rough evaluation, we decided to perform
some expert evaluation within the group, using cognitive
walkthrough and heuristic evaluation. Six more evaluations
were also carried out using the same method than in the

6
first phase (changing volunteers to simulate memory loss,
etc.).
The analysis was performed against both the Patient and the
Caretaker application. The tasks taken into account, when
performing cognitive walkthrough, in the Patient app
evaluation are the following: add reminder, ask for help
(using the bracelet), add reminder, and browse history,
contacts, activities and objects (using the screens).
The following problems arose from the evaluations were:
• Patients did not know how to continue when retaking
the task in the middle of its performance (when the
change between volunteers happened).
• Some inconsistency between buttons’ shape and labels
were found (info and home button were not always
labeled and info button size was not big enough).
• Activity icon was not intuitive. Most of the volunteers
misunderstood it with pulse-rate and patient status
information.
• Objects icon was not intuitive. Volunteers did not
know what its functionality was.
• Objects label (i.e. objects) was not intuitive. Volunteers
did not know what its functionality was.
• In the objects view, a patient might not recognize the
object by its name, or might not know how to use it or
what its functionality is.
• Some volunteers had doubts about the accuracy of the
colors in the patient application.
• The patient might forget the functionalities of the
buttons of the bracelet.
• The patient might not see the buttons if it is dark.
• The patient might not understand if he performed the
action properly when using the bracelet.
• Home-system displays might be too large (when testing
with a TV) or too small (when testing with an iPad).
• Asking for help only alerts the caretaker, without
contacting him directly.
• Patient could forget about the screen presence/use
when they are in an idle state (thus showing nothing).
With regards of the Caretaker app evaluation, the only issue
that arose was the absence of self-explanatory icons to
allow an easier navigation (and to rely on users‘ recognition
instead of recall).
The proposed solutions for the problems found in the
second phase of the evaluation were:
• Buttons were rearranged, resized and labeled
consistently.
• Activity icon and label were changed.
• Objects icon and label were changed.
• Objects have and icon apart from the name so the user
might recognize it.
• Buttons on the bracelet were changed. They are now
labeled and they would have a led so the patient can
see them when the room is dark.
• Home-system displays will be about 17 inches. This
size allows displaying the application UI in a way that
every button can be reached and the user does not
need to move in order to interact with it.
Regarding the colors, as in the first phase, not everyone
agreed in its accuracy. However, almost all the volunteers
agreed that the colors and icons of the caretaker’s
application were appropriate. Some volunteers said that the
green screen used for reminder pending might be a little bit
intrusive, but since it has to be noticed when appearing, we
decided to keep the color.
In the following pictures, the evolution of the system can be
appreciated1
:
The first step was to divide the screen in regions in which to
show the different main functionalities.
Figure 4: Caretaker’s homepage regions.
The next step was to fill in the regions with example
content in order to be able to evaluate it.
1
In order not to make this report too large, not all the old
mockups are shown. All of them can be found in
Support_Files.zip/Mockups/

7
Figure 5: Caretaker’s homepage version 1.
After evaluating, a new version of the homepage was made.
In this case, the regions were changed. Instead of the
content itself, icons to access functionalities and content are
shown.
Figure 6: Caretaker’s homepage version 2 without icons.
The next step was to paint the interface using the previously
chosen colors (see Figure 5).
Figure 7: Caretaker’s homepage version 2 with colors.
After the second phase of the evaluation, icons were chosen
and added to the home screen.
Figure 8: Caretaker’s homepage with icons.
The case of the patient application evolution is quite
simpler since there were fewer changes to be made.
The fist design of the homepage had icons and labels for the
offered features.
Figure 9: Patient’s homepage version 1.

8
The next step in the design was to fill the interface with
colors and make the main buttons distinguishable.
Figure 10: Patient’s homepage version 2.
After the evaluation some changes were made, as the
change of icons and labels.
Figure 11: Patient’s homepage version 3.

9
The last part of the system to take into account is the
bracelet. In this case, only two versions were made.
Figure 12: Patient’s bracelet version 1.
As the buttons could be confusing without labels, they were
changed as shown below:
Figure 13: Patient's bracelet version 2.
RESULTS
Our application is a smart-home interactive system to assist
people with dementia. The application is composed of two
main interfaces for both the patient and the caretaker. Both
patient and caretaker’s applications are constantly
connected to each other, thus receiving and sending
information to each other.
Patient’s interface consists of an automated home-system
and one personal portable device (i.e. bracelet). The home-
system consists of an overall sensing and tracking
application and multiple touch-screens connected to it. The
main function of the application is to help patients live
independently, improve their security, and provide help and
information about daily objects and activities.
Caretaker’s application is designed to operate on
smartphone and/or tablet environments. The application
helps the caretaker handle the task of taking care of the
patient, for example, the caretaker can call the patient, edit
messages and add reminders for the patient, browse
patient’s information, or manage the household devices.
Both the applications are connected to the server, which
keeps both client applications updated.
Figure 14: System architecture
There are several general assumptions and characteristics of
the targeted users of the application. For the dementia
patient, the assumptions are that patient have Alzheimer’s
disease, or Vascular dementia in the early stages.
Furthermore, the patients are assumed able to walk, cook
independently, capable of asking help, and taking her/his
daily medicine. Patients are also not expected to have any
physical weaknesses or paralyses, severe problems with
communication, concentration or thinking, or
hallucinations.

10
Wireframes
Figure 15: Patient app wireframe

11
Figure 16: Caretaker app wireframe

12
Home-system
The home-system, or patient’s application, will remind the
patient about the ones that has been add into the system,
either by the patient himself or by his/her caretaker.
Figure 17: Reminder pending screen of the patient's
application
When the patient touch the screen the reminder will be
shown, giving the possibility to postpone it, delete it, or go
to the homepage of the system.
Figure 18: Reminder screen of the patient's application
After performing an action the patient can finish the
interaction with the system and retake whatever action s/he
was performing before, or keep using the system by going
to the homepage.

13
Figure 19: Patient's application homepage
Once in the homepage the patient can perform various
actions. Among this actions are: looking for an outdoor or
indoor activity, browse contacts or appliances, check
previous reminders and tasks, or add a reminder for
himself2
.
2
To see all the mockups of the mentioned features go to
Support_Files/Mockups/Milestone_3.
Figure 20: Add reminder screen of the patient's application

14
Caretaker application
This part of the system allows the caretaker to stay in touch (i.e. by calling), and control and check patient and household
status.
Figure 21: Caretaker's application homepage.

15
In order to check patient status, the caretaker(s) can check the patient information.
Figure 22: Patient's information screen of the caretaker's application.

16
The caretaker(s) can also check his/her notifications (when and which medicines did s/he take, for how long and what devices
are on/off, etc.).
Figure 23: Caretaker's notifications screen

17
In order to check the household status the caretaker can check its information
Figure 24: Patient's house information screen of the caretaker's application.

18
The caretaker can also check the status of the devices that the house contains, and if necessary change it (e.g. turn off the
oven because it has been on for more than 2 hours and the patient is not cooking something that requires it).
Figure 25: Manage devices screen of the caretaker's application.

19
If the caretaker needs to communicate with the patient, s/he can call him by using the application. Alternatively, simply add a
message or a reminder that will be shown in the patient’s home-system.
Figure 26: Messages and reminder screen of the caretaker's application

20
Bracelet
The third and last part of the system is an interactive
bracelet that will always be carried by the patient.
Figure 27: Patient's bracelet
By using it, the patient can ask for help by clicking the red
“help” button. It will alert the caretaker and the doctors
(e.g. hospital) that something bad had happened.
If the patient is in the house and clicks the green “remind
me” button the nearest screen will turn on and allow
him/her to add a reminder.
DISCUSSION
In general, the designed system tries to take into account
the main requirements an assistive technology for dementia
patients should have: reminders, leisure, communication
and security [4].
The design of the caretaker’s application is fast and easy to
navigate through; this is an important requisite in order to
enable the caretaker to take a quick view of the patient’s
status and events in his/her house without spending too
much time. That is possible because of the different set of
options in the homepage that allows the user to perform
almost every task using just a few clicks. In addition, all the
touchable parts of the interface are big enough to avoid
mistakes when clicking, and that is what makes the
application easy to use on tablet or other mobile device.
The system is trying to help one or more caretakers with
only one patient to look after. The caretaker could be the
patient’s relative(s) or other non-trained user instead of an
actual nurse. Due to this, the application should not be too
complicated nor technical for people without healthcare
education.
Regarding future work and changes to be made to the
system, there should be more help in case of memory loss
and the colors of the applications could be changed.
Concerning the memory loss problem, we noticed that some
volunteers were unsuccessful in finishing the task that
another volunteer started. The main scenario where this
problem arose was when writing a reminder, that is why the
process followed to perform that action should be iterative
(i.e. click the “add reminder” button, then specify about
what the reminder would be about by clicking a button from
a list and then the reminder could be set). In this way the
patient could get more feedback about what was happening
before losing his/her memory.
Concerning the colors, we could not come up with a
suitable combination that every participant liked. This is
due to some inconclusive results (i.e. some participants
liked them, others did not). Since colors are just a matter of
taste, the solution would be to make them customizable by
the patient and the caretaker according to their preferences.
However, not all the colors could be changed (e.g. the
reminder pending view of the patient’s application must be
intrusive so it is noticed).
ACKNOWLEDGMENTS
We thank to all the volunteers that made the evaluation
process possible without asking for anything in return.
The original bracelet image (without buttons) and some of
the icons of the patient application are copyrighted.
REFERENCES
1. Bjørneby, S., Duff, P., & Mäki, O. (2003). Developing
assistive technology for people with dementia.
Assistive Technology: Shaping the Future, 781-
786.Mather, B.D. Making up titles for conference
papers. Ext. Abstracts CHI 2000, ACM Press (2000),
1-2.
2. Cahill, S., Macijauskiene, J., Nygård, A. M., Faulkner,
J. P., & Hagen, I. (2007). Technology in dementia care.
Technology and Disability, 19(2), 55-60
3. Sixsmith, A. J., Gibson, G., Orpwood, R. D., &
Torrington, J. M. (2007). Developing a technology
‘wish-list’to enhance the quality of life of people with
dementia. Gerontechnology, 6(1), 2-19.
4. http://www.atdementia.org.uk