The project aimed to develop route learning software called Route Mate to help people with disabilities travel independently. Route Mate allows users to plan routes, rehearse them, and receive guidance during travel. It underwent extensive testing and piloting to evaluate usability. Testing showed users required some assistance with tasks but engagement was high. Piloting demonstrated increased confidence, motivation, and travel skills with Route Mate's use. However, technical issues and lack of mobile experience caused some initial stress for users.

1. RECALL Project
2009-504970-LLP-1-2009-1-UK-KA3-KA3MP
Evaluation of route learning software
on Android for people with disabilities –
Jacqui Lewis, Camelia Popescu, Penny
Standen and Maria Saridaki
The project is partially funded under Key Activity 3: Informational and Communication Technologies (ICT)
Lifelong learning programme of the European Commission The content of this project does not
necessarily reflect the position of the Commission, nor does it involve any responsibility on its part

2. Project Partners:
Nottingham Trent University – UK
Greenhat Interactive Ltd – UK
BID Services - UK
Marie Curie Association – Bulgaria
University of Athens – Greece
CPPC – Romania

3. Rationale
Evidence presented by the Disability Rights Commission (DRC) indicates
that people with disabilities experience considerable disadvantage in terms
of transport and travel. For example:
• People with disabilities are unable to use 70 per cent of buses and 40
per cent of the rail network.
• 60 per cent of people with disabilities have no car in the household,
compared with 27 per cent of the general population.
• Almost half (49%) of people with disabilities are totally reliant on public
transport.
• Over half (56%) of people with disabilities said that they would like to
go out more.
• People with disabilities are twice as likely to turn down a job due to
travel difficulties.
• More than half (52%) of people with disabilities expressed some
difficulties in getting to all essential services such as GPs and hospitals.
• The research phase of our project showed that travel skills and
confidence to learn new routes are core skills for leading independent
lives; showing users their exact location relative to where they need to
get to can help people who struggle with spatial skills.

4. Project Aims
• To counteract exclusion that occurs when people with learning
disabilities leave statutory services, exacerbated by the anxiety of
carers about their vulnerability in the community.
• To overcome physical and psychological barriers to independent
travel and community inclusion.
• To reengage people into active citizenship
• To increase access to community activities by developing
independent travel skills

5. Objectives
• The project utilizes location based services in
assistive technologies to allow people with
learning disabilities and sensory impairments to
develop lifelong learning skills of: route
planning, memory, concentration, stress
management, time-management &
understanding targets and deadlines.
• Three modes of RECALL allow our target audience
to plan and use routes to reconnect with
community opportunities; and to play to
reinforce the connection between maps and the
real world.

6. Description of Application
• Provides a tool for developing
and supporting routes that
involves and enables the route
learner
• Allows carers and trainers to
have additional confidence
• Uses mainstream and
contemporary location aware
technology
• Provides other travel training
elements such as reminding the
user when to leave
• Provides geographically based
prompts
• Allows specific safety based
prompt e.g. at road junctions
• Provides a means to contact a
carer in the event of an
emergency
• Not a replacement for travel
training, but another tool
trainers can use to engage and
keep users on task

7. Route Mate vs Satnav approach
Route Mate attempts to Traditional route
use technology to guidance (location based)
enhance route learning turn by turn instruction
– Route creation is part – Route generation is
of the process automatic
– User defines their own – Instructions presented
waypoints to the user
– Supports cognitive – Suppresses cognitive
map formation map formation

8. Route Mate - Console
Using a basic desktop/laptop and internet
connection, the user, parent or carer can:
• Load the routes from a device
• Preview and edit routes
• Duplicate routes
• Save/load routes to/from PC
• Save new and modified routes to the phone
• Delete routes from the phone.

9. Console

10. Telephone Start Up

11. Plan Mode
•Plan mode gives the user the option to create a new
route, or load and modify an existing route with the
help of a parent, carer or trainer. They can enter the
start point using selection on a map and can also
enter the start time of their journey, set daily alarms
and end address of their new route.
•An emergency contact can be specified and points of
interest can also be set between the start and end
points using the phone’s camera to break the journey
up into a number of smaller routes connected by key
landmarks and more effectively scaffold its learning.

13. Route Mate ‘Use’ Mode
•The user can rehearse the route a number of
times accompanied by a trainer or teacher before
independent travel. The Use Mode allows the
user to travel more independently and rely less on
the application and more on their own skills.
•To facilitate independent movement, the screen
turns off while travelling between points of
interest in the Use Mode.
•This is important because over reliance might be
dangerous for example looking down at the
screen whilst crossing a road unaccompanied.

15. Route Mate ‘Use’ Mode

16. Challenge Mode - Gamification
To teach/reinforce the concept of
maps and route learning, a games
approach has been adopted that:
• Allows students to insert
different icons to waypoints in
planning mode (e.g., treasure)
• The student is then challenged
to find this treasure in the real
world, using game play around
treasure hunts, or pirates, for
example.
• Promotes the connection
between the map and the real
world

17. Project Evaluation
Completed Ongoing/Future:
• Issues Tracking for • Scientific Review
functionality
• Iterative Testing for • Internal and external
product usability and impact measurement
accessibility against key
• Piloting performance indicators
• Soft outcomes
measurement

18. Scientific Review – Role
The external evaluator will:
1. ensure that we are able to demonstrate the
scientific quality of our work so that others can
trust our outputs by being able to provide
enough evaluation detail to reassure them that
our conclusions are sound and to enable them
to judge whether our products would be
suitable for their client group
2. monitor our use of accepted inclusive design
standards and also the use of the issues tracker
for bug fixing
3. produce a state of the art review of other
developments to test for product innovation

19. Qualitative Evaluation –internal and
external
Further qualitative evaluation will take two
approaches to outcome measurement, both of
which will measure the delivery of the project
against a number of key performance indicators for
measurement of sustainability and social impact of
the project and its results.
•the first of these will be conducted internally with
project partners
•the second externally with project stakeholders.

20. Usability Testing methodology
Between month 9 and month 30 of the project, iterative testing was
conducted by all partners using different tools for experts and final users.
Tools:
• Instructions for deploying testing
• List of tasks that users should attempt with Route Mate
• Observations sheet to record any usability issues
• Template to collect data
Throughout this period of testing an issues tracker was used by all partners
and experts to record any problems with functionality of the device. This
allowed the programmer to bug fix as we went along to keep the process
flowing, so that the device would be ready for piloting

21. Testing Tasks
• Testing with final users was conducted in teams including
one assistant (to assist users and to ensure their safety) and
one evaluator (to observe and record the data).
Spontaneous feedback from the participants and their
facilitators was recorded. The participants were given
training in the use of the phone and were shown how to
navigate the application.
• Relatively quiet and short roads were chosen. A shorter
route allowed more time to be spent gathering feedback.
The length of routes chosen for testing varied between
200-1600 m and the time spent on the road varied
between 10 – 30 minutes.
• At least two testing sessions were carried out with every
user and between 5 to 10 users tested the application in
each country, following the same set routine of tasks.

22. Usability Testing Results
Analysis of Tasks
The main characteristics of Route Mate tested were:
• Open/ close the application
• Use the virtual keyboard
• Use the map (using arrows, using zoom buttons)
• Set points
• Take pictures
• Use panic button
• Take a route
A 4 level scale for registering answers was used:
Level 1= could not complete even with help
Level 2 = requiring considerable physical or verbal help to complete
Level 3 = requiring a little physical or verbal help (eg prompts) but could otherwise complete the
task unaided
Level 4 = completed with no help
The amount of time spent looking at the device (percent of total time of walking) was:
• 22 cases between 70% -90%
• 6 cases between 20%-50%.

23. Testing Results
• Analyzing the main characteristics of Route Mate we found out that for the
users the application was in most of the cases easy and engaging to test and
only in a few cases were no answers registered because of a functionality
error.
• Open/close the application was completed mostly with no help - 20
responses
• Use the virtual keyboard was completed mostly requiring a little physical or
verbal help -18 responses
• Using the map (using arrows, using zoom buttons) was completed mostly
requiring a little physical or verbal help - 17 responses
• Set points was completed mostly requiring a little physical or verbal help -
20 responses
• Take pictures was completed with requiring a little physical or verbal help -
16 responses
• Use panic button was completed mostly requiring a little physical or verbal
help - 21 responses. (4 non answers were recorded because of errors)
• Take a route was completed requiring a little physical or verbal help - 13
responses. (8 non answers were recorded because of errors)

24. Main areas of physical difficulty
Taking pictures was one of the characteristic where the answers varied
the most depending on which version of Route Mate was tested. This
required considerable physical or verbal help to complete in some
cases – 7 responses.
Using the virtual keyboard and using the map (using arrows, using
zoom buttons) required sometimes considerable physical or verbal
help to complete depending of the disability of the users:
• Using the virtual keyboard –required considerable physical or
verbal help to complete - 6 responses
• Using the map (using arrows, using zoom buttons) required
considerable physical or verbal help to complete - 8 responses

25. Users’ Typical Reactions
Some examples of reactions and behaviours of users:
• Users weren’t always sure of menu layout and where to find the
Route Mate icon.
• Users were unsure how to locate the ‘close keyboard’ icon to be
able to progress to the next screen.
• When moving around the map screen in Plan mode, accidental
clicking meant the option to add a point kept coming up, which
caused confusion with users
• Some users found it difficult to know where they were on the map
while taking a route and, therefore, how to proceed.
• Difficulties with using the keyboard due to fine motor limitations.
This cannot be rectified within the project – it is a hardware issue.

26. Functionality
• An online issues tracker (Red Mine) was used
to allow the software engineer to be
constantly updated on functionality problems
encountered during iterative testing.
• These issues were then prioritised and bugs
fixed accordingly.
• Testers were then informed as each issue was
dealt with and signed off.

27. Issues Tracker

28. Issues – Bug report

29. Functionality – Main Issues reported
Main issues have been:
- Crashes / freezes - usually a coding mistake. Crash reporting
tool (ACRA) integrated into the app to allow analysis of the
events leading up to the crash
- Photos not being recorded - only occurred on some
devices, not others. Image capture code was rewritten.
- Off route warning not consistently working - code corrected.
Quite a few issues were determined to be device specific, so it
was decided to pilot with a consistent configuration. Galaxy
Nexus devices were used as they have an up to date version of
android, and don't have any manufacturer (e.g. Sense UI (HTC)
or TouchWiz (Samsung)) or carrier (e.g. hard coded home
page, Telco apps) additions to the operating system.

30. Piloting Tasks
SCENARIO
Outline Scenario
• 3-5 Sessions
• 5-10 end users
Piloting Scenario
1st Session: Training and first use of Route Mate
• Training by the researchers - max 15 mins
• Soft Outcomes Star™ used by End Users (physical disabilities) or
caretakers (learning disabilities) - max 15 mins
• Piloting Process - Participant observation
• Familiarize with the software
• Create first route on console - by facilitator/end user
• Use first route - by facilitator/end user/researcher
• End of route, return to the premises - by facilitator/end
user/researcher
• Debriefing Process - max 15 mins

31. Piloting Tasks contd
2nd Session: Semi-Autonomous use
• Piloting Process - Participant observation
• Use existing route - by facilitator/end user/researcher
• End of route, return to the premises - by facilitator/end
user/researcher
• Debriefing Process - max 15 mins
End Session - Semi-Autonomous (if possible)
• Piloting Process - Participant observation
• Use existing route - by end facilitator/end user/researcher
• End of route, return to the premises - by facilitator/end
user/researcher
• Soft Outcomes Star™ used by End Users (physical disabilities)
or carers (learning disabilities) - max 20 mins
• Observational checklist - Likert Scale - max 20 mins

32. Observational checklist - Likert Scale
• Documents feedback - open questions
• Closed type questionnaire with interval scales
• Invites open comments from both end users and
facilitators

33. SOFT OUTCOMES STAR ™ tool
The soft outcomes star™ is a tool
to measure and record the
difference a service/training has
made to clients by showing
progress made in qualitative areas
of life across a period of time such
as from the beginning to end of a
course delivered.
The tool is used with clients in a
non-threatening way to help
diagnose where they might need
more support as well as what they
have achieved. It can also be used
as evidence of value of a project or
course and to demonstrate a
return on their investment to
project funding agencies in areas
that are hard to quantify.

34. Piloting Results – Likert Scale
•Carers and users were very positive about Route Mate.
•General attitude towards Route Mate was positive even in
cases of technical issues
•Users are willing to use Route Mate when the app has
been finalised
•Most of them credited Route Mate with an increase in
confidence and in helping relatives and families to be
reassured that users are safe when travelling independently
using the app. The panic button was one of the most
important features for the family

35. Piloting Results – Likert Scale
•There were insignificant differences in the way caretakers
(carers, teachers, trainers, support workers) answered the
questionnaire.
•The only demographic characteristic that was found to be
important is experience in technology.
•Experienced caretakers found Route Mate to be usable and easily
accessible and had more positive attitude towards it.
•Users that had previous experience with technology scored better
•All categories of users showed significant improvement in the
second measurement
•Older users performed worse in the first measurement but
improved more than the other categories in the final
measurement
•Impact was the same across disability categories

36. Piloting Results - Star Tool and
Observation
• Users with intellectual disabilities show no
qualitative differentiation in contrast with
individuals with other kinds of
impairments/disabilities, providing us with an
important indication that Route Mate can be
an equally useful learning and assistive tool
for users with different disabilities

37. Piloting Results – Star Tool
Star Tool: All categories increase significantly in the second measurement

38. Piloting Results
•During the piloting sessions and as the technical problems
started to slowly resolve:
–Stress levels decreased
–Users’ confidence increased
–Users felt encouraged and motivated to use Route Mate.
•As expected, previous experience with mobile technology
had an effect on users’ self confidence and therefore their
ease with Route Mate.
•Games and game based learning elements reduce the
steepness of the learning curve and increase satisfaction and
motivation

39. Piloting Results
Reasons that increased the levels of stress:
•Technological problems
•Disbelief about technology’s reliability in general
•Feelings of frustration about not being able to
access the right technology (smartphone) because
of prohibitive costs and accessibility
•Risk (fear of being victimized when holding an
expensive mobile)
According to Soft Outcomes Star™
findings, these stress levels can be reduced
through repeated use of Route Mate.

40. Piloting Methodology Results - Impact
End users could clearly see the immediate impact that Route
Mate could have on their lives:
• “Route Mate will be really helpful next year when I go to
college”
• “Will let my mother know that I can get there safely”,
End users could see the potential that Route Mate could have
on expanding and improving their daily lives:
• “I can see my friends more often”
• “I can learn new and unfamiliar routes”

41. Piloting Methodology Results -
Motivation
• Motivation to use Route Mate was also increased) by
the “Where to go Next?” screen and the photo
gallery, which according to most of the users made the
route more interesting and enjoyable
• Motivation from using Route Mate: two cases where
the use of Route Mate prompted them to improve
themselves or at least triggered the desire to improve
themselves. One student with learning disabilities who
was unable to read or write. After the first session with
RM she expressed to her parents the desire to learn
how to read. In a second case, after a RM session, the
end user started to search on the internet for similar
programs and assistive technologies.

42. Piloting Methodology Results
Bugs and optimization:
•The intermediate stops have to be actual places, things that the
end users can recall instantly and not commands or actions.
•The pop up info about the next stop has to appear before the
users reach a street, otherwise they tend to freeze in the middle of
a street raising important safety issues.
•For some students with sensory impairment or fine motor control
issues, the keys were too small
•Individuals in wheelchairs could not use Route Mate while moving.
They had to stop, use the application and then continue their
journey

43. Piloting Methodology Results
Proposals for further improvement:
• To add a voice command or to be able to
activate audio commands
• To add haptic feedback as an alternative for
people with visual impairments
• To enhance the more game-like aspects of
the app.

44. Thank you for listening!
RECALL team
http://recall-project.eu