The document discusses using context-based log analysis of free user sessions to evaluate a mobile tourism guide app. The analysis revealed how user needs and behaviors varied by context like location, time, and frequency of use. This provided insights for improving the app's functionalities, interface adaptivity, and identified some usability issues. The methodology demonstrated the potential of asynchronously analyzing real-world usage data to inform the evolution of mobile services.
1. Discovering Functional Requirements
and Usability Problems
for a Mobile Tourism Guide
through Context-Based Log Analysis
Elena Not Adriano Venturini
Fondazione Bruno Kessler eCTRL Solutions
Trento, Italy Trento, Italy
Friday, 25th January 2013 ENTER 2013 Research Track Slide Number 1
2. Summary
• The complexity of designing and evaluating
mobile services for tourism
• Context-based log analysis as a method for
remote asynchronous evaluation of system
use
• Testing the methodology with a real system
• Findings and implications
ENTER 2013 Research Track Slide Number 2
3. Mobile services:
powerful but challenging
What makes mobile services powerful, makes
them difficult to design and evaluate
– Ubiquity: possibility to access anywhere
– Convenience: access anytime, at the point of
need
– Localization: location as a key to access
focused services
– Personalization: contextual factors can shape
the service
ENTER 2013 Research Track Slide Number 3
4. Complexity in design
Broad design space
– Functionalities
– Customization
• Content data (e.g., which products are recommended)
• Information presentation (graphical rendering,
language)
• Interaction mechanisms (browsing options or activities
proposed to users)
• Proactivity of the systems (pull/push mode)
Difficult to envisage all possible scenarios in advance
ENTER 2013 Research Track Slide Number 4
5. Complexity in evaluation
Evaluation should occur
– in an ecological setting,
– with users using the system whenever the
actual need arises,
– and tracking the contingent contextual factors
ENTER 2013 Research Track Slide Number 5
6. Evaluation of mobile services
Controlled experiments Synchronous remote Asynchronous remote
usability testing unmoderated usability
testing
• Controlled task • Controlled task • Controlled task
• Co-location of user • Researcher • User doing test when
and researcher (e.g. lab) interacting/observing convenient
• Direct observation user from remote • Researchers analyzing
data afterwards
• Good for major • User can be on the • User can choose time
usability problems move and place freely
• Does not require • Does not require • Still, tasks are not free
complete system complete system
MOD 1000
ENTER 2013 Research Track Slide Number 6
7. Can asynchronous remote
evaluation of free usage help?
Asynchronous remote evaluation of free Benefits:
usage
• Discover information
• User using system when need arises
• Free usage, according to actual needs needs new
• Monitoring of logs and contextual functionalities
factors • Discover patterns of usage
• Researchers analyzing data afterwards
adaptive interaction
• Large sample of users
• System in actual usage • Understand usability
problems
• Complexity of interpretation, but…
• Contextual factors help to understand
ENTER 2013 Research Track Slide Number 7
8. Methodology
• Identify potentially relevant contextual
factors (Baltrunas et al., 2012)*
• Set research hypothesis
• Clean up the log sample
• Context-based analysis of web-logs
• Derive implications for re-design
* Baltrunas, L., Ludwig, B., Peer, S. & Ricci, F. (2012) Context Relevance Assessment and Exploitation in Mobile
Recommender Systems. In Personal and Ubiquitous Computing (2012) 16: 507-526
ENTER 2013 Research Track Slide Number 8
9. Testing the method
• Biella Mobile: mobile services for a
medium-sized DMO www.atl.biella.it
– Average of 74.000 unique visitors per year to
the main portal
• Mobile services online from March 2012;
automatic redirection for smarphone users
– Average of 860 unique visitors per month
ENTER 2013 Research Track Slide Number 9
10. Funtionalities of Biella Mobile v1.0
http://www.atl.biella.it
• Simple information
structure
• Access to products
catalogue, organized
by category
• Products ordered by
distance when user
location is known
• Internal search filters
• Details pages and
maps for products
ENTER 2013 Research Track Slide Number 10
11. Evaluation experiment
• Collection of logs of four weeks of free usage
(25 June – 22 July 2012): 747 sessions
• 355 sessions (with more that one visited page)
considered for analysis (108 with known
location)
• Objective of study, to identify:
– Additional functionalities for next release of system
– Desirable forms of adaptivity of the system
– Usability problems
ENTER 2013 Research Track Slide Number 11
12. Collected information
• Interaction information:
– visit duration
– actual sequence of visited pages
– action buttons used (e.g., “show more”, phone call, an email or the
redirection to the personal web site of a POI)
– usage of text strings to filter search results
– position in the result list and the user distance from inspected POIs
• Contextual information
– current position, distance from Biella
– day and time of access
– new/returning user
– type of searched content
ENTER 2013 Research Track Slide Number 12
13. Specific research hypothesis
• H1: The number of visits to Biella Mobile and
the type of information searched by users
Context
influences
informational
depends on contextual factors, in particular needs
week day, location and use frequency
• H2: Map-based functionalities have a relevant Maps are
pivotal for
?
role in supporting mobile users’ informational mobile users
needs.
• H3: Different product categories are
characterized by different search and
Interface
adaptivity
decision-making patterns. makes sense
ENTER 2013 Research Track Slide Number 13
14. Findings: Influence of context
• The system is significantly* used
more:
• during weekends
• by users in the area of Biella
• Events are by far* the most
searched category for onsite users
(especially local, recurring users)
• Far away users have more varied
informational needs
* chi-square test with α = 0,001 of significance threshold
ENTER 2013 Research Track Slide Number 14
15. Implications
• New functionalities
– Alert service about events/activities
• Forms of adaptivity
– Push of news for local, frequently
returning users
– Prominence of weekend events
ENTER 2013 Research Track Slide Number 15
16. Findings: Patterns of usage
Varied search and decision-making patterns* :
• Events: very few map visualizations, few events
inspected per session the result list is the main source
for decision
• Accommodation, Sports, Itineraries: visualization of maps
and details
• Restaurants: visualization of details is crucial for decision
• Places, interests: many items inspected per session
comparison seems important
* chi-square test with α = 0,001 of significance threshold
ENTER 2013 Research Track Slide Number 16
17. Implications (2)
Forms of adaptivity
• Different display/search methods for different
product categories
– Events: good summary in result list; prominence to
weekend events
– Accommodation, Sports, Itineraries: map-based
search
– Restaurants, places, interests: comparison or
content filters may speed up search
ENTER 2013 Research Track Slide Number 17
18. Findings: usability problems
• Analysis of bounces
– Many bounces are due to search engine indexing done
on web pages of main portal
An effective redirection to corresponding mobile pages
should be guaranteed
• Analysis of internal search
– Users use internal search expecting google-like
behaviour (e.g. “agriturismi”)
Assure flexible internal search or powerful content
filters
ENTER 2013 Research Track Slide Number 18
19. Findings: qualitative analysis
of returning visits
Qualitative identification of specific user segments:
• Frequent local users:
– Repeated access from the Biella area, looking for events and
activities for the upcoming weekend
– Repeated access on Sunday morning, looking for events and
activities
• Incoming tourists:
– Repeated access to accommodation list one day before or during
the weekend
– Access from remote few days before, and when onsite check the
same information again
ENTER 2013 Research Track Slide Number 19
20. Implications (3)
New functionalities
– Personalized
recommendation for local
frequent users
– Wish-list service for quick
access to already
preselected items (Mobile
Travel Planner)
ENTER 2013 Research Track Slide Number 20
21. Architecture (www.suggesto.eu)
Mobile WebApp Travel Planner Travel Widget
DMO/TourOperator/Hotels For DMO/TourOperators sites For partner websites
DMO/Tour
Suggesto Portal (Liferay Based) Operator
Contents
Suggesto XML HarmoSearch
Suggesto Recommender
CMS Feed Network
Relational XML
Database Database
ENTER 2013 Research Track Slide Number 21
22. Conclusion
• Asynchronous remote evaluation of free usage,
through context-based log analysis, may reveal:
– Context-based informational needs
– Context-based patterns of usage
– Usability problems that can be detected only in actual
system usage
• Input for system evolution:
– New functionalities
– Forms of adaptivity
– Usability improvement
ENTER 2013 Research Track Slide Number 22
23. Discovering Functional Requirements
and Usability Problems
for a Mobile Tourism Guide
through Context-Based Log Analysis
Elena Not Adriano Venturini
Fondazione Bruno Kessler eCTRL Solutions
Trento, Italy Trento, Italy
www.fbk.eu www.ectrlsolutions.com
www.suggesto.eu
Friday, 25th January 2013 ENTER 2013 Research Track Slide Number 23