This is my presentation for the paper with the same name on the HCII 2015 conference. Complete reference: Pichiliani, Mauro C., Hirata, Celso M. Evaluation of the Recognition Rate of the Android Accessibility API towards a better user experience. Proceedings of the HCI International, Los Angeles, California, EUA, 2015.

1. Evaluation of the Android
Accessibility API
Recognition Rate towards a
Better User Experience
Mauro C. Pichiliani (pichilia@ita.br)
Celso M. Hirata (hirata@ita.br)
Instituto Tecnológico de Aeronáutica - Department of Computer Science

2. Evaluate the UI recognition accuracy
rate of mobile applications by
measuring how many UI elements
are correctly identified by
an accessibility API
Goal

3. Introduction
 Online stores with thousands of applications
 Interaction model: touches and gestures
 Special users (visually impaired) and eyes-free
scenarios
 Accurate target indentification by accesibility APIs
benefits:
• Accessibility services and applications
• Automatic extraction of task sequences
• UI automation testing
• Collaboration frameworks
What is the recognition accuracy rate of UI
elements provided by Accessibility APIs
on popular applications?

4. Related work
 HCI literature: focus on size, location and visual
properties of UI elements
 Hurst et al. [5]: 74% of correct target identification on
desktop (location and size)
 Content information: used by accessibility services (i.e.
screen readers)
 Other approaches to increase the recognition of UI
elements
 Recent efforts for Web accessibility
 Strategies
 Guidelines
 Resources
[5] Hurst, A., Hudson, S. E., Mankoff, J. Automatically identifying targets users interact with during real world tasks. In:
Proceedings of the 15th international conference on Intelligent user interfaces, p. 11-20 (2010)

5. Mobile accessibility API
 Get low-level information about targets. E.g.: MSAA API
 Operating systems have accessibility applications
(screen readers, magnification glass)
 Android platform provives the most complete
accessibility API:
 Low-level hooks that capture events
 Complete identification of the element
 Reconstruction of the UI View hierarchy
 Default textual description
 New accessibility service creation
 Integration with external devices (e.g. braile
keyboards)

6. Evaluation methodology (1)
 Capture of application’s screenshots
 Accessibility service developed to read the contentDescription and
capture events raised
 Top 10 most popular applications (February 16th
, 2015)

7. Evaluation methodology (2)
 Dynamic content
 Dialog messages
 Web pages
 Common Android activities
 Reused elements on distinct activities

8. Results

9. Discussion (1)
 Overall, apps have a high event trigger ratio (99.48%)
 Dynamic elements and popup’s unreachable by the
accessibility API. Examples:

10. Discussion (2)
 Lack of content description for messages, images and buttons
(94.07%) . Examples:

11. Conclusion & Future work
 Mobile apps guided by visual access to on screen targets
 Impaired users and eyes-free scenarios
 97% recognition rate (99.48% event trigger rate, 94.07%
content rate)
 There is room to improve mobile accessibility APIs
 Better APIs impact other contexts
 Future work:
 Evaluation and comparison on other mobile OSs
 Validation with users
 Comparison of content descripton effectiveness
 Test techniques to augment accesibility APIs