Accessible version: http://www.last-child.com/a11y-data-metrics/ Learn how top companies are tracking and graphing product accessibility progress and incorporating data from automated, manual, and user testing to create management dashboards.

1. Accessibility Metrics
Ted Drake and Sagar Barbhaya
Intuit Accessibility
CSUN 2017
Accessible version: http://www.last-
child.com/a11y-data-metrics/

2. What Do You Need?
Write it on a post it note

3. Three Stages
1. Accessibility Management
Goals
2. State of the Industry
3. Solutions to fill gaps

4. Current State
• Manual Testing and Evaluations
• Continual improvements
• Documentation
• Community Outreach
• Stakeholders
• Centralized Accessibility Leadership

5. What we have
• Product database with evaluations, scores,
stakeholders, and documents
• Bug Tracking
• Scattered Employee Resources
• Centralized documentation
• Non-integrated automation

6. What we need
• Dashboards to surface data and trends
• Connections between platforms
• Automation and unit testing for developers
• Shared test results
• Connected resources
• Support channel integration

7. Future Goal
• Measurable lifecycle from tests/evaluation to
production
• Deep consumer feedback integration
• Diverse workforce with support channels
• Developer ownership of testing and solutions
• Product ownership of process and success

8. State of the Industry

9. How are we doing?
- Product Manager

10. What to Measure?

11. Product Quality Metrics
• Issues by Priority – 77%
• Issues by WCAG category – 54%
• last time product had a manual evaluation – 46%
• Closed Captioning support – 38%
• Number of accessibility bugs opened per month/
year – 31%

12. Priority Levels
• Impact on Customer
• Template / Module / Unique
• Density of Issues
• High Traffic, Key Flows

13. Test to Ticket flow

14. Company Metrics
•AODA compliance – 55%
•Number of media inquiries about product/company
accessibility – 45%
•Conference talks by employees on accessibility – 45%
•Product VPAT Coverage – 45%
•Compliance towards 21st Century Communications Act
(CVAA) – 36%

15. Compliance and Legal

16. Human Resources
• Section 503: Self-Identification
• Accommodations Requests
• Ergonomic Evaluations
• Service/Perk Requests

17. Customer Metrics
•Usability participants with a disability – 63%
•Feedback via customer support – 50%
•Usability activities, including user personas,
with disabilities – 38%
•Customer support calls - 38%
•Tickets filed by customer support – 38%

18. Net Promoter Score

19. Storage and Sharing

20. QuickBase

21. Spreadsheets

22. Pa11y
Testing +
Dashboard

23. Google
Docs

24. Cloud Storage
• DropBox
• Box
• Amazon Web Services
• Google Drive
• Slack

25. WorldSpace

26. Solutions

27. Bug Tracking

28. JIRA

29. JIRA Queries
• labels = accessibility
• labels=accessibility and status=Open
• labels=accessibility and status = blocked
• labels=accessibility and status=Closed and
resolved > “-4w”

31. DOMO

32. Integrated Testing
• Developer testing: Tenon, aXe, Nemo
• Unit tests include accessibility
• Export test results to JIRA
• Tracking tool usage

33. Finding Outliers
• Extremely critical to prioritize
• Outliers need to be
identified and fixed first
• Example - Turbo Tax in
season production bugs

34. Customer Support Logs
• Search for Deaf and HOH customer requests:
[ LEVEL:SENTENCE TYPE:KEYWORD {deaf,
hearing}]
[ LEVEL:SENTENCE TYPE:AND_KEYWORD
{capability, impaired, challenged, hard}]
• “Accessibility” topic within Salesforce for reports
• Microsoft: Dedicated Support Channel

35. What Did You Need?

36. Questions?