As content producers, we invest considerable time and effort in developing, packaging, and delivering content that we think our users need. After publishing the content, we hope that users find our content useful. And we often wonder how users really navigate and consume our content. Web page analytics can help us gauge the information needs of our customers, assess their content consumption behavior, and find opportunities to improve our content and how we deliver it.
Kumar explores the basics of web analytics, pitfalls of relying too much on web analytics for important decisions, the typical web analytics process, and he will share some guidelines for interpreting web analytics numbers.
How to Troubleshoot Apps for the Modern Connected Worker
Making sense of analytics for documentation pages
1. Making sense of analytics for
documentation pages
Kumar Dhanagopal
API The Docs Virtual 2023 - Feedback, Metrics, Analytics
2. TOPICS
What analytics is and isn’t
Overview of key metrics
The analytics process
Interpreting metrics
Summary
2
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
3. How analytics can (and can’t) help…
✓ Understand how users engage with our content
• Who and from where
• How often
• On what devices
• …
✓ Understand user behavior on our site
• How they navigate
• What they click
• Where they spend time
• …
• Analytics can’t help us learn about user satisfaction and sentiment
• What do users need?
• Did they find it?
• Were they satisfied?
• How (and how much) did they read?
• Need other sources (e.g., user ratings, comments)
• …
3
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
4. Data != reality (correlation doesn’t imply causation)
4
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
5. The key metrics
5
A B C
D E
F G H
Page Views Visits Entries Exits Bounces
A 1 1 0 0 0
B 2 2 0 0 0
C 1 1 1 0 0
D 3 2 0 0 0
E 1 1 1 1 1
F 2 1 1 0 0
G 2 2 0 1 0
H 1 1 0 1 0
Session-1
Session-2
Session-3
Note:
❖ Repeat visits and long visits are treated differently
❖ Cookie’s matter
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
6. The analytics process
1. Define the problem or goal
2. Get the required data
• Metrics
• Dimensions
• Period
3. Prepare the data
• Remove noise
• Fix inconsistencies
• Aggregate data
4. Analyze, explore, visualize
5. Describe, diagnose, prescribe, predict
6
"If you torture the data long enough, it will confess – to anything" – Ronald H. Coase
Minimize the garbage-in-garbage-out risk.
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
7. Interpreting metrics: General guidelines
• Validate with other data
• Avoid cross-product comparisons
• Look at trends, not just absolute numbers
• Remember: offline and second-hand are not tracked!
• Consider the content type, length, format, and structure
• Use analytics data as supplementary input for decision making
7
Page Views
A 15k
B 14k
Jan Feb Mar Apr May Jun
3358 2610 3025 2411 2316 1900
0 0 1583 3308 5015 5029
Page A 25k 25k
Page B 25K 25k
Page C -- 25k
Page D -- 25k
Chunking affects analytics!
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
Doc 1 Doc 2
50k views 100k
8. Interpreting low page views
• Popularity != page views
• Look at trends, not just absolute numbers
• Consider the “age” of the page
• Check whether the page is discoverable
• Keep in mind structural changes
• Don’t ignore zero-view pages
8
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
9. Interpreting bounce rate
• Possibly the most misunderstood metric
• High bounce rate is not necessarily a problem
• Text-book guidance might not apply to
documentation websites
• Session time-out = bounce
• Analyze in conjunction with other metrics, like time
spent on page
9
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
10. Interpreting time spent on page
• High time spent:
• Was the content useful/interesting?
• Difficult to understand?
• User left the browser open too long? ☺
• High time spent + high page views
• Too much content, or too complex?
• Opportunity to improve chunking
• High time spent + high bounce rate
• Landing pages: cause for concern
• Other pages: Opportunity to simplify content
• Low time spent + high bounce rates
• Reference pages: positive indicator?
10
Time spent is NOT calculated for exit pages!
Number
of users
Total time
spent (min)
Avg. time
spent (min)
Action after
reading…
Group A 50 500 10.0 Exit site
Group B 50 250 5.0 Another page in
the site
Total 100 750 7.5
2.5
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
11. Analytics “targets”
Should we have analytics KPIs?
Documentation sites != e-commerce portals
Consider data availability and decision-making culture
11
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
12. SUMMARY
✓ Data != reality
✓ Analysis requires non-trivial effort
✓ Metrics won’t give us all the answers
✓ Focus on users, business goals
✓ The forest AND the trees matter!
12
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics
13. SUGGESTED READING
Web Analytics - An Hour A Day, Avinash Kaushik
Practical Text Analytics, Steven Struhl
An Introduction to Data Science, Jeffrey S. Saltz, Jeffrey Stanton
13
Kumar Dhanagopal, API The Docs Virtual 2023 - Feedback, Metrics, Analytics