This document discusses optimizing a website's crawl budget for a mobile-first index. It recommends auditing the mobile site, understanding how mobile crawlers view it, and optimizing things like pruning content and internal redirects. It then discusses growing organic revenue as a case study, noting one site migrated to mobile-first and saw a 46% increase in organic traffic over 8 weeks after crawl budget optimization.
11. We'll use the mobile version of
the content for indexing and
ranking, to better help our
– primarily mobile – users
find what they're looking for.
“
Mobile-First Index Explained
14. M
O
B
I
L
E
F
I
R
S
T
AFTER
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
16. we transition sites slowly to
ensure a good experience for
site owners and users.
“
Mobile-First Index Transition
17. We evaluate each site
individually on its readiness for
mobile-first indexing based on
the best practices and transition
the site when the site is ready.
“
Mobile-First Index Transition
33. JavaScript is the browser’s behaviour language
M
O
B
I
L
E
T
E
S
T
I
N
G
What is JavaScript?
● Tells the browser what to display
● Tree of nodes, described by tags
● Nodes can contain:
○ Text
○ Metadata
○ Links
○ (Assets) - not relevant
○ (Style) - not relevant
● Uses HTML to tell the browser what to
display
● Allows a developer to dictate (‘Script’) a
browser’s behaviour after receiving a Web
Page
● Can make extra requests
● Can react to user interaction (clicks, mouse
movement, keyboard, etc…)
● Can store information on the user’s
computer
● Can modify the HTML
HTML is the browser’s display language
38. ❏ Prune Content
❏ Clean Internal Redirects
❏ Avoid Inconsistent Signals
❏ Use Robots.txt to block sections of your site you don’t want
crawled instead of page level noindex tags
❏ Improve your site structure
❏ 304 Status Code
❏ Use 410 to tell Google when content is permanently gone
C
R
A
W
L
B
U
D
G
E
T
Optimisation
42. Focus on profitability:
# Crawls vs # Visits
Do we have ?
1. few crawls vs. many visits
2. few crawls vs. few visits
3. many crawls vs. few visits
- low frequency ?
- thin content ?
- a spider trap ?
Allows us to focus on Active Page Ratio:
- nb. URLs visited / nb. URLs crawled
G
R
O
W
R
E
V
E
N
U
E
Identify and cut the waste