My #BrightonSEO talk on the best way to refine your crawl bloat and prune your remaining content effectively. Follow me at @WhitworthSEO for more tech seo nonsense.

1. Charlie Whitworth
SEO Director
@WhitworthSEO
http://www.slideshare.net/whitworthseo
Banc

2. what will we cover?
bancmedia.com
Crawl Bloat & Pruning
@WhitworthSEO

3. beat the bloat
Crawl Bloat
Making search engines work too hard
and not taking full advantage of your
crawl budget
@WhitworthSEO bancmedia.com

4. crawl definitions
=
Crawl Budget
bancmedia.com@WhitworthSEO
Crawl Rate & Crawl Demand

5. crawl rate (limit) : maximum fetching rate
Search Console Limit
s
bancmedia.com@WhitworthSEO
Website/Server Health

6. crawl demand
Popularity (Traffic) Staleness
@WhitworthSEO bancmedia.com
How much does Google want to see your URLs?

7. Crawl budget – determined by rate & demand
What Google can & wants to crawl
@WhitworthSEO bancmedia.com
• Empty pages
• No valued added
• Dupe
• Soft 404
• SPAM
• Hacked content

8. crawl budget
“Prioritizing what to crawl, when, and how much resource the
server hosting the site can allocate to crawling is more important
for bigger sites, or those that auto-generate pages based on URL
parameters”
Gary Ilyes
bancmedia.com@WhitworthSEO

9. how to identify crawl bloat
bancmedia.com@WhitworthSEO

10. how to identify crawl bloat
bancmedia.com@WhitworthSEO

11. why is this a problem?
bancmedia.com@WhitworthSEO
Don’t work them too hard
Search engine bots hate crawling rubbish

12. why is this a problem?
Engagement metrics are key
especially now we’re “mobile first”
@WhitworthSEO bancmedia.com

13. examples of rubbish: search result URLs
/search/
bancmedia.com@WhitworthSEO
/sessionkey=
/search-results?q=

14. examples of rubbish: paginated URLs
bancmedia.com
rel=next/prevnoindex
@WhitworthSEO

15. examples of rubbish: faceted navigations
json-ld
disallow parameters
bancmedia.com@WhitworthSEO
noindex

16. who’s seen this?

17. cut the crap
Get Rid Of It
@WhitworthSEO bancmedia.com

18. plan

19. noindex
<meta name=“robots" content=“noindex">
Won’t deal with crawl bloat in the short term
Still an effective tactic
bancmedia.com@WhitworthSEO

20. url parameters
@WhitworthSEO
Powerful tool when used properly
No need for developer support
bancmedia.com

21. xml sitemaps
Redirected URLs
Noindexed URLs
Poor quality pages
@WhitworthSEO bancmedia.com

22. robots.txt
bancmedia.com@WhitworthSEO
Exercise Caution!
Powerful crawl bloat tool when used effectively

23. robots.txt
Blocking Crawl Paths – Stems Authority Flow
bancmedia.com@WhitworthSEO
Google needs to see tags to honour them!

24. google’s hint
Google gives you all you need to
monitor your results
All these features are your friend
s
@WhitworthSEO bancmedia.com

25. crawl stats
Great way to gauge your efforts
@WhitworthSEO bancmedia.com

26. bancmedia.com@WhitworthSEO
Pruning
the prune

27. pruning – the fun bit
SEO Housekeepin
g
Now you have your crawl under control,
show off your best bits
bancmedia.com@WhitworthSEO

28. pruning – why?
• Spinning
• Excessive blogging
• Sparse articles
@WhitworthSEO bancmedia.com
• Over optimisation
• Saturation

29. stage one – content audit/content is king
Content auditing still essential nonetheless
bancmedia.com@WhitworthSEO
SEO’s most annoying cliché

30. stage two: the prune
410 – Gone
@WhitworthSEO bancmedia.com
For pages you want gone forever

31. stage two: the prune
bancmedia.com@WhitworthSEO
Won’t help crawl bloat short term
Will disappear eventually*

32. stage two: the prune
Crawl Bloat
Authority
Indexation
@WhitworthSEO bancmedia.com

33. stage two: the prune
NoIndex
Will be crawled less over time ensures a high quality SER
P
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.com

34. stage two: canonical
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.com
canonical

35. stage three: content repurposing/enhancement
@WhitworthSEO bancmedia.com
Evergreen
Seasonal
Long form
Research

36. search quality guidelines
@WhitworthSEO bancmedia.com
Search Quality Evaluator Guidelines

37. stage four: the finishing touch
Link Profile Analysis
Links pointing at excluded
pages
Usual disavow request
@WhitworthSEO bancmedia.com
Don’t ruin all your hard work with a crap link profile

38. crawl bloat and pruning checklist
Does your site have crawl bloat?
Choose the appropriate fixes for your site
Prune your remaining content
Continually improve and enhance
@WhitworthSEO bancmedia.com

39. final tips: thanks for coming
@WhitworthSEO bancmedia.com
Thanks for Listening!
2 Tips