Brighton SEO presentation given by Martin Fennon, the Head of SEO for Ayima. This talk covers crawling, indexation & the tangible impact reviewing these elements can have on performance
4. Crawl
Index
Rank
Display
Search engines crawl websites to read
your content
The content is basically stored in a giant
database of the web
Based on algorithms, Google decides what
keywords indexed pages can rank for
Indexed pages are then displayed in
search results
5. So, what is the relationship between
crawling & indexing?
6. A page is indexed by Google if it
has been visited by the Google
crawler, analyzed for content and
meaning, and stored in the Google
index
7. A page is indexed by Google if it
has been visited by the Google
crawler, analyzed for content and
meaning, and stored in the Google
index
11. Do Don’t
● Split XML sitemaps on
complex websites
limiting to 10-20k
URLs*
● Include pages deeper
in the site structure
● Include non-indexable
pages
● Forget to submit your
sitemaps via GSC
*https://ohgm.co.uk/an-alternative-approach-to-xml-sitemaps/
12. Internal & external links
Google will crawl and discover
URLs through internal &
external links
13. So, if a page is crawled by Google,
is it indexed?
14. Indexation can be
dependant on a number
of factors
...although sometimes
there's just no winning
with Google.
...not exactly.
16. Can Googlebot
access and render
the content?
Rendering
What can impact indexation?
Linking
Can Google
discover our new
pages?
Canonical tag to a
different page? Or a
meta robots tag?
Index
Directives
17. What can impact indexation?
Linking
Can Google
discover our new
pages?
Index
Directives
Can Googlebot
access and render
the content?
Rendering
18. Problem: A large scale e-commerce
website was experiencing a continuous
drop in performance and market share.
They came to Ayima.
19. ● Huge drops in performance over a prolonged
period of time
● They could not find any direct cause
Performance overview
20. Don’t be the agency/team that try
to deal with this problem with a:
“Lets chuck everything at the wall
and see what sticks” solution
22. ● Longer tail keywords were most prone to large
drops in 2020
● ‘!’ highlights ranking URL changes
Pattern of keyword ranking drops
23. ...but what does this have to do
with crawling & indexing you ask?
24. Ranking URLs were
moving from facet URLs
to category pages
This suggests one of two
things:
1. Google really hates
our website
2. Or Google cannot
access them
34. Robots.txt
● Can manage categories
and facet combinations
on mass by limiting
Google crawls
● Does not remove URL
from Google’s index
● Does not stop Google
indexing the URLs
Offers crawl directives. Tell crawlers what to
crawl and what not to crawl
35. Noindex
● Removes pages from
Google’s index
● Google has to crawl the
page to see the noindex
tag
● Trying to re-index a page,
or groups of pages, can
take time
Manages indexation, helps exclude pages from
Google’s index
37. Utilisation of meta robots tags at scale
will usually require CMS upgrades
But, they will remove duplication and
index bloat
38. Our solution:
CMS restructure to enable
noindex, nofollow on mass
to control indexation using
a mixture of automated
rules and manual overrides
39. Our rationale:
We wanted to remove
poor/low quality URLs from
Google’s index, control
indexing of new pages and
direct Google to pages with
traffic opportunities
