SearchLeeds 2018 - Rachel Costello - DeepCrawl - Stop confusing search engines with conflicting website signals

1. CONFLICTING WEBSITE SIGNALS & CONFUSED SEARCH ENGINES Rachel Costello, Technical SEO SearchLeeds @rachellcostello SearchLeeds

2. When it comes to selecting URLs, Google often assumes you are wrong. @rachellcostello SearchLeeds

3. @rachellcostello SearchLeeds TECHNICAL IMPLEMENTATION Not all website owners have knowledge of SEO. Imagine what their technical implementation looks like.

4. Google uses an algorithm for URL selection to make sense of all the chaos out there. @rachellcostello SearchLeeds

5. If you sit back and do nothing, Google will do its best to figure out canonicals for you. @rachellcostello SearchLeeds

6. TAKE BACK CONTROL BUT the algorithm isn’t perfect, do you really want to be entirely at its mercy? @rachellcostello SearchLeeds

7. The algorithm’s results will be better if you do your part and use consistent signals. @rachellcostello SearchLeeds

8. @rachellcostello SearchLeeds Source: https://twitter.com/JohnMu/status/1003631027494014976

9. WHAT WE’LL COVER 1. How Google handles website signals and selects URLs 2. How you can test the signals your website is sending 3. How to see Google’s interpretation of signals and what it chooses to show @rachellcostello SearchLeeds

10. *DISCLAIMER* To anyone who’s not a fan of Google’s John Mueller: this talk might not be for you. He will be referenced repeatedly. @rachellcostello SearchLeeds

11. HOW GOOGLE HANDLES WEBSITE SIGNALS @rachellcostello SearchLeeds

12. @rachellcostello SearchLeeds 301 REDIRECTION DOESN'T ENSURE THE FINAL DESTINATION URL WILL BE INDEXED & SHOWN A 301 redirect doesn’t mean that the final destination URL will be the page that is favoured. You need to make sure all other signals are consistent. John Mueller, Google Webmaster Trends Analyst Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-9th-2018/#link10

13. The canonical tag is a signal, not a directive. @rachellcostello SearchLeeds

14. Things to note about canonicalization... @rachellcostello SearchLeeds

15. Any unique content on a page with an accepted canonical tag will be ignored. @rachellcostello SearchLeeds

16. Pages have to be very similar, if not identical, for canonicalisation to work. @rachellcostello SearchLeeds

17. Does Google process canonical tags in the rendered version or the raw HTML? @rachellcostello SearchLeeds

18. @rachellcostello SearchLeeds Source: https://twitter.com/rebelytics/status/1003951913510014976

19. @rachellcostello SearchLeeds If the <head> section prematurely ends, Googlebot may not read its contents.

20. For more information on this, read http://ohgm.co.uk/breaking-head-quietly/ @rachellcostello SearchLeeds

21. @rachellcostello SearchLeeds Canonicals SIGNALSSitemaps Parameter Handling Internal Linking BacklinksRedirects The signals Google uses for URL selection.

22. The end goal for all of this is to avoid content duplication. @rachellcostello SearchLeeds

23. @rachellcostello SearchLeeds MANY TO ONE CANONICAL TAGS MAY BE IGNORED Google will try to follow canonical directives by default, but they ignore canonical tags if they think it’s a mistake i.e. if there are significant content differences, or if a lot of URLs canonicalise to the same page. John Mueller, Google Webmaster Trends Analyst Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-thursday-17th-may/

24. @rachellcostello SearchLeeds UPDATE INTERNAL LINKS TO CANONICAL URLS Update internal links to canonicals to give Google a clean signal about which URL to index. If the canonical is accepted, any links to canonicalised URLs will be associated with the canonical URL. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-october-6th-2017/#link8 Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-august-25th-2017/#link8 ALIGN LINKING & REL CANONICAL TO GET A PARTICULAR PAGE INDEXED Ensure internal links and rel canonical are pointing to the preferred page for indexing to ensure you aren’t giving Google conflicting signals.

25. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-may-15th-2018/#link14 @rachellcostello SearchLeeds PARAMETER HANDLING SIGNALS ARE STRONGER THAN THE CANONICAL SIGNAL Google won’t blindly follow URL parameter handling set in GSC, but John says that this is a stronger signal than canonicalization. John Mueller, Google Webmaster Trends Analyst

26. @rachellcostello SearchLeeds CANONICALS ARE CHOSEN BY GOOGLE USING XML SITEMAP URLS XML sitemap URLs are used to help inform Google’s decision on which URL is chosen to be the canonical. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-9th-2018/#link1 Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-november-14th-2017/#link12 Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-july-7th-2017-2/#link11 URLS IN SITEMAPS ARE NOT GUARANTEED TO BE INDEXED Google may choose not to index URLs in sitemaps that are very similar to ones already indexed or if they differ to the ones linked internally. INTERNAL & SITEMAP LINKS MAY OVERRIDE CANONICAL TAGS Canonicalized pages may still be chosen to be shown by Google if you link to them internally and in sitemaps.

27. With website signals, there’s strength in numbers. @rachellcostello SearchLeeds

28. Mixed signals waste Google’s time, your site’s crawl budget and also delay indexing. @rachellcostello SearchLeeds

29. @rachellcostello SearchLeeds CANONICALIZING TO REDIRECTS CAN INCREASE TIME TAKEN TO INDEX PREFERRED VERSION Canonicalizing to redirects can increase the time taken for Google to decide which version to index, especially if these pages aren’t identical. John Mueller, Google Webmaster Trends Analyst Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-23rd-2018/#link5

30. GOOGLE & WEBSITE SIGNALS To summarise: 1. Google uses the following signals: canonicals, internal linking, parameter handling, backlinks, redirects and sitemaps 2. These signals are strongest when used in combination 3. The purpose of these signals is to avoid content duplication @rachellcostello SearchLeeds

31. HOW TO TEST YOUR WEBSITE’S SIGNALS @rachellcostello SearchLeeds

32. KEY ELEMENTS TO TEST 1. Internal linking 2. Redirects 3. Noindex 4. Sitemap inclusion 5. Disallow @rachellcostello SearchLeeds

33. Check for inconsistencies. Are there any signals that don’t quite match up? @rachellcostello SearchLeeds

34. Testing website signals with DeepCrawl. @rachellcostello SearchLeeds

35. @rachellcostello SearchLeeds Examine all canonicalized pages

36. @rachellcostello SearchLeeds Examine links in to canonical pages

37. @rachellcostello SearchLeeds Examine canonical pages with no links in

38. @rachellcostello SearchLeeds Examine redirecting internally linked URLs

39. @rachellcostello SearchLeeds Examine canonical tags to non-200 URLs

40. @rachellcostello SearchLeeds Examine canonical URL sitemap inclusion

41. @rachellcostello SearchLeeds Examine non-indexable pages being shown

42. @rachellcostello SearchLeeds DISALLOWED URLS MAY SHOW IN SEARCH DUE TO INTERNAL LINKING If you’re seeing disallowed URLs showing up in search results, it may be because of internal linking to these pages. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-april-4th-2017/#link2 Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-november-4th-2016/ GOOGLE MAY CHOOSE A REDIRECTED URL INSTEAD OF THE TARGET Even in the case of a redirect, Google might still choose to index the redirect source instead of the target if it is judged to be the canonical.

43. @rachellcostello SearchLeeds DON'T USE NOINDEX ON CANONICAL PAGES Don’t use a noindex on canonical pages because if Google accidentally picks a noindexed page as the canonical, nothing will show up in search results. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-september-9th-2016/ Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-august-11th-2017/#link1 GOOGLE TAKES CANONICAL TO BE A MISTAKE IF PRESENT WITH NOINDEX If a canonical and noindex are both used, Google judges the rel canonical to be a mistake and the noindex as being used to force a canonical.

44. Nothing appearing in search - the biggest canonicalisation disaster you can get! @rachellcostello SearchLeeds

45. Make sure your site’s signals are aligned and pointing in the same direction. @rachellcostello SearchLeeds

46. TESTING YOUR WEBSITE SIGNALS To summarise: 1. Examine the following areas: internal linking, redirects, noindex, sitemap inclusion, disallow 2. Check each of these areas for inconsistent URL focuses 3. Having aligned signals is the key to getting content indexed by Google @rachellcostello SearchLeeds

47. HOW TO TEST GOOGLE’S URL SELECTION @rachellcostello SearchLeeds

48. @rachellcostello SearchLeeds YOU CAN SEE WHICH CANONICAL GOOGLE HAS CHOSEN IN THE NEW SEARCH CONSOLE In the new Search Console you can select individual URLs which takes you to the info query showing you the canonical that Google has chosen. John Mueller, Google Webmaster Trends Analyst Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-9th-2018/#link6

49. @rachellcostello SearchLeeds To Google Search Console we go!

50. The key reports in Google Search Console for canonicalization analysis... @rachellcostello SearchLeeds

51. @rachellcostello SearchLeeds

52. @rachellcostello SearchLeeds Google thinks these are duplicate pages despite you marking them as canonical.

53. GSC REPORT MEANING Google chose different canonical than user “This page is marked as canonical, but Google thinks another URL makes a better canonical which has been indexed instead.” @rachellcostello SearchLeeds Source: https://support.google.com/webmasters/answer/7440203

55. @rachellcostello SearchLeeds Google thinks these are also duplicate pages.

56. GSC REPORT MEANING Alternate page with proper canonical tag “This page is a duplicate of a page that Google recognizes as canonical. This page correctly points to the canonical page, so there is nothing for you to do.” @rachellcostello SearchLeeds Source: https://support.google.com/webmasters/answer/7440203

58. @rachellcostello SearchLeeds Google thinks another page is a better choice to canonicalize duplicates to.

59. GSC REPORT MEANING Submitted URL not selected as canonical “The URL is one of a set of duplicates without an explicitly marked canonical page. Google did not index this URL. Instead, we indexed the canonical that we selected.” @rachellcostello SearchLeeds Source: https://support.google.com/webmasters/answer/7440203?authuser=0#submitted_url_not_selected_as_c anonical

61. @rachellcostello SearchLeeds Google thinks these are part of duplicate page sets which don’t have a canonical.

62. GSC REPORT MEANING Duplicate page without canonical tag “This page has duplicates, none of which is marked canonical. We think this page is not the canonical one. You should explicitly mark the canonical for this page.” @rachellcostello SearchLeeds Source: https://support.google.com/webmasters/answer/7440203

63. @rachellcostello SearchLeeds Go to Google to see the results...

64. CHECKING RESULTS IN GOOGLE The best way to see Google’s URL selection for yourself is through the info query and cache. @rachellcostello SearchLeeds

65. @rachellcostello SearchLeeds Info: query Cache

66. @rachellcostello SearchLeeds USE THE INFO QUERY IN GOOGLE TO FIND CANONICAL URLS Search for a URL with an “info:” query to see if a different URL has been indexed instead. Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-april-4th-2017/#link4 Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-9th-2018/#link8 INCORRECT GOOGLE CACHING SUGGESTS A DIFFERENT CANONICAL HAS BEEN CHOSEN If the cached content from a different page is displayed, this could be because Google has determined duplication and has canonicalised one.

67. TESTING GOOGLE’S SELECTION To summarise: 1. Despite your website signal implementation, Google has the final say in what is indexed 2. Check Google Search Console canonicalization reports 3. See the live results in the SERPs by checking what appears for “info:” queries and caching @rachellcostello SearchLeeds

68. @rachellcostello SearchLeeds All you can do is give aligned signals to make Google’s job easier.

69. @rachellcostello SearchLeeds The rest is in Google’s hands.

70. TO SUM UP Understand Google’s URL selection algorithm. - Canonicalization, internal linking, parameter handling, sitemaps, redirects, backlinks Monitor your site for conflicting signals. - Internal linking, redirects, noindex, sitemaps, disallow Analyse Google’s interpretation of signals. - Google Search Console, SERP “info:” queries and caching @rachellcostello SearchLeeds

71. THANK YOU! Any questions? Send me a tweet: Rachel Costello Technical SEO, DeepCrawl @rachellcostello SearchLeeds deepcrawl.com @DeepCrawl

SearchLeeds 2018 - Rachel Costello - DeepCrawl - Stop confusing search engines with conflicting website signals

Branded3

SearchLeeds 2018 - Rachel Costello - DeepCrawl - Stop confusing search engines with conflicting website signals