Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

@dawnieando #BigDigitalADL
Generational
Cruft
&
Technical
Debt
in

SEOGONE
IS

NEVER
GONE Dawn
Anderson
@
dawnieando

A New Beginning
§ “A
new
website
will
solve
ALL
our

problems”
§ “Let’s
start
again”
§ “We’ll
just
migrate…
and
redirect

everything”

404 Page Not Found
§ “Of
course,
we
won’t

redirect
everything…”
§ “Not
everything
will

be
worth
redirecting”

410 Gone
§ “Some,
we’ll
just
kill

off
with
a
410…”
§ “Then
the
URLs
will

be
gone”

But in Reality…it Still Exists

Because…Web Crawler System’s History Logs
SEARCH
ENGINES
HAVE
A

BIG
MEMORY
AND
A
LOT
OF

STORAGE

Web Crawler System History Logs

Web Crawler System
GOOGLE
NEVER
FORGETS
The
history
logs
play
a

role
in
deciding
when

every
URL
that
was
EVER

discovered
gets
visited

again

History Log Records Include:
• URL
fingerprint
• Timestamp
(last
crawl
or
download

attempt)
• Crawl
status
(success
or
error)
(Response

code)
• Content
checksum
(binary
code)
• Source
ID
(accessed
from
cache
or

downloaded)
• Segment
identifier
(Crawl
segment
assigned

to??)
• Page
importance
(a
measure
of
importance

assigned
to
the
URL)
May
be

calculated
by

identifying

historical

importance

scores
based
on

past
X
number
of

crawls

Gone Is Never Gone
“We
knew
there
was
content

there
at
some
point
so
we

just
swing
by
every
now
and

then
to
see
if
anything
came

back”
(John
Mueller,
2016)

Generational Cruft … NOT ‘Crufts’

The Generational ’Snail Trail’
• Old
XML
sitemaps
• Redirects
drop
away
on
old
site

.htaccess
• DNS
issues
• People
link
to
old
site
but
wrong

protocol
• Old
sites
not
verified
in
GSC
• Not
all
protocols
redirecting
Leaving
it’s

slithery

footprint

The Generational ’Snail Trail’
• All
eating
away
at

Googlebot’s attention
on

your
server’s
IP
WATCH
OUT
FOR
THE
SNAIL
TRAIL
&

GENERATIONAL
CRUFT

The Slow Page Evolution of Near Duplicates
In
a
study
over
11
weeks
Denis
Fetterly and

Mark
Najork found
that
near-‐duplicate
pages

rarely
change
and
that
they
are
still
near-‐
duplicates
of
each
other
10
weeks
later.

Therefore
once
identified
their
download

priority
may
be
reduced
so
that
resources
may

be
used
more
efficiently
/
productively

elsewhere
(Fetterly &
Najork,
2003)
Fetterly &

Najork,

2003

‘Transitive’??
Transitive
-‐ A
==
B
+
B
==
C
then
A
==
C
THEORY:
Maybe
for
some
types
of

content
more
than
others
– e.g.

ecommerce/directories
but
not
news
THEORY
ALERT
!!!!!!!

DUSTBUSTER & DUST CRAWLING RULES
DO
NOT

CRAWL
IN

THE
DUST
BUILDS

‘HINTS’
ON

WHAT
NOT

TO
CRAWL

‘Sampling’in Crawling for Efficiency
‘SMALL
TEST
VISITS
TO
A
SITE
TO

UNDERSTAND
WHETHER
IT
IS
WORTH

CRAWLING’

Every Site Will Have Its Own Crawling Rules
UNSURE
AS
TO
WHETHER

THIS
IS
BEING
USED
DUSTBUSTER

CRAWLING
RULES

Popular CMS ’Rule Patterns’ (URL Parameters)
ALL
WILL
HAVE
COMMON

CANONICALIZATION
PATTERNS
WHICH

CAN
BE
LEARNED

CRAWLING RULES BUILT OVER TIME
Crawl
Frequency
Patterns
No
two
sites
will
have
the
same
crawl

schedules
or
rules
built
Moving
from
one
CMS
to
another
may

mean
that
different
parameters
are

created.

New
parameters
=
new
rules

Every Version of Your Past Ecommerce Sites
“Exponentially

multiplicative

URLs”
Had
potential
to
spew…
at
some
point…

URLs Take Their Place in Crawling Queues
The
Queue
Gets
Long
&

Congested

YOU INHERITED SEO TECHNICAL DEBT
• Previous
content
/
link
manual
actions
• Previous
algorithmic
suppressions
• Past
infinite
loops
• “We’ll
SEO
it
after
launch”
• “SEO
is
dead…
so
we
won’t
optimise”
• Dodgy
URL
parameters
• Misconfigured
URL
parameters
• Old
URL
crawling
‘rules
/
hints’

TECHNICAL DEBT

IT WASN’T ME – PASSING THE BUCK

… with Interest
AT
SOME
POINT
IT

MUST
BE
REPAID

GENERATIONAL CRUFT
EVERY
SINGLE
TIME
YOU
MIGRATE,
CHANGE
DESIGN,
REDIRECT,
REINVENT
A
SITE
/
URL
A
CLEAN
START
REDIRECTIONS
ANOTHER
STRUCTURE
FIRST
SITE

STRUCTURE
NEW
CRAWLING
‘RULES’

BUILT
CRAWLING

‘RULES’
BUILT
EVERYTHING

IS
‘200
OK’
MORE
URLs
MIXED
RESPONSE
CODES
REDIRECTIONS
‘FUZZINESS’
IS
EMERGING
NEW
CRAWLING
‘RULES’
BUILT
MORE
URLs
REDIRECT
CHAINS
&
MIXED

RESPONSE
CODES
NEW
SEO’s
DON’T

KNOW
THE
‘HISTORY’
TARGET
URLs
NOW
‘VERY
FUZZY’

Aged ‘Patchwork Quilt’Sites
A
LITTLE
BIT
OF
THIS
CMS
AND
A

LITTLE
BIT
OF
THAT
CMS
MANY
HISTORICAL
PARAMETERS

CREATED

’Fuzzy’ URL Targets with Each Site Generation
EVERYTHING
GETS

A
BIT
BLURRED
‘Which
is
the
target
URL

again?

Time Seems To Fly… The Older You Get
Your
new
site
URL
is
just

one
of
very
many
historical

URLs
on
your
IP
to
be

visited
periodically
A
tiny
fish
in
a
very

big
URL
pond
queue

The Great 302s Pass PageRank Debate

SOLUTION - THE BELOVED CANONICAL
§ 30X
redirects
§ Canonical
tag
§ Href lang
§ HTTPS
protocol
§ Global
canonicalization

rules
In
’ALL’
its
forms

Chocolate Boxes Research

Advanced Technical SEO?
50% of SEOs surveyed
considered;;
“CANONICALIZATION
IS ADVANCED
TECHNICAL SEO”

Oh Yeah – Canonicalization is Easy
76% of SEOs surveyed
considered;;
“CANONICALIZATION
IS AN EASY CONCEPT
TO UNDERSTAND”

REL NEXT REL PREV is NOT Canonicalization
47% of SEOs
categorizing themselves
as ‘TECHNICAL SEO’s
considered;;
“REL=NEXT / REL =
PREV” IS A FORM OF
CANONICALIZATION

On Redirections as Canonicalization Forms
Lots were unsure about;;
“301s and 302s are
BOTH forms of
canonicalization”

On Href Lang as Canonicalization
Only 64% of ’Technical
SEOs’ thought HRef
Lang was a form of
Canonicalization
IT IS

URL Parameter Handling is Your Friend
Help
Google
Build
‘Crawling

Rules’
for
your
site
rather

than
wasting
time
on

‘sampling’
and
giving
a
bad

impression
GIVE
HELP
AND

GUIDANCE
WITH
THE

CRAWL
RULE
AND

HINT
BUILDING

SOLUTION - Understand URL Parameters
ACTIVE
PARAMETERS
==
CHANGE
THE
CONTENT
ON

YOUR
PAGE
(e.g.
sort,
filter,
translate,
paginate,
specify)
PASSIVE
PARAMETERS
==
DO
NOT
CHANGE
THE

CONTENT
ON
YOUR
PAGE
(Often
used
for
tracking)

(ALIAS:
REPRESENTATIVE)

ACTIVE Parameters (CHANGE CONTENT)
SORT
==
Sorts
dynamic
items
and
reorders
in
descending
/
ascending
price
/

popularity
/
added
NARROWING
==
Filters
dynamically
added
items
down
to
include
only

features
&
attributes
in
a
chosen
consideration
set
SPECIFYING
==
Identifies
a
particular
dynamically
variable
populated

content
set
within
a
site
section
(e.g.
store=women)
TRANSLATING
==
Indicates
a
language
driven
translation
URL
(e.g.
lang=fr)
PAGINATING
==
Indicates
a
paginated
display
of
long
content
(e.g.
page=2)

Understand How URLs with
Multiple Parameters Are Handled
The
most
restrictive
parameter
blocked
overrules

lesser
restrictions

Examples of Multiple Parameter Handling
KNOW
THE

RULES
http://www.example.com?shopping-‐category=DVD-‐movies&sort-‐
by=production-‐year&sort-‐order=asc WILL
BE
CRAWLED
http://www.example.com?shopping-‐category=shoes&sort-‐by=size&sort-‐
order=asc WILL
NOT
BE
CRAWLED
(production-‐year
blocks)

Help Googlebot Get Round its Shopping List
OPEN
MORE
CHECKOUTS
WIDEN
THE
AISLES
MAKE
THINGS
EASY
TO
FIND
DON’T
CONFUSE

GOOGLEBOT
HELP
FILL
THE
TROLLEY

QUICKLY
SPEED,
SPEED,
SPEED

SOLUTION - XML SitemapsAre Your
Friend… (Strong Foundations)
They
help
to

pass

‘importance’

signals
within

a
site
But…
never

leave
them
to

just

autogenerate
without

periodically

checking
‘The

foundations’

underneath
a

site

Validate & Retain in GSC ALL Past Domains &
Past Site Versions (Protocols (HTTPS / HTTP)
THERE
MAY
STILL
BE
UNDETECTED
ACTIVITY
GOING
ON
THERE

Server Log FileAnalysis is Your Friend…
You’ll
be
surprised
by
what
you
find
Find
out
what
Googlebot is

visiting
and
when
(how

often)
and
whether
it

should
be
visiting
it
at
all

SOLUTION - Save & Grow The URLs
Not
EVERYTHING
is

worthy
of
its
own
URL
VARIANTS
STEMMINGS
PLURALS
RANDOM
TAGS
LONG,
LONG,
LONG

TAIL
PARAMETERS

SOLUTION - Save & Grow The URLs
A
URL
is
like
a

fine
wine
Maturing
over

time

Pass Strong Clues - Highly Relevant New Structure
STRONG
SEMANTICS

Wiki Page Redirects
https://dbpedia.org/sparql
Wikipedia

Redirects
thesaurus.com
OR
A
GOOD
OLD
FASHIONED
THESAURUS

Increase ‘Importance’ quickly of target URLs
• Internal
link
optimization
• Canonicalise to
(if
relevant)
• Strengthen
up
importance
signals
• Inclusion
in
front
facing
and
XML

sitemaps
• Improve
the
content
&
keep
it

updated
• 301
redirect
to
(if
relevant

redundant
content)

Reduce ‘Importance’ quickly of old URLs
• Internal
link
unoptimization
• 410
• Dig
out
URLs
with
links
to
them
• Orphan
URLs
• Canonicals
to
HTTPs
• Exclusion
from
XML
sitemaps
(even

old
ones
on
the
server)
• Strip
out
content

304 IF MODIFIED HEADERS
ONLY
DOWNLOAD
IF
THE

CONTENT
CHECKSUM
HAS

CHANGED

CHOP BACK CHAINS

SOLUTION – Be Careful About Creating
New Dynamic Parameters
QUEUEING…
AGAIN
Waiting
for
good
URLs
to
be

visited…
AGAIN

REVISIT PAST .HTACCESS FILES
Can
you
rewrite
the
rules
to
be
more

efficient
or
cut
out
some
old
rules
still

firing
unnecessarily?

SOLUTION - NEVER TRY TO ‘OUTRUN’
GOOGLEBOT

You Have a Shiny New Site… So What?
You
may
still
have…
GENERATIONAL
CRUFT
&
TECHNICAL

DEBT
TO
PAY
OFF
GONE
IS
NEVER
GONE

REMEMBER
”Gone
is

Never
Gone”
“Google
Has
A

Big
Memory”
Dawn
Anderson
@
dawnieando
THANK
YOU

Sources & References
• https://patentimages.storage.googleapis.com/US8042112B1/US08042112-‐
20111018-‐D00000.png
• Randall,
K.H.,
Google
Inc.,
2010. Scheduler
for
search
engine
crawler.
U.S.
Patent

7,725,452.

Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017

Semelhante a Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017 (20)

Mais de Dawn Anderson MSc DigM

Mais de Dawn Anderson MSc DigM (20)

Último

Último (20)

Technical SEO - Gone is Never Gone - Fixing Generational Cruft and Technical Debt in SEO - Big Digital Adelaide 2017