Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings

Bartosz Góralewicz
ELEPHATE
Can Google properly crawl and index JavaScript?
SEO Experiments - Results and findings
@bart_goralewicz
https://www.linkedin.com/in/goralewicz/

Why is JavaScript SEO such a complex topic?

DATANO
Why is JavaScript SEO such a complex topic?

DATANO
Making JavaScript SEO easy

What is the problem with JavaScript?
HTML
JS

What is the problem with JavaScript?
HTML
HTML

*more about it later
not as “forgiving” as HTML*

Reverse
engineering
algorithms is
no longer an
option.
Times have changed. Today, as
long as you're not blocking
Googlebot from crawling your
JavaScript or CSS files, we are
generally able to render and
understand your web pages like
modern browsers. To reflect this
improvement, we recently updated
our technical Webmaster
Guidelines to recommend against
disallowing Googlebot from
crawling your site's CSS or JS files.
ele.ph/ajax

Hulu.com
JavaScript & SEO backfire

Google and JavaScript – difficult relationship

The only solution…
Java Script Experiment

Inspect Element
vs.
View Source

First JavaScript rendering issues

JavaScript links were
not parsed by Googlebot
Google
kept us
waiting…

/html/test/ (duh!)
/plain-inline/test/
/jquery-inline/test/
/jquery-ajax/test/
/vue/test/
/react-inline/test/
And… that’s it! /test/

Let’s start with JS frameworks created by

Checklist
Check if ”framework/test/” URL was crawled
Search for unique content from framework’s page
Is URL indexed by Google?
Fetch and render test (GSC)

Search for unique content from framework’s page

Check if ”framework/test/” URL was crawled

V2 Inline
GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
Inline Inline

GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
External Ajax callInline

GSC
fetch
URL
Indexed?
Content
Indexed?
Test
URL
Inline External

Inline vs. External JS makes a huge difference
Not all JavaScript frameworks are crawled
and indexed in the same way
Mindblowing findings
JS generated links aren’t always crawled

ele.ph/angularU
Also…
Regardless of what your
developers say, Angular JS 2
always has to be server
rendered…
according to
Angular JS 2 community.

If you search for any competitive
keyword terms, it’s always going
to be server rendered sites. And
the reason is because although
Google does index client-side
rendered HTML, it’s not perfect
yet and other search engines
don’t do it as well. So if you care
about SEO, you still need to have
server-rendered content.
Jeff Whelpley
Angular U conference, June 22-25, 2015, Hyatt Regency, San Francisco Airport
“Angular 2 Server Rendering”
ele.ph/angularU

?How about Search Engines other than

63,4%
22,8%
11,7%
1,3% 0,8%
Google Bing Yahoo! Ask AOL
Search Engine market share
Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/

83,5%
11,1%
3,8% 0,8% 0,6%
Google Bing Yahoo! MSN DuckDuckGo
Search Engine market share
Source: https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/

Tell me
why I
can’t…
Clear your
mind of
questions
*not the actual conversation

http://jsseo.expert/plain-external/test/
http://jsseo.expert/react-external/test/
http://jsseo.expert/jquery-external/test/
we found this URL from a link ca 3 weeks ago, but we haven't crawled it
we picked this one up ~19 days ago, and similar to above,
our systems just didn't feel like crawling it yet
picked up ~21 days ago, also not crawled yet.

Thanks,
John!
You’re the
best,
Bartosz!*
*he didn’t say that
John Mueller’s feedback got me thinking…

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit
amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada
fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce
posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros
quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin
pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar
ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi
tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra
imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy
pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus
malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et
netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor
congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce
est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem
ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet
commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac
turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere,
magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus.
Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet,
consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros
quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin
pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar
ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi
tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra
imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Proin pharetra nonummy
pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus
malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce est. Vivamus a tellus. Pellentesque habitant morbi tristique senectus et
netus et malesuada fames ac turpis egestas. Proin pharetra nonummy pede. Mauris et orci. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas porttitor
congue massa. Fusce posuere, magna sed pulvinar ultricies, purus lectus malesuada libero, sit amet commodo magna eros quis urna. Nunc viverra imperdiet enim. Fusce
JavaScript vs. crawler budget
Some JavaScript frameworks are indexable,
but there is a fine print…

Do you already know where I am going with this?

Crawler budget experiment (early beta)

We repeated this experiment a few times and
the results were always the same.

JavaScript,
even when indexable,
can kill your crawler budget…

JS is less
effective than
HTML
JavaScript is not
bad for SEO, if
done right*
*he said exactly that
Ilya Grigorik

Understanding technology behind

ele.ph/chrome41
Download Chrome 41 here:

Using Chrome 41
for debugging JS indexing

http://jsseo.expert/angular2/
in Chrome 41

http://jsseo.expert/angular2/
in up-to-date Chrome

JavaScript SEO isn’t only affecting technical

At the same times those links count just as HTML ones.
Most link indexes
(e.g. Majestic, MOZ etc.)
can’t crawl JavaScript generated backlinks.

Also…
...there is more JavaScript generated links out there
than I would imagine
(I’ll tell you more about it in a second)

Problems?
Manual penalties
Tracking mentions
Google Penguin
Link audits
Link building

Crawling JavaScript is expensive…

To execute JS for every page at
our scale [would] require
10,000-15,000 servers, and
we believe our customers are
not ready to pay for that yet”
Dmitry Gerasimenko, CEO of
Ahrefs
~ July 2017

Ahrefs guys changed their mind after
reading my article…
ele.ph/js-ahrefs

…We accepted the challenge
and crawled this same
website with our JS crawler,
which rendered all of the
pages that were there.
ele.ph/js-ahrefs

ele.ph/js-ahrefs
So earlier this
year Dmitry (our
Founder & CEO) decided that
Ahrefs has to be the first
among its rivals to crawl
JavaScript links.
Which we achieved with
ease last month.

787622915746127654196745167546245246157964127432546
12794214512161504918805916076096067461350967619097
645684795464436557433443675698347856654756434587
6554364655437876954878796865645654456453212543321
5645466556762615655244675988799980879876424223314
003132425654342698754736755645654465474587698374
7876229 4322546
12794214512161504918805916076096067461350967619097
645684795464436557433443675698347856654756434587
6554364655437876954878796865645654456453212543321
5645466556762615655244675988799980879876424223314
003132425654342698754736755645654465474587698374
787622915746127654196745167546245246157964127432546
So how many JS links are out there?

Is there a perfect JavaScript SEO scenario?

The setup for that is called
Isomorphic JavaScript on
React & Universal on
Angular. That's what I'd
recommend doing in general,
if you can.
Isomorphic JavaScript
ele.ph/Iso-John

user
server
middleware
Prerendering
crawlers
JS
HTML
JS
?
?

JS
Only Google and Ask
can handle JavaScript
Key takeways

JS
Only Google and Ask
Not all JavaScript
frameworks are indexable
Key takeways

Not all JavaScript
Indexing JavaScript
is only half the battle
Key takeways

Indexing JavaScript
JavaScript even when
rendered and indexed,
can kill crawler budget
Key takeways

Use Chrome 41
to debug JS
Key takeways

Use Chrome 41
to debug JS
If too late,
use prerendering
Key takeways

If too late,
use prerendering
Key takeways
Isomorphic JS
=
perfect scenario

Not all JavaScript
Indexing JavaScript
Use Chrome 41
to debug JS
JS
Only Google and Ask
If too late,
use prerendering
Key takeways
Isomorphic JS
=
perfect scenario

@bart_goralewicz
/in/goralewicz/
Thanks! :*

Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (19)

Semelhante a Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings

Semelhante a Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings (20)

Mais de Onely

Mais de Onely (8)

Último

Último (20)

Can Google properly crawl and index JavaScript? SEO Experiments - Results and findings