1.
HOW TO AUTOMATE YOUR
KEYWORD RESEARCH
Paul Shapiro – Organic Search Director at Catalyst
SearchLove Boston – May 4th, 2016

2.
@fighto 2
Without Automation

3.
@fighto 3
With Automation

4.
@fighto
WHO AM I?
4

5.
@fighto 5
Tool Data Sources Visualization
+ +

6.
@fighto
WHAT YOU WILL LEARN
• The problems with the standard keyword research process
• How you can save time on keyword research with automation
• How to automate keyword research with a powerful, but easy-to-use
tool
• How APIs work and how to leverage them
• What data sources are useful for keyword research automation
• How to use data visualization to make keyword research decisions
6

7.
MANUAL KEYWORD
RESEARCH IS
OFTEN BLAND &
TIME CONSUMING
7

8.
@fighto 8

9.
@fighto 9
Most Keyword Research Looks Like This

10.
@fighto 10
That’s just a
Google Keyword
Planner Export

11.
@fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
5
8
12
21 21
6
10
18
22
28
0
5
10
15
20
25
30
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages) Large (250-499 pages) Extra Large (>500 pages)
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
11

12.
@fighto 12
AIN’T NOBODY GOT TIME FOR MANUAL
KEYWORD RESEARCH!

13.
@fighto
SIZE OF THE DATA SET
13
vs.

14.
@fighto 14
We can do better!

15.
WE CAN SAVE TIME WITH
AUTOMATION

16.
@fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
5
8
12
21 21
6
10
18
22
28
0
5
10
15
20
25
30
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages) Large (250-499 pages) Extra Large (>500 pages)
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
16

17.
@fighto 17
If we do keyword research
once per month, we will save
around 24 to 45 full days
over a 5-year period using
automation.
http://xkcd.com/1205/

18.
WE CAN DO BETTER THAN AN
EXCEL SPREADSHEET

19.
@fighto
THAT’S MORE LIKE IT!
19
h/t @victorpan

20.
@fighto
HOW TO USE FOR KEYWORD SELECTION / TARGETING
20
This top-right quadrant contains
keywords with:
• Low competition
• Good growth
Larger bubbles show higher
search volumes.
You can alternatively use
current rank on the x-axis to
signal organic market share like
a traditional growth-share
matrix.

21.
@fighto 21
That’s much better!
actionability.

22.
MY KEYWORD RESEARCH
AUTOMATION PROCESS

23.
@fighto 23
https://www.knime.org
It starts with a tool…

24.
@fighto
WHY KNIME?
• Singular tool for data collection,
manipulation, analysis, and reporting
• Easy to use and no development skills
necessary
• If you are a developer, it provides faster
prototyping
• Large library of built-in “nodes”
• Free/Open Source
• Run on Windows/Mac/Linux
• Infinitely expandable
24

25.
THE BASICS OF KNIME

26.
@fighto
WHAT’S A NODE?
• Nodes are prebuilt, drag and drop modules designed perform a singular task
• Nodes are strung together like a chain to accomplish larger, more complex tasks
• Nodes can be grouped together into “meta-nodes”, which can be configured in unison
26

27.
@fighto
HOW DO YOU ADD NODES & HOW DO THEY CONNECT?
How do you add nodes to your “workflow”?
How do you string nodes together?
27

28.
@fighto
HOW DO YOU CONFIGURE & RUN NODES?
Configuring Nodes
Running Workflows
28
OR

29.
APIs 101
29

30.
@fighto
EASY & APPLICABLE API: GOOGLE AUTOSUGGEST
http://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q=ninja%20turtles
30
Simple API using “GET Request”
This is a variable
that changes

31.
@fighto 31
Output = XML

32.
A KNIME-API EXAMPLE
32

33.
@fighto
STEP 1: LIST OF KEYWORDS
33

34.
@fighto
STEP 2: ENCODE SPACES IN KEYWORD PHRASES
34
Space
Character

35.
@fighto
STEP 3: CONCATENATE API URL + KEYWORDS
35

36.
@fighto
STEP 4: DOWNLOAD API OUTPUT
36

37.
@fighto
STEP 6: LOOP THROUGH EACH KEYWORD
37

38.
@fighto
STEP 7: PARSE THE XML OUTPUT FORM THE API
38

39.
@fighto 39
Auto-Complete
Suggestions

40.
@fighto 40

41.
@fighto
FYI: I’M NOT A KNIME SALESMAN
41

42.
DATA INPUTS / SOURCES

43.
@fighto 43
# of primary data sources to aggregate

44.
@fighto 44
http://searchwilderness.com/gwmt-data-python/
Google Search Analytics – Historical MySQL
Database via Python Script

45.
@fighto 45
AdWords - Search Query Report API or CSV

46.
@fighto 46
SEMRush – API (if <100,000)
or Domain vs. Domain:
Competitor Keywords (Advanced Mode)
https://www.semrush.com/info/domain_vs_domain

47.
@fighto 47
GrepWords API – For Google Keyword Planner
related query suggestions & search volumes
http://grepwords.com/

48.
@fighto 48
Google Autosuggest API
or Emulation with Selenium

49.
@fighto 49
Rank Checking/Full SERP API - GetSTAT
https://getstat.com/

50.
@fighto 50
MarketMuse – Semantic Keyword
Recommendations
https://www.marketmuse.com/
Alternative: http://pshapi.ro/SemanticKWR

51.
@fighto 51
Moz API + Moz Nodes
https://tech.knime.org/book/mmi-labs-nodes
https://moz.com/products/mozscape/access

52.
@fighto 52
Google Trends – Unofficial API
https://github.com/GeneralMills/pytrends
Script soon to follow: http://pshapi.ro/slopetrends

53.
@fighto 53
More data sources:
http://pshapi.ro/kwmetrics

54.
THE DATA-TO-KNIME
PROCESS
54

55.
@fighto
Filtering +
Data
Manipula-
tions
ONE BIG KEYWORD LIST
Seed Keywords
- List
- GWMT
- SEMRush Comp. KWs
- SQR Keywords
Keyword Planner
Suggestions (via
GrepWords)
Google Autocomplete
Semantic Keyword
Recommendations
(via MarketMuse)
Google Autocomplete
SEMRush
Domain vs. Domain
Keywords
Google Autocomplete
55

56.
@fighto
STRING ‘EM ALL TOGETHER AND THEN…
56

57.
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
Search Volume
Keyword Trends
57

58.
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
Keyword Trends
58

59.
@fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
• Use 2 Years of Google Trends data to calculate slope and
determine growing/declining keywordsKeyword Trends
59

60.
@fighto 60

61.
@fighto 61
Reusable
5-Minutes

62.
@fighto 62
You can use this data to
filter keywords and reduce
massive list... OR, you can
leverage data visualization!

63.
VISUALIZING KEYWORD
RESEARCH

64.
@fighto 64
We have a
few options…

65.
@fighto
OPTION: EXCEL + BUBBLE CHART
65

66.
@fighto
OPTION: TABLEAU
66

67.
@fighto
OPTION: PYTHON + BOKEH CHARTS IN KNIME
67

68.
@fighto
THEY’RE INTERACTIVE…
68

69.
@fighto 69
Sample Python Code:
http://pshapi.ro/SLBokeh

70.
@fighto
SO WHAT…
• Free up time spend doing mundane, repetitive work and spend more doing impactful, strategic work
• Free up staffing resources and utilize them in other ways
• Extract and analyze a larger pool of data for greater data pool
70
From this… To this…

71.
@fighto
SUMMARY
• You should rethink how your currently do keyword research.
• Keyword research is often bland, time consuming, and not as actionable as it can be.
• KNIME provides an easy-to-use interface for automating your keyword research
• APIs are easier to use than they sound; Use them
• There are great data sources you can leverage via APIs/built-in KNIME nodes
• Visualize your keywords using a growth-share matrix like-graph to make keyword usage
decisions
• Emojis are awesome
71

72.
@fighto
RESOURCES: DATA SOURCES > KNIME
72
Data Source KNIME Node(s)
Google Search Analytics Database Table Connector/Reader
AdWords Search Query Report XLS Reader
SEMRush Domain vs. Domain (or API if less
than 100,000 keyword)
XLS/CSV Reader or GET Resource & Read
REST Representation
Google Keyword Planner/GrepWords GET Resource & Read REST Representation
Google Autosuggest
GET Resource & Read REST Representation
(or Selenium nodes)
Rank Checking (via GetSTAT) GET Resource & Read REST Representation
Semantic Keywords/MarketMuse POST Resource
MOZ Moz API Connector/URL Metrics nodes
Google Trends TBD – Either CSV or Python Script nodes

73.
@fighto
ADDITIONAL RESOURCES
• KNIME Version to install: “KNIME Analytics Platform + all free extensions”
• To install KNIME Add-ons, Choose Help->Install New Software
• Moz nodes
• Should come with Nightly Builds. If not can add http://update.knime.org/community-
contributions/3.1 as a repository.
• Add-on Selenium nodes
• http://seleniumnodes.com/
• Search Engine Land article addressing additional data inputs: http://pshapi.ro/kwmetrics
• Information about Growth-share matrices:
https://en.wikipedia.org/wiki/Growth%E2%80%93share_matrix
• Sanple Python + Bokeh script for an interactive keyword research visualization in KNIME:
http://pshapi.ro/SLBokeh
73

74.
@fighto
Questions?
74

75.
THANK YOU!
@fighto
http://searchwilderness.com
http://catalystdigital.com