Ranking Factors Going Causal by Micah Fisher-Kirshner at TechSEO Boost in Boston 2019
•Download as PPTX, PDF•
0 likes•748 views
A presentation into why ranking factor studies should take from the scientific field the statistical tools and methods, how we should analyze or produce these studies, and why there is an opportunity today in our field. Note: Watch the video via SiteCatalyst to get the full talk track.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Ranking Factors Going Causal by Micah Fisher-Kirshner at TechSEO Boost in Boston 2019
- 1. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Regressions, Machine Learning, and Neural Networks Ranking Factors Going Causal
- 2. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost The challenge is in learning the mechanics
- 3. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost And nitpicking…
- 4. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Micah Fisher-Kirshner Director of SEO & Content @ Turn/River Capital President & Founder @ BayAreaSearch.org A modern nerd
- 5. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Understanding the Victory Conditions
- 6. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost y = 𝜷x
- 7. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Correlation != Causation highly significant
- 8. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost
- 9. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Regressions y = 𝜷0 + 𝜷1x1 + 𝜷2x2 + 𝜷3x3 + 𝜷4x4 + ... + 𝜷5x1x2 + 𝜷6x1x3 + 𝜷7x2x3 + ... + 𝜷8x1x2x4 + ... + 𝜷nxn + 𝜺
- 10. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost y = Keyword’s Position y = 𝜷0 + 𝜷1(title tag length) + 𝜷2(order in title tag) + 𝜷3(page load time) + 𝜷4(total in-body internal link value) + ... + 𝜷5x1x2 + 𝜷6x1x3 + 𝜷7x2x3 + ... + 𝜷8x1x2x4 + ... + 𝜷nxn + 𝜺
- 11. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost y = Keyword’s Position y = 𝜷0 + 𝜷1(title tag length) + 𝜷2(order in title tag) + 𝜷3x3 + 𝜷4x4 + ... + 𝜷5(title tag length * order in title tag) + 𝜷6x1x3 + 𝜷7x2x3 + ... + 𝜷8x1x2x4 + ... + 𝜷nxn + 𝜺
- 12. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost y = Keyword’s Position y = 𝜷0 + 0.001x1 + 0.002x2 + -0.003x3 + 0.05x4 + ... + 0.3x1x2 + -0.03x1x3 + -0.07x2x3 + ... + 0.1x1x2x4 + ... + 𝜷nxn + 𝜺
- 13. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost
- 14. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost
- 15. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost 1. Solves complex issues 2. Finds hidden connections 3. Is black box in nature Neural Networks title tag length title tag order page load time
- 16. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Plotting the Victorious Win
- 17. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost
- 18. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Data Collection • Correct & Consistent Mobile or desktop, quality, etc. • Removals Low correlations and limited data • Time Series Weekly changes
- 19. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Interactions • Positive Combinations Authority, branding, etc. • Negative Combinations Panda, spam, etc.
- 20. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Interpretations • Reasonable Direct traffic, Google AdWords, etc. • Outliers Wikipedia, Amazon, etc.
- 21. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Categorizations & Quality Checks • Grouping Search volume & ranking positions • Subsets Keyword intent & heteroskedasticity
- 22. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Visual Outputs • Graphs Correlation goes on scatter plots • Error Ranges Uncertainty goes on whisker plots • Regression Results Standardized data formatting
- 23. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Sanity Checks • Data Separation Cross-validation • Peer Review Do others see the same thing • Predictive Models Has similar accuracy going forward
- 24. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Taking the Time & Effort for Victory
- 25. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Total: ~$20,000 for the year • URL Profiler (scaling): $300 • SEMrush (rankings): $5,000 • link.fish (content): $2,000 • Majestic (external links): $5,000 • Ahrefs (internal links): $5,000 Cost
- 26. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Why • Algorithm Updates Knowing content or link based • Prioritization Changes and how to react • Linkbait Nothing grates engineers more
- 27. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost y = 𝜷0 + 𝜷1x1 + 𝜷2x2 + 𝜷3x3 + 𝜷4x4 + ... + 𝜷5x1x2 + 𝜷6x1x3 + 𝜷7x2x3 + ... + 𝜷8x1x2x4 + ... + 𝜷nxn + 𝜺 What I Planned…
- 28. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost What I Planned… (High Level) y = 𝜷0 + 𝜮𝜷2(CONTENT) + 𝜮𝜷3(LINKS) + 𝜮𝜷4(DOMAIN) + 𝜮𝜷5(STRUCTURE) + 𝜮𝜷6(UX) + 𝜺
- 29. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost What I Planned… (Content) y = ... + 𝜷2a(exact phrase title tag) + 𝜷2b(title tag length) + 𝜷2c(order in title tag) + 𝜷2d(brand name used) + 𝜷2e(title tag relevance) + 𝜷2f(exact phrase h1) + 𝜷2g(exact phrase largest font) + 𝜷2h(h1 relevance) + 𝜷2i(largest font relevance) + 𝜷2j(exact phrase body copy) + 𝜷2k(BM25 score) + 𝜷2l(Flesch-Kincaid readability) + 𝜷2m(exact phrase first 100 words) + 𝜷2n(exact phrase URL) + 𝜷2o(URL relevance) + 𝜷2p(exact phrase title tag * order in title tag) + 𝜷2q(exact phrase title tag * largest font * URL) + 𝜷2r(exact phrase title tag * BM25 score) + ... + 𝜺
- 30. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost So… • Internal Links Checker API Ahrefs: I need you! • SEO Testing Platform Distilled ODN: Need a lot of traffic • Build-a-Google Custom search: Need engineers
- 31. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Enjoying the Taste of Victory
- 32. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Demand Better • Use the scientific fields’ toolsets • QA your data before analyzing • Be reasonable and avoid grandiose claims • Analyze where one can dive deeper • Use plots, not charts • Get the study reviewed before publishing • VICTORY!
- 33. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Citations & Personal Board Gaming Tips • https://scikit-learn.org/stable/tutorial/ machine_learning_map/index.html • https://e360.yale.edu/features/flooding-hot- spots-why-seas-are-rising-faster-on-the- u.s.-east-coast • https://www.scientificamerican.com/article/ graphics-that-seem-clear-can-easily-be- misread/ • https://sdm.lbl.gov/snta/2018/SNTA18_KW ON.pdf • https://www.mdpi.com/journal/futureinternet /special_issues/SEO • Axis & Allies Invade Spain with the United Kingdom • Star Trek Ascendancy Monitor Romulan research continually • Tridimensional Chess Look always at the board top-down • Suburbia Build income to max then go reputation • Galaxy Trucker Focus on weapons and engines over exposure • Chaos of the Old World Don’t let Khorne and Nurgle team up
- 34. Micah Fisher-Kirshner | @micahfk | #TechSEOBoost Huge thanks to Christian, our Data Scientist – Twitter, LinkedIn, and the Internet: micahfk
Editor's Notes
- Ranking factor studies have a controversial history with our industry, and I’m here today to posit that the issue in ranking studies is not that they are being done, but that they aren’t using the right toolsets with the right statistical framework when determining what is or is not a ranking factor.
- And the challenge behind solving why something ranks requires learning the statistical mechanics to help improve our game play, not about chasing or cheating the algorithm. It is these mechanics that I personally love learning for both ranking studies and board gaming, in other words, solving the challenge through understanding how those mechanics work.
- And yes, I’ll admit to a bit of nitpicking on published studies… Nonetheless, my hope is not that ranking studies must be true scientific papers (yet), but to make you better data analyzers and critiquers when producing or reading ranking studies.
- Who is this nitpicker? My name is Micah Fisher-Kirshner and I’m the Director of SEO & Content at Turn/River Capital, a growth equity firm based in San Francisco and I have 10+ years of SEO experience. I’m also the President and Founder of BayAreaSearch.org, the San Francisco SEO meet-up association. And yes, there's a mate in a few moves set-up in that photo, so a I’m a bit of nerd.
- In any case, let’s talk about what we should know with ranking studies when we look at the SERPs.
- We've become quite familiar with ranking studies using correlations (the y equals beta times x) which may seem complicated, fun to try, and applicable to what we believe we see on Google, yet these analyses traps us in an overly simplistic gaming environment like trying to win by owning Australia in a game of Risk.
- Because claims of a 0.2 being “highly” correlated is nothing more than cloud of uncertainty even when significant, and this leads people astray even with the correlation not equaling causation qualification… because people focus on the "highly“ and “significant”, not the warning.
- And rather than devalue the purpose of doing studies, we need to take from the scientific fields how they dive in to solve real world challenges. And that's where knowing what the toolsets are to achieve victory through various paths, be it technological advancement (aka Regressions), deck completion (aka Machine Learning), or points (aka Neural Networks). So why these?
- Well, regression-based analyses like this simplified example gives us far more information than any one to one correlation analysis.
- It allows us to determine what the causal factors are in one formula, …
- … which factors individually barely matter but together massively do…
- … and by how much they directly impact the result.
- Just off the harbor here is a great example of this. We know that sea levels are globally rising, but for Boston it's nearly 3 times the average. Scientists use regression analyses to determine why this is occurring without nature telling them. They were able to determine that the Gulf Stream slowing, major weather pattern shifts, and El Nino climate cycles were the reasons for that rise above the average. They don't use simplistic correlations, and neither should we when determining what is going on with Google’s algorithm.
- And in all honesty, the right statistical model depends on what you’re looking to analyze, making sure you have enough data for a valid analysis before deciding which statistical model to use. Where machine learning comes in as a tool for SEO analyses, is as pattern recognition for large datasets, fundamentally taking your data and running it back on itself to continually learn. It is the combination of statistics and programming that allows you to constantly update models as you get new data applying statistical best fit for an on-going analysis, provided you’re not accidentally overfitting, where you make the data fit what you put in.
- With neural networks, which is basically an even more advanced level of statistical analysis, it is helpful at solving extremely complex issues with a sufficiently massive dataset. It finds connections between variables that other statistical models aren’t designed to detect and is why Google turned to neural networks with BERT to better understand the relationship between words. The downside of this approach compared to a regression model, however, is the black box nature doesn’t provide the weights from the layers to the output, making it difficult, though not impossible, to explain causal reasons for what is going on.
- Nonetheless, a tool is only valuable to those that know how to properly use it; and that means it is imperative to know how to analyze the board in front of you when you plot how to win in the SERPs.
- To start, the scientific world knows to look deeper than a surface level thinking that, for example, a higher obesity rate means a higher life expectancy. They try different groups from their data bucketing by income to see if patterns hold (hint: it doesn’t). And so they dig deeper by collecting a sub-set of their original data and analyze if other factors might be the true causal reason. And this all goes to show the importance of looking at the factors as one set, not in isolated parts. So, let’s plot this game out with SEO.
- A good study will make sure that the data collected is consistent across one device, as measuring mobile rankings against desktop layouts is a false way to analyze what is going on. At the same time, making sure that there is a directional quality to the data being used before including in any analysis, even using different tools to analyzing the same metrics. Then on the flip side, it also requires an understanding that any initial low one-on-one correlation analysis should be removed, probably anything below 0.2. And importantly, it requires looking at the data set over a long time period, weekly probably being the safest, as it’ll weed out temporary effects, particularly around CTR which can be notoriously temporal.
- Advanced regression analyses common within the Econometrics field is to include interaction effects, also known as feature engineering, trying to find potential factors that only come out in combinations, such as positive ones like with authority, say where an author’s bio by itself doesn’t matter, but in combination with trusted links and citations can result in a positive boost. On the other end the potential negative combinations with spam such as a site with bad links and hidden text may trip quality checks.
- The crux of any study is how to interpret what you’re seeing. An SEO without a good data background or a Data Scientist without a good SEO background is going to come out with poor interpretations, it must be a mixture of both for successful reporting. And avoiding situations where the independent variable affects the dependent, known as endogeneity, becomes important. Further, does the result seem reasonable or is it going to cause consternation among your peers by claiming direct traffic or AdWords impacts SEO? As they say: extraordinary claims require extraordinary proof – if you don’t have that extraordinary proof, one must temper claims and review whether what you have is an outlier because of specific sites that take up and bias the data, be it Wikipedia, Amazon, or any other potential selection bias.
- You can further review your data set to see if groups of similar phrases, based on search volume, ranking positions, and more all show a similar result, solidifying that what you see isn’t an aberration. Further, check for the dreaded heteroskedasticity (yes, real word) where your correlation data is splitting apart requiring you to collect even more data behind what may be the real causes, including whether you’ve bucketed your keywords by the intent of buying or research.
- Once confident in your data, how you display your results really does matter for others to quickly grasp your presentation. Thus, fitting your graphs to a scatter plot showcases, from your goals, how strong the results are, though next level confidence in your uncertainties is the use of whisker plots as the way to display outputs, but that choice is really up to you based on what you expect people to take away. Just be sure for any regression analyses to include standardized data results for those more data-minded, as that makes it easier for everyone versed to analyze your study.
- The final round on your path to winning is running a last sanity check that what you’ve created is accurate. That requires separating out a portion of the data and running the results back in on itself to make sure it holds up. And cross-validation is a great way to make sure you’re not overfitting as you check on the accuracy of the model. But a true test is whether others see the same thing when they run the calculations themselves via a peer review. Because fundamentally they make you a better data analyzer when your work is cross-checked to find your blindspots. And while that is sufficient for the current game, by testing that same data strategy in subsequent matches, that truly tests your model for repeatable wins in the SERPs.
- All that energy, time, resources, and skills we spend as SEOs on the sites we work on deserve the same, if not more, effort when conducting SERPs ranking studies. There’s no excuse to shortcut how to make the claim for causal ranking factors.
- Ready to get rolling? Let’s look at just the starting cost for analyzing moving towards a more advanced ranking study. Assuming no penalties, no separate algorithms, etc. that’s easily ranging to about $20,000 for the year, which would certainly rise the more sophisticated and accurate you want to make this across the areas you want to analyze in Google’s algorithm. So, unless you’re working at a large company or at a tool vendor, it won’t be a cheap project to propose early in your new SEO job to say the least.
- But there really is an opportunity behind this to rank better on Google. Rather than solely relying on the goodwill of others, including Marie Haynes or Glenn Gabe providing their insight, you could personally know whether something was truly content and/or link based during any major Google algorithm update. And if you’re going to run it weekly, then simple shifts in positions from competitor changes would keep you on your toes in an ever-increasingly competitive environment (especially from those contra-the-guidelines) to help you prioritize your limited time, resources, and budget. Plus, nothing grates an engineer more than claiming you can hack their super-complicated 1,000+ variable algorithm down to a few monetary factors. Okay, so that’s more tongue in cheek. In all seriousness the ability to gain links and visibility (if you’re so inclined to share as you just watched via the competition before me) is a potential for an agency, a tool vendor, or individual to grab right now in our data-expanding industry with research journals hungry for info on SEO.
- Let’s reference back to this simplistic example to give an idea of what I planned to start with and showcase what I had in mind…
- The idea was to break down the algorithm into the major components and determine what data was available to collect within each section…
- … and within each section there would obviously be dive deeps into collecting individual components I thought realistically possible to directionally test including those I believe do and do not factor into the algorithm as a way to avoid my own sets of bias, all of which would be done as a time series and with categories as well…
- So, yeah… with all this together why haven’t I given this a go? It's not for the lack of trying to build the blueprints as it comes down solely to getting internal link data at scale given the size of the firm where I work. Thus, here's my personal begging at Ahrefs to get that into the API ASAP so I can come back next year and send in a research submission. In the meantime, some supplementary options we can consider - get the SEO testing platform via Distilled ODN, though just be aware you’ll need a lot of traffic for it to be effective. Because it’s either live testing or work towards constructing your own Google search engine and that requires some good engineers for that undertaking. But it is something John Mueller even advocated as it would give SEOs a better understanding of how search engines really work.
- Regardless, we can all today enjoy working on improving our personal SERP data analyzer skills.
- We all like succeeding, and to have that happen more often, we must demand better analyses. See a ranking study? Ask those to do better - use the scientific fields' toolset for complex questions, QA your data to avoid bad data, be reasonable with your claims based off of your proof, analyze how deep you can go given the resources available, use the right scatter or whisker plots to properly showcase your results, and make sure those studies have been peer reviewed for accuracy, because it's time to stop flying by the seat of our pants in a barely functioning ship in order to achieve: Victory of the SERPs.
- So before I go, I want to leave you with my citations plus a place to submit academic research submissions (thanks Dawn for the tip) and personal board gaming tips for some of the games shown within this presentation.
- And with that, I want to give a huge thanks to my colleague Christian, our in-house Data Scientist at Turn/River Capital for his help on fact checking this presentation. Cheers.