Presented at Lucene/Solr Revolution 2017

1. Our Tale From the Trail of
Shadows
Chris Phillips and Dale Smith
REI Co-op

2. Who Are We?
Dale Smith
● REI Co-Op for 5 years
○ Lead Engineer: Search
Platform
● 40 years of software
engineering
● Boeing manufacturing
automation
● Seattle tech, including
○ Go2Net: Dogpile &
Metacrawler
@dasmith836

3. Who Are We?
Chris Phillips
● REI Co-op for 7 years
5 Years on Search
● 20 years in the industry
○ Half Life
○ Exchange Indexing
○ Launch of Xbox Kinect
@realchrispdad

4. 01
Who is REI?

5. Agenda
• What We Did
• Where We Were
• The Big Challenges
• Solutions – The Cool Stuff
• Questions

6. What We did
We replaced our SaaS search engine with an on-premise Solr stack

7. Where We Were
Stuck in the mud!!
● Not meeting customer or
internal expectations
● Slow searches: 750-900 ms
● Random outages
● Not flexible for our business
partners
● Limited visibility into the
servers

8. 01
The Goal
REI Co-Op: Quality Comes First

9. 01
The Goal
Customer Expectations
● Find the most relevant content quickly
● Go outside and be happy

10. The Goal
Internal Expectations
● Fast and stable search platform
● Rapid development cycle
● Save budget on licensing and support

11. The Big Challenges
No changes to the existing API’s
● Website
● Mobile apps
● Point of Sale apps
● Internal tools
Limit SEO impact

12. The Big Challenges
We were on our previous search engine
for 4 years.
● Had to tune relevance and
merchandising rules to match in a
matter of months
● Internal users had to learn new tools
● Biggest Challenge: How do we verify
we haven’t broken something in
search?

13. 01
The Big Challenges
The Solr Stack Is Awesome, But Scary
● Lucidworks Fusion
● Innovent Solutions Findtuner
● Multi-term synonyms
● “47 knobs that all go to Volume 11!”

14. The Trail of Success
Build a Lean Dream Team
● 2 Senior Back-End Developers
● 1 Senior QA Analyst
● 1 SDET
● PM created a “bubble of safety”
● Product Owner is the search
expert at REI

15. The Trail of Success
Nurture the Team Dynamic
● Trusted by leadership
● We trusted each other
● Accountability
● Committed to quality
● We knew what success looked like

16. The Trail of Success
Embrace The Team Philosophy

17. The Trail of Success
Quality: The measure of the excellence of
your product.
● “Quality is not an act. It is a habit.” --
Aristotle

18. The Trail of Success
Be Serious About Who Owns
Quality
● Everyone owns quality
● Build good habits

19. The Trail of Success
Good Habits
● Co-locate
● Keep the business owner
involved
● Engineers are cool!
● Find bugs before code is written
○ Architect together
○ Mental lifting
○ Risks
○ Unit tests
○ Test scenarios

20. The Trail of Success
Good Habits
● Avoid legacy code with 3
questions
○ Testable
○ Supportable
○ Maintainable

21. Solutions - Use Great Tools
● Innovent Solutions FindTuner
● Lucidworks Fusion
● Solr Cloud
● These tools were designed to
work together

22. Solutions - Innovent Solutions FindTuner
● Manages search rules
○ Product promotion
○ Banners, redirects, etc.
○ Query adjustments
● UI For Business Users
● Eliminated the need to
implement this in-
house to replicate
SaaS solution

23. Solutions - Lucidworks Fusion
● Manages Solr Cloud
configuration
● Provides configurable
search pipelines
● Nice UI for business
users
● Many more features!

24. Solutions - A Great Software Ecosystem
● Continuous Integration Model
● Docker Platform
● Microservices Architecture
● High-functioning Monitoring &
Metrics

25. Solutions – Dynamic Facets
The Problem
● ~200,000 product variants
● ~250 “facet-worthy” attributes
● Asking Solr to calculate facets
for all of these attributes on
each request would lead to
significant performance issues

26. Because Two Queries Are Better Than
One!
● Indexed products with list of
important attributes
● First Query: read and calculate
10-15 key attributes for facet request
01
Solutions – Dynamic Facets
● Second Query: full request with
selected facets
● Solr query only ever includes the
10-15 most important facets

27. The Problem:
● How do we verify product results from our new search stack?
● How do we verify performance capabilities under production load?
Solutions – The Shadow Query
The Solution:
● “Shadow Query”: all production searches are sent to both
search engines, the results compared using special comparison
functions, and metrics recorded on everything.
● All while not affecting existing search performance!
● Basically, we let our customers help test the system

28. Solutions – The Shadow Query Model

29. Solutions – The Shadow Query Model

30. Solutions – The Shadow Query Model

31. Solutions – Search API Normalization
REI search API is search-engine
agnostic
● Avoid exposure of search
engine-specific parameters or
attributes
● Preserve existing API contract
● Preserve existing SEO patterns
● Enable queries to multiple
search engines

32. Solutions – Search API Normalization
An Example
Normalized Refinement on REI.com:
r=features:Insulated;size:Large;color:Purple
fq=features:"Insulated"&fq={!tag=SIZE}size:"Large"&fq={!
tag=COLORFAMILY}color_family:"Purple"
Translated to Solr:
q1=insulated&x1=tab.ftrs&q2=large&x2=tab.size&q3=purple&x3=
colorfam
Translated to SaaS:

33. Solutions – Result ComparisonsComparing the Shadow Query Results
● Record query result times and deltas
● Record total number of results and deltas
● Record count of products in result A and not in result B, and vice-
versa
● Record Damerau-Levenshtein Distance, using product IDs instead
of letters
● Use graph dashboards to “visualize quality” as the search metrics
converged

34. Calculated based on operations transforming a list of products
from one sequence to another
● Insertion
● Deletion
● Substitution
● Transposition
● Each operation has a cost, weighted higher at the start of the
sequence
Solutions – Modified Damerau-Levenshtein
Distance

35. Preparing for ‘Solr Day’
● Load Test using recorded production queries replayed through the
shadow query at high load
● Monitored CPU and Memory - It’s all about that swap space
● Diagnose crashes from good bots doing bad things

36. It’s On!!!
We turned it on for our customers and…..
● Nobody noticed anything had changed
● They just enjoyed it being 600ms faster!

37. After The Party...
We continued to use the Shadow
Query to verify the new Solr stack
● Kept making calls to the SAAS
search engine in the
background
● Provided a safety net for an
important sale period
● Continued to use metrics to
validate rule refinements

38. Wisdom From The Trail
Pack It Out!
● Start with a solid team
● Trust matters in all directions
● Use the best tools
● Use data to validate everything
● Stop relying on hope and
heroics and instead bake QA
into the process and the
system

39. Awesome
People
Innovent Solutions
Jason Hellman, Anria Billavara, &
Team
Lucidworks
Greg Harris & Support
Team

40. Contact Us
Dale Smith
● Twitter: @dasmith836
● dasmith836@yahoo.com
● LinkedIn: dasmith836
● Facebook: dasmith836
(hm, I’m sensing a pattern...)
Chris
Phillips
● Twitter:
@realchrispdad
● LinkedIn:
realchrispdad

41. Thank
You