1. SolrCloud Failover
and Testing
Mark Miller
(Cloudera)

2. Mark Miller
Lucene Committer, Solr Committer.
Works for Cloudera.
A lot of work on SolrCloud.

3. At Cloudera…
We are building an Enterprise Data
Hub
Search is a part of that.
Solr is our search engine.

5. Solr On HDFS
Performance is
good.
It can be even
better.
A shared filesystem
has advantages.

6. SolrCloud Reminder

7. Limitation
We replicate via
both Solr and HDFS.
Replicating with
just one has huge
tradeoffs.
We are working on
better tradeoffs.

8. autoAddReplicas
A new per collection
option.
When a replica goes
down, it is replaced on
a node that is still up.
A shared filesystem as
well means all replicas
can go down and you
can still automatically
failover.
+
-

9. How Does it Work?
SolrCloud elects a fault tolerant,
single node to be an Overseer.
The Overseer monitors the cluster
state in ZooKeeper.
Creates a new SolrCore on a machine
that is up when necessary to replace
‘downed’ replicas.

10. Let’s Do A Demo!

11. SolrCloud Testing
Let’s talk about
tests.

12. SolrCloud Tests
We did a straw man implementation
of SolrCloud first.
We did the same for tests.
We favored integration tests over
unit tests.
We did not make enough tests.

13. Distributed Tests
Are hard.
For a variety of reasons.
The Lucene / Solr testing
framework hurts in order to help.

14. The Lucene / Solr
Test Framework
Randomized Testing.
Rule Enforcement.
The Jenkins Cluster.

15. Mocks
We avoided doing them early - too
much churn.
They can be dangerous to future
contributors / refactoring.
Some of the early mocking that did get
in is a little painful.
We need them for good unit tests.

16. Testing Culture
Lucene has A+ testing culture. In
many cases, it’s easier for Lucene.
Solr has a C testing culture.
Solr needs to get better.

17. Prescription?
More focus on back filling tests
when adding features or changing
code.
More focus on fixing frequently
failing tests.
More focus on unit tests.

18. The End
@heismark
Thank You.