Query performance can either be a constant headache or the unsung hero of an application. MongoDB provides extremely powerful querying capabilities when used properly. As a senior member of the support team I will share more common mistakes observed and some tips and tricks to avoiding them.
9. E-S-R
The ordering of index keys in a compound index is critically important.
Sure, but does it really matter?
10. E-S-R
The ordering of index keys in a compound index is critically important.
E-S-R provides guidance that is useful in most cases:
• Equality first
• Sort next
• Range last
11. E-S-R
The ordering of index keys in a compound index is critically important.
E-S-R provides guidance that is useful in most cases:
• Equality first
• Sort next
• Range last
What is the difference between
Equality and Range?
12. Equality Fields
Definitions
An exact match on a single
value. For example:
• {x:123}
• {x:{$eq:123}}
• {"x.y":123}
Bounds:
"[123.0, 123.0]"
Sort Range Predicates
13. Definitions
Sort
The (entire) requested sort.
.sort({a:1})
.sort({a:1, b:-1})
An exact match on a single
value. For example:
• {x:123}
• {x:{$eq:123}}
• {"x.y":123}
Bounds:
"[123.0, 123.0]"
Bounds:
"[MinKey, MaxKey]"
Range PredicatesEquality Fields
"[MinKey, MaxKey]",
"[MinKey, MaxKey]"
14. Definitions
Sort Range Predicates
Any predicates that are not
exact matches. Some
operators include:
• {z:{$gt:9000}}
• {foo:{$lte:1000}}
The (entire) requested sort.
.sort({a:1})
.sort({a:1, b:-1})
An exact match on a single
value. For example:
• {x:123}
• {x:{$eq:123}}
• {"x.y":123}
Bounds:
"[123.0, 123.0]"
Bounds:
"[MinKey, MaxKey]",
"[MinKey, MaxKey]"
Bounds:
"(9000.0, inf.0]"
"[-inf.0, 1000]"
Equality Fields
16. Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
17. Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
Why?
18. Equality
Equality keys are placed first in any order
If present in the query shape,
equality fields should always form the prefix for the index.
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
19. Equality
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
If present in the query shape,
equality fields should always form the prefix for the index.
Equality keys are placed first in any order
40. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
R
E R
E
Equality before Range
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
41. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
R
E R
E
Equality before Range
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
Hey! Don’t forget about your
demo!!
43. Sort fields are placed next
Placing sort predicates after sequential equality keys allows the index to:
- Provide a non-blocking sort.
- Minimize the amount of scanning required.
db.games.find( {gamertag: "Ace"} ).sort( {score: 1} )
Sort
68. Sort after Equality
S
E S
E
db.games.find( {gamertag: "Ace"} ).sort( {score: 1} )
Don’t do this, it may require a full index scan
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
70. Range fields are usually last
This allows them to :
Still participate in filtering the data
But does not force a blocking sort.
db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
Range
72. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
R
E R
E
Range after Equality
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
73. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
R
E R
E
Range
But should range come after
sort too?
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
74. R
E R
E
Range
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
But should range come after
sort too?
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
75. R
E R
E
Range
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
But should range come after
sort too?
Index: {gamertag:1, score:1}Index: {score:1, gamertag:1}
db.games.createIndex({…, score:1, gamertag:1})ANTI-pattern index:
76. 100 15,000 50,000 99,999
Ace Bob Bob Ace
*
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
Range
gamertag
score
db.games.createIndex({…, score:1, gamertag:1})ANTI-pattern index:
77. 100 15,000 50,000 99,999
Ace Bob Bob Ace
*
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
Range
gamertag
score
db.games.createIndex({…, score:1, gamertag:1})ANTI-pattern index:
78. 100 15,000 50,000 99,999
Ace Bob Cal Ace
*
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
Range
gamertag
score
db.games.createIndex({…, score:1, gamertag:1})ANTI-pattern index:
79. 100 15,000 50,000 99,999
Ace Bob Cal Ace
*
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
Range
{} {} {} {}{1}{2} {3}
gamertag
score
db.games.createIndex({…, score:1, gamertag:1})ANTI-pattern index:
104. BS
R S
R
S
db.games.find({*, score: {$gt: 9000}}).sort({gamertag: 1})
Range after Sort
Index: {…, gamertag:1, score:1}
Index: {…, score:1, gamertag:1}
113. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
114. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
115. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
planecar racecar
116. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
planecar racecar
117. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
planecar racecar
118. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
p???ecar racecar
119. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
p???ecar racecar
120. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
121. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry zebra
122. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry zebra
123. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
124. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
125. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
126. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
raincarry Carpool
Since the regex is left
anchored, can’t we skip
the middle tree?
127. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
Since the regex is left
anchored, can’t we skip
the middle tree?
128. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
Since the regex is left
anchored, can’t we skip
the middle tree?
129. Operator Type Check - Regex
Regex Operators: E, S, or R?
• {str:/car/}
• {str:/^car/i}
Range!
cArrycarry Carpool
130. Operator Type Check - $in
$in filters: E, S, or R?
• {field:{$in:[1,3]}}
… it depends with respect to query shape
and key ordering
Alone: a series of Equality matches
Combined: possibly a Range
132. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
db.coll.createIndex({field1:1, field2:1})
133. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
?
1 3
134. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
?
31
135. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
?
31
136. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
?
31
137. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
1 3
?
138. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
1 2 3
C G B F A D
db.coll.createIndex({field1:1, field2:1})
139. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
]
}
?
2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
1 3
140. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"field2" : [
"[MinKey, MaxKey]"
]
}
?
2
C G B F A D
1 3
db.coll.createIndex({field1:1, field2:1})
141. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"field2" : [
"[MinKey, MaxKey]"
]
}
?
2
C G B F A D
1 3
db.coll.createIndex({field1:1, field2:1})
142. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"field2" : [
"[MinKey, MaxKey]"
]
}
?
2
C G B F A D
1 3
db.coll.createIndex({field1:1, field2:1})
143. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"field2" : [
"[MinKey, MaxKey]"
]
}
?
2
C G B F A D
BS
1 3
db.coll.createIndex({field1:1, field2:1})
144. Operator Type Check - $in
.find({field1:{$in:[1,3]}})
.sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
"[3.0, 3.0]"
],
"field2" : [
"[MinKey, MaxKey]"
]
}
?
1 2 3
C G B F A D
BS
db.coll.createIndex({field1:1, field2:1})
145. Operator Type Check - $in
db.coll.createIndex({field1:1, field2:1})
.find({field1:{$in:[1,3]}}).sort({field2:1})
150. .find({field1:{$in:[1,3]}}).sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field1" : [
"[3.0, 3.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
Operator Type Check - $in
2 3
C G B F A D
?
1 2 3
C G B F A D
db.coll.createIndex({field1:1, field2:1})
?
1
151. .find({field1:{$in:[1,3]}}).sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field1" : [
"[3.0, 3.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
Operator Type Check - $in
1 2 3
C G B F A D
1 2 3
C G B F A D
db.coll.createIndex({field1:1, field2:1})
? ?
152. .find({field1:{$in:[1,3]}}).sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field1" : [
"[3.0, 3.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
Operator Type Check - $in
2 3
C G B F A D
1 2
C G B F A D
db.coll.createIndex({field1:1, field2:1})
3
?
1
?
153. .find({field1:{$in:[1,3]}}).sort({field2:1})
"indexBounds" : {
"field1" : [
"[1.0, 1.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
"indexBounds" : {
"field1" : [
"[3.0, 3.0]",
],
"field2" : [
"[MinKey, MaxKey]"
]
}
Operator Type Check - $in
2 3
C G B F A D
1 2
C G B F A D
{2} {4} {1} {3}
Sort Merge
db.coll.createIndex({field1:1, field2:1})
?
1
?
3
156. Exceptions
Is the E-S-R “rule” always optimal?
Nope.
Consider the following query:
db.games
.find({gamertag:"Ace", date:{$gte:2019}})
.sort({score:1})
181. E-S-R Guidance
A good starting place applicable to
most use cases
Place keys in the following order:
Equality first
Sort next
Range last
Remember:
• Some operators may be range
instead of equality
• Having consecutive keys used in
the index is important
• Specifics about your data set may
need a different approach
182. Tips and Tricks++ for Indexing MongoDB
Ronan Bohan https://www.surveymonkey.com/r/KF8KX7B
187. Consecutive Index Keys
Given Indexes:
{gamertag:1, date:1, game:1}
{gamertag:1, game:1, date:1}
Which one is “better”?
It depends on the query shape(s)!
Consider the query:
.find({
gamertag: "Ace",
game: "Halo"
})