Sometimes your fastest queries can cause the most problems. I'll take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits.. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

1. @molly_struve
Cache is King
Get the Most Bang for Your Buck From Ruby

2. @molly_struve
Site Reliability Engineer

3. @molly_struve

4. @molly_struve
Adding Indexes
Using SELECT statements
Batch Processing

5. @molly_struve
Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

6. @molly_struve
Resque

7. @molly_struve

8. @molly_struve
Demo Time!

9. @molly_struve
Quantity of Datastore Hits

10. @molly_struve

11. @molly_struve
The average company has...
60 thousand
assets
24 million
vulnerabilities?

12. @molly_struve
MySQL
Elasticsearch
Cluster

13. @molly_struve
Serialization

14. @molly_struve
MySQL
Elasticsearch
Cluster
ActiveModelSerializers

15. @molly_struve
module Beehive
module Serializers
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end
end
end

16. @molly_struve
200 MILLION

17. @molly_struve
11 hours and counting...

18. @molly_struve

19. @molly_struve

20. @molly_struve
(1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

21. @molly_struve
MySQL

22. @molly_struve
Bulk Serialization

23. @molly_struve
class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

24. @molly_struve
module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

25. @molly_struve
class Vulnerability
has_many :custom_fields
end

26. @molly_struve
CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

27. @molly_struve
The Result...
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

28. @molly_struve
Decrease in database hits
Individual Serialization:
Bulk Serialization:
2,100
7

29. @molly_struve
1k vulns 1k vulns 1k vulns
Vulnerability Batches

30. @molly_struve
1k vulns 1k vulns 1k vulns
Vulnerability Batches
7k 7

31. @molly_struve
MySQL Queries
Bulk Serialization
Deployed

32. @molly_struve
Bulk Serialization
Deployed
RDS CPU Utilization

33. @molly_struve
Process
in Bulk

34. @molly_struve

35. @molly_struve

36. @molly_struve
Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

37. @molly_struve
Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

38. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

39. @molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

40. @molly_struve
(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

41. @molly_struve

42. @molly_struve

43. @molly_struve
client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

44. @molly_struve
indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

45. @molly_struve
1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

46. @molly_struve
1000x

47. @molly_struve
65% job
speed up

48. @molly_struve
Local Cache

49. @molly_struve
Redis

50. @molly_struve
Process
in Bulk
Hash
Cache

51. @molly_struve
Sharded Databases
CLIENT 1
CLIENT 2
CLIENT 3

52. @molly_struve
Asset.with_shard(client_id).find(1)

53. @molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration

54. @molly_struve

55. @molly_struve
Sharding Configuration Size
20 bytes 1kb 13kb

56. @molly_struve
285 Workers

57. @molly_struve
7.8 MB/second

58. @molly_struve
ActiveRecord::Base.connection

59. @molly_struve
(pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

60. @molly_struve
module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

61. @molly_struve
Know your gems

62. @molly_struve
Process
in Bulk
Framework
Cache
Hash
Cache

63. @molly_struve
Avoid making datastore hits you don’t
need

64. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

65. @molly_struve
FALSE

66. @molly_struve
(pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

67. @molly_struve

68. @molly_struve
return unless user_ids.any?
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

69. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
(pry)> Benchmark.realtime do
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

70. @molly_struve
(pry)> Benchmark.realtime do
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
“Ruby is slow”Hitting the database is slow!

71. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

72. @molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end
users = User.where(:id => user_ids).active.short.single

73. @molly_struve
.none

74. @molly_struve
(pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND
`users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []
.none in action...

75. @molly_struve

76. @molly_struve

77. @molly_struve
Logging
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

78. @molly_struve
Preventing useless
datastore hits

79. @molly_struve

80. @molly_struve
Report
Elasticsearch
MySQL Redis

81. @molly_struve
(pry)> Report.blank_reports.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

82. @molly_struve
Report
Elasticsearch
MySQL Redis

83. @molly_struve

84. @molly_struve
return if report.asset_count.zero?

85. @molly_struve
10+ hrs

86. @molly_struve
3 hrs

87. @molly_struve
Process
in Bulk
Framework
Cache
Database
Guards
Hash
Cache

88. @molly_struve
Resque Workers
Redis

89. @molly_struve
45 workers
45 workers
45 workers

90. @molly_struve
70 workers
70 workers
70 workers

91. @molly_struve

92. @molly_struve

93. @molly_struve

94. @molly_struve
48 MB
16 MB

95. @molly_struve
Redis Requests
70 workers
100k
200k

96. @molly_struve

97. @molly_struve
?

98. @molly_struve
Resque
Throttling

99. @molly_struve
Redis Requests
100k
200k

100. @molly_struve
Redis Network Traffic
48MB
16MB

101. @molly_struve

102. @molly_struve
Process
in Bulk
Framework
Cache
Database
Guards
Remove
Datastore
Hits
Hash
Cache

103. @molly_struve
Every
datastore hit
COUNTS

104. @molly_struve

105. @molly_struve
Questions

106. @molly_struve
Contact
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com