1. Setting up New Relic's Infrastructure:
A bunch of things we did wrong and a few we
did right.
DAVE PETERSON & NIC BENDERS
2013-10-25

2. David Peterson
(dave@newrelic.com)
Nic Benders
(nic@newrelic.com)

3. [“star” graph of code base at launch]

5. 100,000req/sec
2Gbps
200TB

6. Who Does What?

7. Site Operations != IT

8. You are not Batman
Do your work during the day

9. Lowest hour is still 60% of peak hour!

10. Put your servers
on the moon

11. Never open the box

12. Domain names are cheap

13. 10.0.0.0/31
10.0.0.0/30
10.0.0.0/29
10.0.0.0/28
10.0.0.0/27
10.0.0.0/26
10.0.0.0/25
10.0.0.0/24
10.0.0.0/23
10.0.0.0/22
10.0.0.0/21
10.0.0.0/20
10.0.0.0/19
10.0.0.0/18
10.0.0.0/17
10.0.0.0/16
10.0.0.0-10.0.0.1
10.0.0.0-10.0.0.3
10.0.0.0-10.0.0.7
10.0.0.0-10.0.0.15
10.0.0.0-10.0.0.31
10.0.0.0-10.0.0.63
10.0.0.0-10.0.0.127
10.0.0.0-10.0.0.255
10.0.0.0-10.0.1.255
10.0.0.0-10.0.3.255
10.0.0.0-10.0.7.255
10.0.0.0-10.0.15.255
10.0.0.0-10.0.31.255
10.0.0.0-10.0.63.255
10.0.0.0-10.0.127.255
10.0.0.0-10.0.255.255
255.255.255.254
255.255.255.252
255.255.255.248
255.255.255.240
255.255.255.224
255.255.255.192
255.255.255.128
255.255.255.0
255.255.254.0
255.255.252.0
255.255.248.0
255.255.240.0
255.255.224.0
255.255.192.0
255.255.128.0
255.255.0.0
Private IPs
are FREE
2
4
8
16
32
64
128
256
512
1024
2048
4096
8192
16384
32768
65536

14. Hardware

15. System Management

16. 189 ms
167 ms
158 ms
137 ms
Average response time
X6550
E5-2450
E5-2643
E5-2690

17. 25% performance improvement
2x 6-Core X6550
@ 2.67 GHz
2x 8-Core E5-2690
@ 2.90 GHz

18. Storage
√

19. Databases

20. 32-bits isn't enough
class CreateTribbles < ActiveRecord::Migration
def change
create_table :tribbles do |t|
t.boolean :hungry
t.string :color
t.references :parent, index: true
end
end
end

21. 32-bits isn't enough
`id` int(11) NOT NULL AUTO_INCREMENT
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT
`id` SERIAL

22. Your database is not a queue

23. Long running queries
will kill you

24. And now, a thanks to the people who are out
there every day fixing the mistakes that we
made:

25. Jonathan Owens
Sean Kane
Jonathan Thurman
Paul Showalter
Aaron Bento
Karl Matthias