08448380779 Call Girls In Greater Kailash - I Women Seeking Men
SSL Certificate Expiration and Howler Monkey's Inception
1. SSL* Certificate Reporting
BayLISA
March 21st, 2013
@royrapoport rsr@netflix.com
Friday, March 22, 13
This is the story of how we went from SSL certificates expiring without notice in production to
deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL
certificate expiration as a production-class issue.
2. SSL* Certificate Reporting
BayLISA
March 21st, 2013
@royrapoport rsr@netflix.com
Friday, March 22, 13
This is the story of how we went from SSL certificates expiring without notice in production to
deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL
certificate expiration as a production-class issue.
7. Culture Overview
@royrapoport rsr@netflix.com
Friday, March 22, 13
We hire very smart people, give them all the context and situational awareness they want, and
set them free. We design our environment, our systems, and our teams to be empowered to
make decisions without requiring slow approval processes, cumbersome formal
communication, or any other unnecessary friction.
8. Culture Overview
• Freedom and
Responsibility
@royrapoport rsr@netflix.com
Friday, March 22, 13
We hire very smart people, give them all the context and situational awareness they want, and
set them free. We design our environment, our systems, and our teams to be empowered to
make decisions without requiring slow approval processes, cumbersome formal
communication, or any other unnecessary friction.
9. Culture Overview
• Freedom and
Responsibility
• Distributed
Operations
@royrapoport rsr@netflix.com
Friday, March 22, 13
We hire very smart people, give them all the context and situational awareness they want, and
set them free. We design our environment, our systems, and our teams to be empowered to
make decisions without requiring slow approval processes, cumbersome formal
communication, or any other unnecessary friction.
10. Culture Overview
• Freedom and
Responsibility
• Distributed
Operations
• Get out of the
way of
Developers
@royrapoport rsr@netflix.com
Friday, March 22, 13
We hire very smart people, give them all the context and situational awareness they want, and
set them free. We design our environment, our systems, and our teams to be empowered to
make decisions without requiring slow approval processes, cumbersome formal
communication, or any other unnecessary friction.
12. So Certificates ...
• Dozens of Certificates
@royrapoport rsr@netflix.com
Friday, March 22, 13
13. So Certificates ...
• Dozens of Certificates
• Different kinds of places
@royrapoport rsr@netflix.com
Friday, March 22, 13
14. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
@royrapoport rsr@netflix.com
Friday, March 22, 13
15. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
@royrapoport rsr@netflix.com
Friday, March 22, 13
16. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
• ELBs
@royrapoport rsr@netflix.com
Friday, March 22, 13
17. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
• ELBs
• EC2
@royrapoport rsr@netflix.com
Friday, March 22, 13
18. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
• ELBs
• EC2
• Source Control
@royrapoport rsr@netflix.com
Friday, March 22, 13
19. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
• ELBs
• EC2
• Source Control
• EIPs
@royrapoport rsr@netflix.com
Friday, March 22, 13
20. So Certificates ...
• Dozens of Certificates
• Different kinds of places
• Datacenter/private
• Datacenter/public/LB
• ELBs
• EC2
• Source Control
• EIPs
• Totally Distributed Design
@royrapoport rsr@netflix.com
Friday, March 22, 13
21. So Certificates ...
• Some Certificates Weren’t[sic]
@royrapoport rsr@netflix.com
Friday, March 22, 13
Some certificates weren’t even SSL certificates -- we have certificates we get from a partner
that cannot be accessed via SSL, and for which the answer to the question “when does this
expire?” require scraping a web page.
22. So Certificates ...
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
23. So Certificates ...
• SSL Certificates expire
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
24. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
25. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
• Hilarity ensues
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
26. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
• Hilarity ensues
• Standard Ways to Solve This
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
27. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
• Hilarity ensues
• Standard Ways to Solve This
• Excel worksheets
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
28. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
• Hilarity ensues
• Standard Ways to Solve This
• Excel worksheets
• Wiki documents
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
29. So Certificates ...
• SSL Certificates expire
• Millions of people can’t stream
• Hilarity ensues
• Standard Ways to Solve This
• Excel worksheets
• Wiki documents
• Events on public calendars
@royrapoport rsr@netflix.com
Friday, March 22, 13
(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in
fact, the standard ways in which most organizations try to deal with keeping up with SSL
certificate expirations)
30. Let’s Do This Thing
Cassandra
Certificate
@royrapoport rsr@netflix.com
Friday, March 22, 13
Start with a very simple model -- a Certificate entity, which is really just a combination of
name, expiration date, and a series of locations where we can find this. It’d be trivial to feed
this thing from my todo list, if I wanted to (but given the state of my todo list, probably a bad
idea)
31. Let’s Do This Thing
ELB
Cassandra
Certificate
@royrapoport rsr@netflix.com
Friday, March 22, 13
Then start building location-aware spiders -- e.g. this spider that knows how to probe all our
ELBs to see if they listen on 443 and gets their certificate if they do.
32. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
Certificate
@royrapoport rsr@netflix.com
Friday, March 22, 13
Or this spider that knows how to talk to a specific kind of EC2 instance we have with some
certificates.
33. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
IP Range
Certificate
@royrapoport rsr@netflix.com
Friday, March 22, 13
etc ...
34. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
IP Range
Certificate
Filesystem
@royrapoport rsr@netflix.com
Friday, March 22, 13
35. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
IP Range
Certificate
Filesystem
DNS
@royrapoport rsr@netflix.com
Friday, March 22, 13
36. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
IP Range
Certificate
Filesystem
DNS
@royrapoport rsr@netflix.com
Friday, March 22, 13
Once you have all this information, you can easily generate a web page showing certificates,
where they are, and when they expire
37. Let’s Do This Thing
ELB
Cassandra
EC2 Instance
IP Range
Certificate
Filesystem
DNS
@royrapoport rsr@netflix.com
Friday, March 22, 13
And send out emails, too -- once we built the capability for teams to subscribe to emails for
a given certificate and specify how many days before expiration they should start getting
notified
38. Since Then
@royrapoport rsr@netflix.com
Friday, March 22, 13
We validated the design by continuing to iterate on it -- recently, when building the DNS
spider component, that work took only about 15 minutes to implement. We also expanded
subscription capabilities so teams could subscribe to certificate expiration warnings based on
certificate name regular expressions.
39. Since Then
• No Production Emergencies due to SSL
certificate expiration
@royrapoport rsr@netflix.com
Friday, March 22, 13
We validated the design by continuing to iterate on it -- recently, when building the DNS
spider component, that work took only about 15 minutes to implement. We also expanded
subscription capabilities so teams could subscribe to certificate expiration warnings based on
certificate name regular expressions.
40. Since Then
• No Production Emergencies due to SSL
certificate expiration
• Validated Design
@royrapoport rsr@netflix.com
Friday, March 22, 13
We validated the design by continuing to iterate on it -- recently, when building the DNS
spider component, that work took only about 15 minutes to implement. We also expanded
subscription capabilities so teams could subscribe to certificate expiration warnings based on
certificate name regular expressions.
41. Since Then
• No Production Emergencies due to SSL
certificate expiration
• Validated Design
• Better Subscription Capabilities
@royrapoport rsr@netflix.com
Friday, March 22, 13
We validated the design by continuing to iterate on it -- recently, when building the DNS
spider component, that work took only about 15 minutes to implement. We also expanded
subscription capabilities so teams could subscribe to certificate expiration warnings based on
certificate name regular expressions.
42. Soon ...
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
43. Soon ...
• Customized, automated alerting
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
44. Soon ...
• Customized, automated alerting
• Automated renewal
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
45. Soon ...
• Customized, automated alerting
• Automated renewal
• Telling you a problem is about to happen:
Good
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
46. Soon ...
• Customized, automated alerting
• Automated renewal
• Telling you a problem is about to happen:
Good
• Preventing the problem automatically:
Priceless
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
47. Soon ...
• Customized, automated alerting
• Automated renewal
• Telling you a problem is about to happen:
Good
• Preventing the problem automatically:
Priceless
• Open Source
@royrapoport rsr@netflix.com
Friday, March 22, 13
We should be able to figure out who owns a certificate, most of the time, and alert them
directly even if they don’t set up a subscription.
48. Remember ...
@royrapoport rsr@netflix.com
Friday, March 22, 13
49. Remember ...
• Be Lazy
@royrapoport rsr@netflix.com
Friday, March 22, 13
50. Remember ...
• Be Lazy
• Help Others Be Lazy
@royrapoport rsr@netflix.com
Friday, March 22, 13
51. Remember ...
• Be Lazy
• Help Others Be Lazy
• Computers Are Better Than Humans
@royrapoport rsr@netflix.com
Friday, March 22, 13
52. Remember ...
• Be Lazy
• Help Others Be Lazy
• Computers Are Better Than Humans
• For some things
@royrapoport rsr@netflix.com
Friday, March 22, 13
53. Remember ...
• Be Lazy
• Help Others Be Lazy
• Computers Are Better Than Humans
• For some things
• Don’t compete on their terms
@royrapoport rsr@netflix.com
Friday, March 22, 13
54. Questions?
@royrapoport rsr@netflix.com
Friday, March 22, 13