O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Another Day in the Life of a Cloud Network Engineer at Netflix (NET312) - AWS re:Invent 2018

Making decisions today for tomorrow's technology—from DNS to AWS Direct Connect, ELBs to ENIs, VPCs to VPNs, the Cloud Network Engineering team at Netflix are resident subject matter experts for a myriad of AWS resources. Learn how a cross-functional team automates and manages an infrastructure that services over 125 million customers while evaluating new features that enable us to continue to grow through our next 100 million customers and beyond.

  • Seja o primeiro a comentar

Another Day in the Life of a Cloud Network Engineer at Netflix (NET312) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Another Day in the Life of a Cloud Network Engineer at Netflix Joel Kodama Cloud Network SRE Netflix N E T 3 1 2 Donavan Fritz Cloud Network SRE Netflix
  2. 2. RE:INVENT 2018 NET-312 AWS Cloud
  3. 3. RE:INVENT 2018 NET-312 AWS Cloud Amazon EC2 Classic VPC VPN Connection
  4. 4. RE:INVENT 2018 NET-312 Subnets IGWs VGWs NGWs Route tables Endpoints Prefix lists AWS Direct Connect CGWs
  5. 5. RE:INVENT 2018 NET-312 ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  6. 6. RE:INVENT 2018 NET-312 Amazon EC2 Classic VPC Zone 1 Region AWS Cloud Zone 2 Zone 3 Zone 1 Zone 2 Zone 3 Public Public Public Private Private Private NAT Gateway NAT Gateway NAT Gateway
  7. 7. RE:INVENT 2018 NET-312 Amazon EC2 Classic VPC Zone 1 Region AWS Cloud Zone 2 Zone 3 Zone 1 Zone 2 Zone 3 Public Public Public Private Private Private NAT Gateway NAT Gateway NAT Gateway
  8. 8. RE:INVENT 2018 NET-312 ●Inter-VPC communication VPC Region VPC Instances Instances 1 2 3 AWS Cloud VPC peering Internet
  9. 9. RE:INVENT 2018 NET-312 Bidirectional private IP communication Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  10. 10. RE:INVENT 2018 NET-312 Bidirectional private IP communication Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  11. 11. RE:INVENT 2018 NET-312 Bidirectional private IP communication Security group references Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  12. 12. RE:INVENT 2018 NET-312 Bidirectional private IP communication Security group references Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  13. 13. RE:INVENT 2018 NET-312 Bidirectional private IP communication Security group references No bandwidth constraints Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  14. 14. RE:INVENT 2018 NET-312 Bidirectional private IP communication Security group references No bandwidth constraints Internet VPC peering AWS Direct Connect ●Inter-VPC communication
  15. 15. RE:INVENT 2018 NET-312 Account 4 VPC Account 1 VPC Account 2 VPC Account 3 VPC Region 1 Instances Instances Instances Instances
  16. 16. RE:INVENT 2018 NET-312 AWS Cloud VPC us-west-2 Office building Netflix MPLS Backbone VPC us-east-1 VPC eu-west-1 SEA IAD LHR Office building
  17. 17. RE:INVENT 2018 NET-312 Single unified network across all accounts and regions
  18. 18. RE:INVENT 2018 NET-312 Unique
  19. 19. RE:INVENT 2018 NET-312 $ python3 >>>
  20. 20. RE:INVENT 2018 NET-312 $ python3 >>> import cldnet >>>
  21. 21. RE:INVENT 2018 NET-312 $ python3 >>> import cldnet >>> >>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16) >>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16) >>>
  22. 22. RE:INVENT 2018 NET-312 $ python3 >>> import cldnet >>> >>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16) >>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16) >>> >>> vpc0.peer_to(vpc1) >>>
  23. 23. RE:INVENT 2018 NET-312 $ python3 >>> import cldnet >>> >>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16) >>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16) >>> >>> vpc0.peer_to(vpc1) >>> >>> vpc0.resize(prefix_length=16) >>>
  24. 24. RE:INVENT 2018 NET-312 $ python3 >>> import cldnet >>> >>> cldnet.standardize_vpc_peerings(”peerings.yaml”) >>> # peerings.yaml --- VpcGroups: UsWest2FullMesh: - vpc-111111 - vpc-222222 - vpc-333333 UsEast1FullMesh: - vpc-444444 - vpc-555555 - vpc-666666 VpcPeerings: - FromGroup: UsEast1FullMesh ToGroup: UsEast1FullMesh - FromGroup: UsWest2FullMesh ToGroup: UsWest2FullMesh
  25. 25. RE:INVENT 2018 NET-312
  26. 26. RE:INVENT 2018 NET-312 Account 1 VPC us-west-2 Instances AWS Cloud Account 1 VPC us-east-1 Instances Account 1 VPC eu-west-1 Instances
  27. 27. RE:INVENT 2018 NET-312 Network-level encryption Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  28. 28. RE:INVENT 2018 NET-312 Network-level encryption Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  29. 29. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  30. 30. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  31. 31. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Security group references Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  32. 32. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Security group references Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  33. 33. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Security group references Summarizable inter-region VPC routes Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  34. 34. RE:INVENT 2018 NET-312 Network-level encryption No bandwidth constraints Security group references Summarizable inter-region VPC routes Inter Region VPC peering AWS Direct Connect ●VPC inter-region communication
  35. 35. RE:INVENT 2018 NET-312 Pushing limits
  36. 36. RE:INVENT 2018 NET-312
  37. 37. RE:INVENT 2018 NET-312 ●Titus network requirements IP per container (fidelity of IP address) Fast deployments
  38. 38. RE:INVENT 2018 NET-312 IP IPv4 addressElastic Network Interface Amazon EC2 Instance
  39. 39. RE:INVENT 2018 NET-312 IP IP IP IP IP IP IPv4 address Container Elastic Network Interface Amazon EC2 Instance
  40. 40. RE:INVENT 2018 NET-312 ●Container startup AWS API calls 1. CreateNetworkInterface 2. AttachNetworkInterface 3. AssignPrivateIpAddress 4. ModifyNetworkInterfaceAttribute
  41. 41. RE:INVENT 2018 NET-312 Can we populate network interfaces at start up?
  42. 42. RE:INVENT 2018 NET-312 0 100000 200000 300000 400000 500000 600000 1 2 3 4 5 NumberofIPaddress Instances w/ 8 Elastic Network Interfaces (ENIs) 2 IPs per ENI 4 IPs per ENI 8 IPs per ENI 16 IPs per ENI 32 IPs per ENI 1x /16 IPv4 CIDR 5x /16 IPv4 CIDR m4
  43. 43. RE:INVENT 2018 NET-312 VPC Public Private NAT Gateway Private Availability Zone Public Private NAT Gateway Private Availability Zone Public Private NAT Gateway Private Availability Zone EC2 Instances Containers EC2 Instances Containers EC2 Instances Containers EC2 Instances Containers EC2 Instances Containers EC2 Instances Containers
  44. 44. RE:INVENT 2018 NET-312 IP isolation Co-tenant ●Titus network options
  45. 45. RE:INVENT 2018 NET-312 IP isolation Co-tenant ●Titus network options
  46. 46. RE:INVENT 2018 NET-312 IP isolation Shared security groups Co-tenant ●Titus network options
  47. 47. RE:INVENT 2018 NET-312 IP isolation Shared security groups Co-tenant ●Titus network options
  48. 48. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Co-tenant ●Titus network options
  49. 49. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Co-tenant ●Titus network options
  50. 50. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Max number of IPs Co-tenant ~98K shared ●Titus network options
  51. 51. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Max number of IPs Co-tenant ~98K shared Dedicated subnet ~196K ●Titus network options
  52. 52. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Max number of IPs Co-tenant ~98K shared Dedicated subnet ~196K Dedicated VPC ~295K ●Titus network options
  53. 53. RE:INVENT 2018 NET-312 IP isolation Shared security groups AWS API isolation Max number of IPs Co-tenant ~98K shared Dedicated subnet ~196K Dedicated VPC ~295K Dedicated account ~295K ●Titus network options
  54. 54. RE:INVENT 2018 NET-312
  55. 55. RE:INVENT 2018 NET-312 ●DNS service discovery
  56. 56. RE:INVENT 2018 NET-312 Auto Scaling group State Eureka Foo Bar Where is bar? ●DNS service discovery
  57. 57. RE:INVENT 2018 NET-312 Auto Scaling group Auto Scaling group State Eureka In service Out of service Foo Bar v0 Bar v1 Where is bar? ●DNS service discovery
  58. 58. RE:INVENT 2018 NET-312 Auto Scaling group Auto Scaling group State Eureka Out of service In service Foo Bar v0 Bar v1 Where is bar? ●DNS service discovery
  59. 59. RE:INVENT 2018 NET-312 Eureka Foo Where is bar.netflix.net? ●DNS service discovery Auto Scaling group Auto Scaling group State Out of service In service Bar v0 Bar v1
  60. 60. RE:INVENT 2018 NET-312 Eureka Amazon Route53 Sync State Foo Where is bar.netflix.net? ●DNS service discovery Auto Scaling group Auto Scaling group State Out of service In service Bar v0 Bar v1
  61. 61. RE:INVENT 2018 NET-312 Eureka Amazon Route53 Sync State Foo Where is bar.netflix.net? ●DNS service discovery Auto Scaling group Auto Scaling group State Out of service In service Bar v0 Bar v1
  62. 62. RE:INVENT 2018 NET-312 Decision time
  63. 63. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution ●DNS service discovery decision
  64. 64. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution ●DNS service discovery decision
  65. 65. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution ●DNS service discovery decision
  66. 66. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution ●DNS service discovery decision
  67. 67. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming ●DNS service discovery decision
  68. 68. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming ●DNS service discovery decision
  69. 69. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming ●DNS service discovery decision
  70. 70. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming ●DNS service discovery decision
  71. 71. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming ??? ●DNS service discovery decision
  72. 72. RE:INVENT 2018 NET-312 Auto remove stale DNS records Allow “out of service” AWS API rate limits Existing solution Route53 auto naming Eureka DNS ●DNS service discovery decision
  73. 73. RE:INVENT 2018 NET-312 Auto Scaling group Auto Scaling group State Eureka Amazon Route53 Sync In service State Out of service Foo Bar Bar Where is bar.netflix.net?
  74. 74. RE:INVENT 2018 NET-312 Auto Scaling group Auto Scaling group State Eureka In service Out of service Foo Bar Bar Where is bar.netflix.net? Auto Scaling group Eureka DNS
  75. 75. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Authoritative name server static IP
  76. 76. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Authoritative name server static IP
  77. 77. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Authoritative name server static IP
  78. 78. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Elastic IP Authoritative name server static IP
  79. 79. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Elastic IP Authoritative name server static IP
  80. 80. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Elastic IP Elastic network interface Authoritative name server static IP
  81. 81. RE:INVENT 2018 NET-312 Static IPv4 IPv6 UDP Network Load Balancer Elastic IP Elastic network interface Authoritative name server static IP
  82. 82. RE:INVENT 2018 NET-312 Authoritative name server static IP VPC us-west-2 Subnet 203.0.113.100 2001:db8::100
  83. 83. RE:INVENT 2018 NET-312 Eureka DNS v0 Authoritative name server static IP VPC us-west-2 Subnet 203.0.113.100 2001:db8::100
  84. 84. RE:INVENT 2018 NET-312 Eureka DNS v0 Authoritative name server static IP VPC us-west-2 Subnet 203.0.113.100 2001:db8::100 Eureka DNS v1
  85. 85. RE:INVENT 2018 NET-312 Eureka DNS v1 Eureka DNS v0 Authoritative name server static IP VPC us-west-2 Subnet 203.0.113.100 2001:db8::100
  86. 86. RE:INVENT 2018 NET-312 Eureka DNS v1 Authoritative name server static IP VPC us-west-2 Subnet 203.0.113.100 2001:db8::100
  87. 87. RE:INVENT 2018 NET-312 Authoritative name server static IP VPC Public us-west-2 AWS Cloud us-west-2a Auto Scaling group Eureka DNS Public us-west-2b Eureka DNS Public us-west-2c Eureka DNSEureka DNS Eureka DNS Eureka DNS
  88. 88. RE:INVENT 2018 NET-312 DNS delegation hierarchy eureka.netflix.net
  89. 89. RE:INVENT 2018 NET-312 DNS delegation hierarchy test prod eureka.netflix.net
  90. 90. RE:INVENT 2018 NET-312 DNS delegation hierarchy test us-east-1 us-west-2 prod us-east-1 us-west-2 eureka.netflix.net
  91. 91. RE:INVENT 2018 NET-312 DNS delegation hierarchy test us-east-1 us-west-2 MyService MyService prod us-east-1 us-west-2 MyService MyService eureka.netflix.net
  92. 92. RE:INVENT 2018 NET-312 DNS delegation hierarchy test us-east-1 us-west-2 MyService MyService prod us-east-1 us-west-2 MyService MyService myservice.us-east-1.test.eureka.netflix.net myservice.us-west-2.test.eureka.netflix.net myservice.us-east-1.prod.eureka.netflix.net myservice.us-west-2.prod.eureka.netflix.net eureka.netflix.net
  93. 93. RE:INVENT 2018 NET-312 Let’s be more proactive!
  94. 94. RE:INVENT 2018 NET-312 DNS steering VPC us-west-2 AWS Cloud VPC us-east-1 VPC eu-west-1 Where is www.netflix.com?
  95. 95. RE:INVENT 2018 NET-312 DNS steering VPC us-west-2 AWS Cloud VPC us-east-1 VPC eu-west-1 ? ? ?
  96. 96. RE:INVENT 2018 NET-312 DNS steering: Geography VPC us-west-2 AWS Cloud VPC us-east-1 VPC eu-west-1 ? ? ?
  97. 97. RE:INVENT 2018 NET-312 DNS steering: Geography VPC us-west-2 AWS Cloud VPC us-east-1 VPC eu-west-1 Users in San Francisco
  98. 98. RE:INVENT 2018 NET-312 DNS steering: Geography VPC us-west-2 AWS Cloud VPC us-east-1 VPC eu-west-1 Users in London
  99. 99. RE:INVENT 2018 NET-312 DNS steering: Geography Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Latency: Not optimized
  100. 100. RE:INVENT 2018 NET-312 DNS steering: Geography Perfect balance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Latency: Not optimized
  101. 101. RE:INVENT 2018 NET-312 DNS steering: Geography Perfect balance Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Latency: Not optimized Availability risk
  102. 102. RE:INVENT 2018 NET-312 DNS steering: Geography Perfect balance (3 region) Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Latency: Not optimized Availability risk Cost driver Perfect balance (2 region)
  103. 103. We can do one better
  104. 104. RE:INVENT 2018 NET-312 Amazon Route53 latency based routing
  105. 105. RE:INVENT 2018 NET-312 DNS steering: Route 53 latency routing VPC us-west-2 AWS Cloud VPC us-east-1 Account 1 VPC eu-west-1 Users in San Francisco
  106. 106. RE:INVENT 2018 NET-312 VPC us-west-2 AWS Cloud VPC us-east-1 Account 1 VPC eu-west-1 Users in London DNS steering: Route 53 latency routing
  107. 107. RE:INVENT 2018 NET-312 Latency: optimized DNS steering: Route 53 latency routing
  108. 108. RE:INVENT 2018 NET-312 Latency: optimized Availability risk Cost driver Perfect balance (3 region) Perfect balance (2 region) DNS steering: Route 53 latency routing
  109. 109. RE:INVENT 2018 NET-312 DNS steering
  110. 110. RE:INVENT 2018 NET-312 DNS steering
  111. 111. RE:INVENT 2018 NET-312 DNS steering 120ms 80ms 100ms
  112. 112. RE:INVENT 2018 NET-312 DNS steering 120ms 80ms 100ms 20ms 200ms 100ms
  113. 113. RE:INVENT 2018 NET-312 DNS steering 120ms 80ms 100ms 300ms 120ms 150ms 20ms 200ms 100ms
  114. 114. RE:INVENT 2018 NET-312 DNS steering 120ms 80ms 100ms 300ms 120ms 150ms 20ms 200ms 100ms
  115. 115. RE:INVENT 2018 NET-312 us-west-2 us-east-1 eu-west-1 Wisconsin, USA 100ms 80ms 120ms Argentina 150ms 120ms 300ms Spain 200ms 100ms 20ms Real User Measurements DNS steering: Route 53 latency routing
  116. 116. RE:INVENT 2018 NET-312 us-west-2 us-east-1 eu-west-1 Wisconsin, USA 100ms 80ms 120ms Argentina 150ms 120ms 300ms Spain 200ms 100ms 20ms Real User Measurements DNS steering: Geography
  117. 117. RE:INVENT 2018 NET-312 us-west-2 us-east-1 eu-west-1 Wisconsin, USA 100ms 80ms 120ms Argentina 150ms 120ms 300ms Spain 200ms 100ms 20ms Geo IP Prefix Wisconsin, USA 203.0.113.0/24 Argentina 192.0.2.0/24 Spain 2001:DB8::/32 Geo Database Real User Measurements DNS steering: Geography
  118. 118. RE:INVENT 2018 NET-312 Prefix steering
  119. 119. RE:INVENT 2018 NET-312 DNS steering: Prefix steering us-west-2 us-east-1 eu-west-1 203.0.113.0/24 100ms 80ms 120ms 192.0.2.0/24 150ms 120ms 300ms 2001:DB8::/32 200ms 100ms 20ms Real User Measurements
  120. 120. RE:INVENT 2018 NET-312 DNS steering: Prefix steering us-west-2 us-east-1 eu-west-1 203.0.113.0/24 100ms 80ms 120ms 192.0.2.0/24 150ms 120ms 300ms 2001:DB8::/32 200ms 100ms 20ms Real User Measurements 100ms latency delta
  121. 121. RE:INVENT 2018 NET-312 DNS steering: Prefix steering Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Latency: delta optimized Perfect balance Availability risk
  122. 122. RE:INVENT 2018 NET-312
  123. 123. RE:INVENT 2018 NET-312
  124. 124. RE:INVENT 2018 NET-312 Decision time
  125. 125. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Joel Kodama Cloud Network SRE jkodama@netflix.com Donavan Fritz Cloud Network SRE dfritz@netflix.com
  126. 126. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×