Twilio provides a communications API that enables voice, VoIP, and messaging capabilities for web and mobile apps. They migrated their infrastructure from the isolated EC2-Classic platform to EC2-VPC to enable global routing between regions and services. This reduced complexity, improved performance and latency, and allowed for more frequent and less risky deployments. The migration required bridging traffic between EC2-Classic and EC2-VPC instances and using software routers and service discovery for peering between regions. The new global VPC infrastructure improved customer experience and satisfaction.
13. Issues
All regions completely separated
–Traffic has to go through known endpoints
•Un-necessary hops
•Complicates deployments
•More difficult to debug
•Easier to create routing bugs
14. Forcing us to…
•Open up firewalls
•Secure traffic between regions using our own “VPN”
•Traffic has to go through known endpoints
•Known endpoints assigned EIPs
15. Which means…
•Un-necessary hops
•Complicates deployments
•More difficult to debug
•Easier to introduce bugs
•Cannot deploy nodes behind EIPs without affecting traffic
16. That translates to…
•Fewer deploys
•Riskier deploys
•Harder to nail down bugs
•Takes longer to get fixes out
•Less happy customers!
18. “EC2 2.0”(aka EC2-VPC)
•Global routing tables
•Enhanced Networking with SR-IOV
•Elastic network interfaces
•Software defined network
•Hardware security manager
Twilioconsiders VPC an evolutionary step or upgrade of the Amazon EC2 platform.
19. Global routingtables
•Per subnet or per VPC routing tables
•Route traffic to instances
•Tunnel traffic between regions
Routing traffic to instances enables the easy creation of things like loadbalancers, tunnels, or even VPCs inside of VPCs.
20. HVM and SR-IOV
•HVM images with Enhanced Networking
•PCI Express speeds to network adapter
•Low-latency access to network adapter
•Up to 10gb network speeds
Enhanced Networking with SR-IOV means fast performance even under virtualized hardware.
21. Elastic networkinterfaces
•Multiple EIPs and multiple private IPs
•Multiple ENIs per instance
•Security groups follow an ENI
•ENI has a MAC address
ENIs are more like network cards that you can move around and attach to different instances.
22. Software definednetwork
•Control over my instances’ routes
•Number my own network
•Network ACLs
•Data-in-transit protected by more than just a security group
•Provision networks like virtual machines
Use of a software defined network solves the data-in-transit issue that many certifications require.
23. Hardware securitymanager
•Easily integrates with IAM policies
•Centralized management of keys and certificates
•Easily and quickly encrypt customer data
Use of the HSM solves the data-at-rest issue that many certifications require.
25. TwilioCloudRequirements
•Services can be deployed anywhere
•Services can communicate anywhere
•Services can be discovered anywhere
Solving the issue of global service discovery is easy once the underlying cloud infrastructure is in place.
27. EC2-VPC Building Blocks
•Global routing tables
•HVM and SR-IOV
•Elastic Network Interfaces
•Software Defined Network
•Hardware Security Manager
28. Region-to-regionconnectivity
Performing routing among multiple VPCs in different regions is a bit more complicated and necessitates the use of a routing protocol.
router
router
us-east-1 / 10.1.0.0
us-west-2 / 10.2.0.0
vpc-abcdef
vpc-zyxwv
IPSEC tunnel
host
host
host
host
32. Which may look insignificant,but...
•A single global network
–Global servicediscovery
•Much easier call flow
–Easier to debug
–Less risk to deploy
–More frequent deploys
–Call setup latency down 25%
•Less infrastructure and complexity
33. Also…
•Blocking firewall rules
–Important for stopping attacks
•ENI
–Aid us in deploying new edge services
–Improved network performance
–Better audio quality
36. Migration Requirements
•Equivalent to moving a datacenter
–Zero downtime
–Bridge traffic between services in a region
–Easily discover services in EC2-Classic or EC2-VPC
37. Peering vs bridging
Peering is two VPCs talking in different regions.
Bridging is EC2-Classic and EC2-VPC in the same account talking in the same region.
vpc-bbb
vpc-aaa
vpc-aaa
classic
us-east-1
us-west-2
us-east-1
38. Migrating from EC2-Classic to EC2-VPC
•Use IP Tunnel Manager for bridging traffic
•Use software routers for peering traffic
•Use Service Discovery for discovering new services as they move
Make sure any services you want to move from EC2-Classic to EC2-VPC share the same AWS account and are in the same region!
40. •Services can be deployed globally
•Services can communicate globally
•Services can be discovered globally
•New VoIP infrastructure deployed in:
–all regions around the world
–taking live traffic for new products
–existing carrier traffic is being migrated
Where we are today
41. How could this have been easier?
•Feature to bridgeEC2-Classic and EC2-VPC
•Feature to connect VPCs in different regions
Are you listening,AWS? Maybe. :-)