Openstack@ebay: Practical SDN deployment with Quantum
1.
2. Prod
Prod
QA DEV
PCI
Secure
DEV
QA QA DEV
Copyright eBay Inc. 2012 2
3. Any Application Anywhere
Dedicated physical environments cause fragmentation
Soft Cabling
Datacenter reconfiguration is costly and cannot be automated
Shared Standardized Infrastructure
Simplifies automation and improves supply chain efficiency
Virtualize everything
White space between applications and infrastructure helps agility
Automate everything
Automation helps agility and efficiency
Copyright eBay Inc. 2012 3
4. • Translation of physical environment properties into configurations
• Assigned to projects (logical environments), drives scheduling and policies
• For example, network selection
Production DEV
Obligations Restrictions Capabilities Obligations Restrictions Capabilities
QA Approved Builds No Login Access Core DB access Certified OS versions Limited Prod Full root
Access
Prod OS version No Corp Access 24/7 Incident Mgt
Limited QA Access
Monitoring No QA Access Site traffic Access
No site Traffic Filtered Internet
External
Obligations Restrictions Capabilities
No Prod Access Private DB
Certified OS Versions No Corp Access 24/7 Incident Mgt
Monitoring No QA Access Site traffic Access
Copyright eBay Inc. 2012 4
5. Core
4 spines
(Nx10Gb)
Spine
N leaves
(48x1Gb)
Leaves
48 -> N “½ racks”
M servers
2x1Gb
Flat L3 (all switches are routers too)
Line rate from any server to any server (oversubscription = 48/40)
OSPF/ECMP to advertise routes
Copyright eBay Inc. 2012 5
6. Dedicated Network VLAN Based
VLAN trunk
vlan 1
Prod
QA vlan n
Production QA
- physical network build out + Physical isolation - Limited scale (n = 4096) + L2 isolation
- Fragmentation + fool proof - Large fault domain (STP) + somewhat soft Cabling
- coarse grained isolation
Copyright eBay Inc. 2012 6
7. Security Groups or Virtual Firewall
+ no/minimal infrastructure requirement - Difficult to combine provider policies and user policies
+ good for user policies (ip tables) - Management of rules
- Impact of group membership modification
- Aggregation/summarization difficult/impossible
Copyright eBay Inc. 2012 7
8. Virtual Networks using Software Defined Networks
Overlay 1
Prod
Other
Networks
QA
Overlay n
Cloud Fabric
+ L2 isolation + Can complement L3 isolation
+ compatible with large scale + large number of networks (n>4096)
+ can be fully automated - Tunnel overhead
+ firewall can be interposed between - L2 size limited by # of tunnels
virtual networks
Copyright eBay Inc. 2012 8
9. Traditional SDN
The The
Network Network
Network protocols
Network protocols
Routing/switching engine Routing/switching engine
controls The Switch/Router
controls
Logic
Logic API
The Switch/Router Controller
Copyright eBay Inc. 2012 9
11. A logical environment defined as a class of service on top of shared infrastructure
Self Service VM for developers.
Access must be similar to their desktops (access to QA, Corp, …)
Should allow collaboration
Implemented as a set of L2 networks (/24) with in a given L3 (/20)
No private networks : all developers on same shared networks
No private IP space: traffic is routed within core, no need for floating Ips
Isolated from infrastructure
Overlay network using OpenVswitch / STT tunneling
Nicira NVP controllers integrated with Quantum (Essex)
Routed out through perimeter firewall
Copyright eBay Inc. 2012 11
12. From 10.9.1.0/24 default->10.9.0.1 10.9.0.0/20 ->10.9.0.10
From 10.9.2.0/24 default->10.9.0.1
Standby Gateway
Eth1/vlan 1
Dev Cloud : 10.9.0.0/20
Eth0/vlan 2 Corp
10.9.1.0/24 10.9.1.1 N
gtw-xxxx
trunk
gtw-xxxx 10.9.0.10 10.9.0.1
Internet
10.9.2.0/24 N
M
10.9.2.1
gtw-xxxx
QA
vswitch M Eth1/vlan 1
Eth0/vlan 2
vswitch
Nicira
default->10.9.2.1 Nicira Nicira
Active Gateway Service Nicira
Service controllers
Nodes controllers
Nodes
vif
K C Hypervisor S A Q
N:Nova-network+dnsmasq K:Ubuntu + KVM
vswitch C:Nova-compute A:Nova-api
S:Nova-scheduler Q:Quantum
M:Metadata
Infrastructure/Internal Virtual network
Infrastructure/External
Copyright eBay Inc. 2012 12
13. Developer Admin
Create network
(project = admin, Create routes
eBay Cloud Portal Cidr=10.9.x.0/24)
Create instance
1 (COS,OS, size)
Nova-manage Gateway
2 Get Free Networks
eBay IaaS
Create DNS Boot Instance Nova Network
(A,PTR) (Image ID,Flavor, NIC)
Create
4 3 gtw-xxxx
DNS Nova API Quantum
nova
Management db
Create Create
Nova Scheduler port lswitch
13
Get IP
Create port Nicira Controller
Nova Compute
Copyright eBay Inc. 2012
15. Perimeter firewalls configured once, not No capacity/policy based assignment of
dependent on the instance networks – had to be implemented outside.
creation/deletion/movement Moving it to nova scheduler.
Network are pre-created using nova- One network flavor supported in Essex.
manage, good for provider networks
Cannot have, e.g., one gateway per
network, with different behavior (dhcp)
Can be extended with other COS using same
pattern
Scale out requires bigger links out of the
gateway, or more gateways
Stability of both Nicira NVP and Openstack +
Ubuntu + KVM
Upset the separation of concern
Looking forward to new features in Folsom – requirement: Netsec + Networking + Sys
Quantum v2 Admins in same box = ‘interesting’
15
16. New classes of service
External : private networks + VIP and Floating IP on the Internet
Production : Bridged network
Scale out
80 today, going to a lot more
More gateways/10Gb
Folsom upgrade
L3 Routers
Load Balancers
Cleaner Openstack integration
Network Allocation
DNS configuration
AuthN/AuthZ
16
17. We are Hiring !
http://www.ebaycareers.com/
Copyright eBay Inc. 2012 17
Notas do Editor
Add a title to all presentations. A subtitle is optional.
L3 rules are configured in either A firewall appliance or the hypervisor