Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Using OpenStack In a Traditional Hosting Environment
1. Using OpenStack
In A Traditional Hosting Environment
Jun Park (Ph.D.), Sr. Systems Architect
Mike Wilson, Sr. Systems Architect
EIG/Bluehost
OpenStack Summit 2013
2. EIG/Bluehost 2
• Scale @10x what we were in about a year’s time
• Needed a system to manage it all
• Provisioning these systems had to be automated
• System had to scale to multiple data centers
• Btw, we have 2 months to come up with this
3. High level requirements
• Centralized management, horizontal scalability.
• Abstractions for physical and logical deployments of
devices and resources.
• Open-source project with lots of support and momentum
• Typical Cloud features
(migration, imaging, provisioning, etc.)
Easy transition into future cloud offering with support for
industry standard APIs
=
OpenStack!
3EIG/Bluehost
4. Bluehost Environment
• Scale
– More than 10,000 physical
servers
– Adding hundreds of nodes
per day
• Network
– Public network directly attached to VMs
– Plan on adding private network later
• OpenStack/Folsom Components
– Nova (compute, api, scheduler)
– Quantum (network)
– AMQP: Qpid (messaging)
– Mysql (db)
4EIG/Bluehost
6. Why Difficult To Scale up?
• Components that don’t scale
– Messaging system
– Mysql server
– Heavy APIs
• Hard to Diagnose
– No simulator/emulator for high scalability testing
– No quantitative guide as to how to scale up
– No detailed error message or even not at all
– Requires detailed knowledge of codebase
6EIG/Bluehost
Look at that line!
7. Nova Enhancements
• Monitoring/troubleshooting
– Added service ping (similar to grizzly)
– Additional task_states
– Better errors to instance_faults
• Functionality
– Added LVM resize and injection
– Added stop_soft and stop_hard similar to reboot
• Stability/Scalability
– Some bug fixes to OVS VIF driver
– Added custom scheduler
7EIG/Bluehost
8. MySQL/Innodb Concurrency
• Nova behavior
– Direct connection to DB (addressed in grizzly though)
– Too MANY queries; much more read than write
– Some queries often return huge number of results
– Periodicity of loads due to periodic tasks
• Mysql behavior
– innodb_thread_concurrency = num of threads that can
execute queries in the queue
– innodb_concurrency_tickets = number of rows you can
return before requeue (default 500!)
– With default settings your mysql server will fall over with a
few thousand compute nodes
EIG/Bluehost 8
9. 9EIG/Bluehost
So the answer is just increase concurrency and tickets right?
Sadly, no.
Tuning concurrency and tickets is a must! But we still requeue.
We don’t know why, but we have a workaround.
Our workaround:
Send most read queries to a cluster of mysql slaves.
I say most because many queries would be sensitive to
possible slave replication lag
10. Messaging System
• Qpid Chosen
– Speed and easy cluster ability
– Incredibly unstable at large scale (Even at small scale)
– Possibly poorly configured
– Older release in CentOS6
• Problem
– Broker bottleneck
– Unnecessarily complex
• Recommendation
– AVOID BROKER!
– ZeroMQ as replacement
EIG/Bluehost 10
11. Server Farm
> 10,000 physical servers
BlueHost OpenStack
Group of Controllers
• Nova-api
• Nova-scheduler
• Quantum-server
• Keystone
Read-only mysql slave
Read-only mysql slave
Read-only mysql slaves
mysql master
Qpid servers
Load Balanced
EIG/Bluehost
Host
BH-enhanced Nova-
compute
BH-enhanced OVS
Quantum Plugin
12. Rethink Quantum Network
• Problem
– Quantum API only for DB; no efficient API for Nova
– No API-around design for actual network objects
– Premature OpenvSwitch
(OVS) quantum plugin
• Our approach
– Adding intelligence to OVS plugin
– Optimizing API
• E.g., get_instance_nw_info() in nova; Python-level join operation
– Expand current implicit communication with nova-
compute (although not ideal)
12EIG/Bluehost
13. OpenvSwitch (OVS)
• Support
– OpenFlow 1.0 (1.2, 1.3 in experiment)
– Various Hypervisors including KVM
– Merged into main stream since Linux 3.3
• Functionality
– Filtering rules and associated actions
• E.g., anti-IP spoofing, DMAC filtering
– Replacement or superset of ebtables/iptables/linuxbridge
– QoS or Linux traffic control (TC)
– Various third party controllers for full-fledged OF
functionality
13EIG/Bluehost
14. Controller
Host
Quantum With Nova-Compute
Quantum-server
Quantum-agent
Nova-compute
Quantum
DB
Nova-api server
2. Call quantum-api to create port
4. Create OVS tap
5. Set up external-ids on tap
1. Monitor any change of external-
ids of the existing taps
1. Nova boot API
3. Modify external-ids on tap &
setup OVS flows
2. Call quantum-api to get port info
3. Create port in DB
Why nova-compute
still create tap?
14EIG/Bluehost
6. Do the rest of steps
for creating a new VM
Implicit communication
with nova-compute
15. BH-Enhanced OVS Quantum Plugin
• Idiosyncratic Requirements
– Focus on a Zone; plan on addressing multiple-datacenters issue
– Direct public IP using flat shared provider network with No NAT
– Still required to be isolated while allowing intra-traffic among VMs
• Our Approach
– Developing “Distributed OpenFlow controller” using OVS plugin
– Either no-tag or 24bit ID (e.g., QinQ, VxLAN, GRE), but NO VLAN
– Caveats: Neither API-around approach nor Virtual Appliance
• New Features
– Anti-IP/ARP spoofing OF flows
– Multiple-IPs per public port
– Network bandwidth throttling (QoS)
– Optimal Intra-traffic flows
15EIG/Bluehost
16. BH-Enhanced OVS Quantum Plugin
br-int
tap1
tap2
br-int-eth0
br-eth0
phy-br-eth0
eth0
pair of veth
• DMAC matching for incoming
packets
• TPA matching in ARP query
• Anti-IP
spoofing: SRC
IP matching for
outgoing
packetsVM1
VM2
Public Networks
• Deploy QoS for
outgoing packets
10 Mbps
50 Mbps
Int-loopback phy-loopback
pair of veth
• DMAC matching for
internal traffic to VMs
• DMAC matching for
internal traffic to VMs
16EIG/Bluehost
Tag=1 (MTU size bug)
Tag=2
17. Operational Issues
• Reboot hosts
– Problem
• Circular dependencies between libvirtd and nova-compute
• OVS bug, allows to add non-existing tap interfaces
– Solution
• A simple workaround to restart services in rc.local
• Restart services
– Problem
• Quantum recreates taps, resulting in network hiccups
– Solution
• Check out existing taps, no touch when unnecessary
17EIG/Bluehost
18. Operational Issues
• Monitor Health
– Problem
• Hard to track down
– Solution
• Adding health check APIs
• XML customization
– Problem
• No way to modify XML on the fly
• Not even allowed for static customization
– Solution
• Add more parameters
18EIG/Bluehost
20. Problem vs. Solution
EIG/Bluehost 20
Problem Solution
Monitoring/troubleshootin
g
Service ping, task_states, etc.
Nova
No LVM
Overloaded scheduler
Add LVM driver
Custom scheduler
OVS VIF driver issue
Mysql
Fix bugs
Overloaded Read-only Mysql slave server
Innodb issue Optimized confs
Qpid
Instability issue Clustered qpid
Future plan No broker, ZeroMQ
Quantum
Heavy API Optimized API
Premature OVS plugin Add features (Anti-IP, No-tag flows, QoS)
Operations
Reboot, restart of services Simple workarounds
XML XML customization (cpu custom)
21. For True OpenStack Success
Scalability
Scalable Messaging System & DB Infra
Networks
Good Networks Abstraction
So … We don’t do live demo until …
21EIG/Bluehost
22. We’re sorry, we haven’t had time to contribute a whole lot YET. But here’s
some code:
https://github.com/upoopoo for BH nova
https://github.com/JunPark for BH Quantum (as in live production)
Browse the branches, many have to do with features discussed today
22EIG/Bluehost
Editor's Notes
Love!30mins talk + 10 minsqna during the main talk we have short answers, have longer answers in qna. May have off-line chats for more details.Take a quick poll: How many people with openstack and cloudstack?Time: Tell our story… main tasks remaining …
Quick poll, how many nodes?
Quick poll : how many nodes?
LOVE!!!Just a small step for us to try to help community and share what we’ve done.
Quick poll: how many people know about OVS plugin? How many people use OVS in production?
The main motivation of quantum network is to pull out network functionality from nova-compute as much as possible. Because of this, nova-compute still needs to know all the details of OVS. No code reuse, confusing, and causing inconsistency, succinct failures when VMs do not have network connectivity. With some garbled taps, quantum-agent thinks it needs to do something even for them.
Let me introduce our BH-enhanced OVS plugin. NO NAT, WHY? Avoid single point of failure, easy to diagnose, meet some specific software requirement such as cPanel.
Vlan tagging No tag or double-tag or VxLAN whatever…Avoiding O(n^2) solution….
Let’s get this straightened out!Structurally means most distributed structure among all other cloud platforms.