Mais conteúdo relacionado Semelhante a Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月 (20) Mais de VirtualTech Japan Inc. (20) Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月1. Mellanox for OpenStack
Senior System Engineer | Mellanox Technologies Japan KK
友永 和総 (Kazusa Tomonaga)
2014年10月8日 – OpenStack最新情報セミナー
※本内容は予告なく変更となる場合があります。
2. Agenda
1. Mellanox Overview
2. Mellanox CloudX
3. CloudX Deep Dive
• Network Virtualization Acceralation – VXLAN NICハードウェアオフロード
• High Speed Storage Interconnect – iSER (iSCSI over RDMA)
• SR-IOV / NFV
7. Cloud Application Performance
8. Demo
9. まとめ
© 2014 Mellanox Technologies 2
4. メラノックス会社概要
Ticker: MLNX
高帯域、低レイテンシーなサーバ・ストレージ間インターコネクトのリーディングカンパニー
• FDR 56Gb/s InfiniBandと10/40/56ギガビットEthernet
• アプリケーションのデータ処理時間を大幅に削減
• データセンタ・サービス基盤のROIを劇的に向上
会社概要
• 本社:イスラエルYokneam, 米国Sunnyvale, California
• 従業員数:全世界で1,428名(2014年6月末時点)
堅実な財務内容
• 2013年度売上 : $390.9M
• 2014年度第二四半期売上 : $102.6M
• 2014年度第三四半期売上 : $114M~$118M
• Cash + investments @ 6/30/14 = $343.7M
© 2014 Mellanox Technologies 4
5. Exponential Data Growth – The Best Platforms Are Needed
We Live in a World of Data
More Devices More Applications More Data
Data Needs to be Accessible Always and in Real-Time
© 2014 Mellanox Technologies 5
6. Leading Supplier of End-to-End Interconnect Solutions
Analyze Store
Enabling the Use of Data
Comprehensive End-to-End InfiniBand and Ethernet Portfolio
ICs Adapter Cards Switches/Gateways Software and Services Metro / WAN Cables/Modules
At the Speeds of 10, 40 and 100 Gigabit per Second
© 2014 Mellanox Technologies 6
7. メラノックスのコアテクノロジー : 高性能・高集積ASIC
VPI (Virtual Protocol Interconnect) テクノロジー
• IB and EN with single chip (ConnectX-3、SwitchX-2)
• IB and EN port by port (ConnectX-3、SwitchX-2)
• IB/EN Bridging (SwitchX-2)
高スループット、低レイテンシー、超低消費電力 (Ultra Low Power)
RDMA (Remote Direct Memory Access) 対応、高速データ転送
VXLAN/NVGREオフロード (ConnectX-3 Pro)
Typ power: 7.9W
3.0 x8
17mm
2 x IB FDR (56Gbps)
3.0 x16
45mm
2 x 56Gbps
Ethernet mode:
1/10/40/56GbE
InfiniBand/Ethernet
InfiniBand/Ethernet
144組のネットワークSerDesを搭載
36 x 40/56GbE
64 x 10GbE
48 x 10GbE + 12 x 40/56GbE
Ethernet mode:
1/10/40/56GbE
• InfiniBand or Ethernet
• InfiniBand + Ethernet
• InfiniBand / Ethernet Bridging
36x 40GbE: 83W
64x 10GbE: 63W
2pt 40GbE (100% load power)
© 2014 Mellanox Technologies 7
8. メラノックス製品の3つの特長
1) 高スループット
アダプタ/スイッチ - 10/40/56Gbps
2) 低レイテンシー
スイッチ : 220ns (L2 40GbE), 330ns (L3 40GbE)
アダプタ : ~1us
3) 超低消費電力
スイッチ : 83W (36 x 40GbE 100% load)
アダプタ : 7.9W (2x 40GbE typ. Power)
エンドトゥエンドソリューション:
アダプタ向けASIC、スイッチ向けASICを自社開発(フルカスタムASIC)
アダプタ、スイッチ、ケーブルを自社開発
• 100Gbps実現に向けた光モジュール/ケーブル関連技術を自社で保有
(2013年、Kotura社とIPtronics社を買収)
ドライバ、高速化ソフトウェア、管理ソフトウェアを自社開発
© 2014 Mellanox Technologies 8
10. Making the Cloud Easy
The World is Moving to the Cloud
But Building a Cloud is a Challenge
Private Cloud Public Cloud
Mellanox Makes it Easy: Deploy, Use and Maintain
© 2014 Mellanox Technologies 10
11. CloudX is the Most Efficient Cloud Platform
Lower Your IT Cost by 50%!
The Platform for Creating the Applications of Tomorrow
© 2014 Mellanox Technologies 11
12. CloudX: Optimized Cloud Platform
CloudX is a group of reference architectures which
allow building the most efficient, high performance
and scalable Infrastructure As A Service (IaaS)
clouds based on Mellanox superior interconnect and
off the shelf building blocks
Supports the most popular cloud software
• OpenStack
• Windows Azure Pack (WAP)
• VMware
© 2014 Mellanox Technologies 12
13. Mellanox OpenStack Reference Documents
http://www.mellanox.com/openstack/
Designing CloudX Solution using Mirantis
Fuel OpenStack Software
http://community.mellanox.com/docs/DOC-1464
HowTo Configure iSER Block Storage for
OpenStack Cloud with Mellanox ConnectX-3
Adapters
http://community.mellanox.com/docs/DOC-1462
今後、随時拡充されていく予定です。
© 2014 Mellanox Technologies 13
14. Mellanox CloudX
• メラノックスが最先端の大規模クラウドシステムでグローバルに培ったテクノロジーを
• リファレンスアーキテクチャのドキュメントを無償提供、プロフェッショナルサービス
• 世界最先端のクラウドシステム構築を実現
• エンドトゥエンドの高性能ネットワークを実現する
ハードウェア
• Mellanox ネットワークアダプタ
• Mellanox ネットワークスイッチ
• Mellanox ネットワークケーブル
• ハードウェアサポートサービス
• CloudXシステム構築プロフェッショナルサービス
• CloudXシステム運用・保守テクニカルサポート
パッケージ化
メニューも用意
© 2014 Mellanox Technologies 14
15. Mellanox CloudX ハードウェア : Ethernetスイッチ
SX1036
The Ideal 40GbE ToR/Aggregation
SX1024
Non-blocking 10GbE 40GbE ToR
SX1016
Highest density 10GbE ToR
SX1012
Ideal storage/Database 10/40GbE Switch
Highest Capacity in 1RU
• 12 to 36 QSFP
• 64 x 10GbE
Value
Latency
Power (SX1036)
• VPI 56Gb/s InfiniBand & Ethernet
• End to end solution
• 220ns L2 latency
• 330ns L3 latency
• Under 1W per 10GbE interface
• 0.6W per 10GbE of throughput
SX1036 – 83W
SX1016 – 62W
SX1024 – 75W
SX1012 – 50W
For 100% L oad Power
© 2014 Mellanox Technologies 15
16. Shattering Ethernet Switching Performance Records
Leading throughput: 2.5X better
• 2.88Tb/s throughput on a single chip, running Full Wire Speed at any packet size
Leading L2 unicast/multicast latency for L2/L3 switches: 2X better
• 198-223ns for any packet size
Leading L3 latency: 2X better
• 321-337ns for any packet size
Industry record power efficiency: 6X better than competition
• Sub 0.6Watt per 10GbE throughput with 100% load at Full Wire Speed
‘Zero’ Jitter!
206 209 220 216 219 223 222 222 223
1,000
800
600
400
200
0
64 128 256 512 1,024 1,280 1,518 2,176 9,216
L2 Min
L2 Average
L2 Max
Packet Size (Bytes)
Latency (ns)
© 2014 Mellanox Technologies 16
17. Best ROI – Switch Silicon Example
What matters in switch silicon?
• Highest switching capacity
• Lowest power
• Lowest latency
x1.7 Better x2.5 Better x5 Better
Switching Capacity (Bpps)
0.95
1.44
2.5
1.2
1
0.8
0.6
0.4
0.2
Latency (μsec)
3.5
3
2.5
2
1.5
1
0.5
Power (Watt/Gb)
© 2014 Mellanox Technologies 17
3
2.5
2
1.5
1
0.5
0
Trident+ Trident2 SwitchX
1
0.5
0.2
0
Trident+ Trident2 SwitchX
3
2
0.4
0
Trident+ Trident2 SwitchX
18. Mellanox CloudX ハードウェア : Ethernetネットワークアダプタ
• 10/40/56ギガビットEthernetサポート
• HPCシステムでも広く活用される世界トップレベルの性能
• 低レイテンシー、高スループット、超低消費電力
• RDMA (Remote Direct Memory Access) サポート
• RoCE (L2 Ethernet RDMA)
• RoCEv2 (L2/L3 Ethernet RDMA)
• オーバーレイネットワークオフロード
• VXLAN (Linux, VMware*)
• NVGRE (Windows) *Available soon
The Foundation of Cloud 2.0
The World’s First NVGRE / VXLAN Offloaded NIC
© 2014 Mellanox Technologies 18
19. Mellanox CloudX プロフェッショナルサービス
Mellanox CloudXテクノロジーを活用した200ノードまでのクラウドシステム構築を実現する技術
サービス
• Mellanox CloudX Technical implementation package up to 200 nodes
• CloudX Planning and design, Installation, Configuration, Performance Tuning, Testing
and Knowledge Transfer
GPS-0200-ONST-CLOUDX
オンサイトサポート
GPS-0200-REMT-CLOUDX リモートサポート
※価格等、詳細は個別にお問い合わせください。
© 2014 Mellanox Technologies 19
20. Mellanox CloudX テクニカルサポート
Ordering P/N Description Comments
SUP-CLOUDX-1S CloudxX™ System Support
1 YEAR SILVER SUPPORT
Includes support for CloudX™ operational assistance, including
integrated components for Mellanox Opensack plugins: Neutron
Mellanox Plug-in, eswitchd, Cinder over iSER and/or other
CloudX supported architecture plug-in’s operational assistance,
Mellanox OFED CloudX™ operational assistance
下記のようなMellanox CloudX使用にあたって必要となる
コンポーネント群の運用についての技術サポートを提供
・Mellanox Neutron Plug-in
・Mellanox eSwitchd (Embedded Switch)
・Mellanox iSER for Cinder
・Other CloudX components
・MLNX OFED
SUP-CLOUDX-3S CloudxX™ System Support
3 YEAR SILVER SUPPORT
SUP-CLOUDX-1G CloudxX™ System Support
1 YEAR GOLD SUPPORT
SUP-CLOUDX-3S CloudxX™ System Support
3 YEAR GOLD SUPPORT
※価格等、詳細は個別にお問い合わせください
© 2014 Mellanox Technologies 20
22. Next Generation Software Defined Networks
EVN: Efficient Virtualized Network
Fully Integrated Solution Combines
RDMA, Convergence, & SDN/Virtualization
Efficient Virtualized Network (EVN)
Fully Integrated, World Class SDN Solution
VIRTUALIZATI ACCELERATION CONVERGENCE
ON
EVN: More than SDN - Efficient Clouds Need an Efficient Virtualized Network
© 2014 Mellanox Technologies 22
23. Accelerate All Three Elements Required for SDN Networks
= + +
Software Defined Networks Virtual Network Management Overlay Network Tunnels OpenFlow
1.Centralized Software Based Control Plane
• Enables network virtualization
2.Overlay Networks – NVGRE/VXLAN
• Isolation, Scalability, Simplicity
• Mellanox accelerates overlay networks to offer bare metal speed
3.Industry Standard API – OpenFlow
• Enables an industry ecosystem and innovation
© 2014 Mellanox Technologies 23
24. Comprehensive OpenStack Integration for Switch and Adapter
Integrated with Major
OpenStack
Distributions
In-Box With Havana &
Ice House
Neturon-ML2
support for
mixed VM
environment
(VXLAN, PV,
SRIOV)
Ethernet
Neutron :
Hardware
support for
security and
isolation
Accelerating
storage
access by up
to 5x
OpenStack Plugins Create Seamless Integration, Control, & Management
© 2014 Mellanox Technologies 24
26. ConnectX-3 Pro Accelerates Overlay Networks
VM1 VM2 VM3 VM4
Server
VXLAN Overlay Networks
Virtual Domain 1
Virtual Domain 2
Virtual Domain 3
OpenFlow
Virtual Network
Management API
Virtual
View
Physical
View
Virtual Overlay Networks Simplifies
Management and VM Migration
ConnectX-3 Pro
Overlay Accelerators Enable
Bare Metal Performance
VM5 VM6 VM7 VM8
Server
Mellanox SDN Switches & Routers
Overlay Network Virtualization: Isolation, Simplicity, Scalability
© 2014 Mellanox Technologies 26
27. VXLAN Performance
VXLAN Offload Engine – 5X higher throughput, 75% lower CPU utilization
CPU Usage Per Gbit/sec with VxLAN
4.50
4.00
3.50
3.00
2.50
2.00
1.50
1.00
0.50
0.00
1 VM 2 VMs 3 VMs
CPU% / Bandwidth (Gbit/sce)
Lower is Better
VxLAN in software 3.50 3.33 4.29
VxLAN HW
0.90 0.89 1.19
Offload
Total VM Bandwidth when using VxLAN
25
20
15
10
5
0
1 VM 2 VMs 3 VMs
Bandwidth [Gbit/sec]
Higher is Better
VxLAN in software 2 3 3.5
VxLAN HW Offload 10 19 21
© 2014 Mellanox Technologies 27
28. Turbocharge Your OVS with Mellanox ConnectX-3 Pro
Source: PLUMGrid white paper
“Mellanox ConnectX-3 Pro card is the only way to scale-out PLUMgrid’s
Virtual Network Infrastructure (VNI) overlay-based Architecture”
© 2014 Mellanox Technologies 28
30. iSER – iSCSI Extension for RDMA
Zero copy using RDMA
IB and Ethernet (RoCE)
Transport protocol implemented in hardware (minimal CPU cycles per IO)
Open stack integration
Support for T10/DIF
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
iSCSI/TCP iSCSI/RDMA
IO Latency @ 4K IO [micsec]
iSCSI (TCP/IP)
1 x FC 8 Gb
port
4 x FC 8 Gb
port
iSER 1 x
40GbE/IB Port
iSER 2 x
40GbE/IB Port
(+Acceleration)
2500
2000
1500
1000
500
0
KIOPs 130 200 800 1100 2300
K IOPs @ 4K IO Size
5-10% the latency under 20x the workload
© 2014 Mellanox Technologies 30
31. Mellanox Accelerates Storage: More than 4X Greater Throughput
OpenStack Storage Performance *
1.3
5.5
6
5
4
3
2
1
0
iSCSI over TCP iSER
GBytes/s
* iSER patches are available on OpenStack
branch: https://github.com/mellanox/openstack
Built-in OpenStack Components/Management &
Cinder/iSER to Accelerate Storage Access
© 2014 Mellanox Technologies 31
33. Mellanox Single Root I/O Virtualization (SR-IOV)
PCIe device presents multiple instances to the OS/Hypervisor
Enables Application Direct Access (ADA)
• Reduces CPU overhead and improves application performance
Eliminates virtualization penalty with RDMA & ADA
• Low latency applications benefit from the Virtual infrastructure
VM
VF Device Driver
VM
VF Device Driver
VMn
VF Device Driver
VM
Virtual NIC
Physical Function Device Driver
PF VF VF VF
© 2014 Mellanox Technologies 33
34. SR-IOV Boosts Ethernet Performance
SR-IOV Accelerates RoCE
• Enables native RoCE performance in virtualized environments
40
35
30
25
20
15
10
RoCE – SR-IOV Throughput
1 VM 2 VM 4 VM 8 VM 16 VM
Throughput (Gb/s)
Throughput (Gb/S)
1 VM 2 VM 4 VM 8 VM
No Performance Compromise in Virtualized Environment
3
2.5
2
1.5
1
0.5
0
© 2014 Mellanox Technologies 34
Latency (us)
RoCE - SR-IOV Latency
Message Size 2B Message Size 16B Message Size 32B
35. SR-IOV and eSwitch
VM
OS
tap tap SR-IOV
Para-virtual
VM
OS
VM
OS
VM
OS
to the
VM
Embedded
Switch
Mellanox
Neutron
Agent
Create/delete,
configure policy
per VM vNIC
OpenStack Manager
Neutron
Plug-Ins
Servers
Manager
Compare eSwitch vs OVS
Qperf (TCP) Latency
Provision VM & Fabric Policy in hardware, through standard APIs
Benefits: Isolation, functionality, performance & offload, simpler SDN
© 2014 Mellanox Technologies 35
36. ConnectX-3 Family QoS
Port based ETS
• Max bandwidth per TClass
• Bandwidth reservation per TClass
Per Function rate limiter
• Average bandwidth
• Peak bandwidth
• Maximum burst at peak bandwidth
Packet pacing
• Low jitter packet pacing for work
queues
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Rate Shaper Rate Shaper
Priority 0
Arbiter
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Priority 1
Arbiter
Priority ...
TC2
Flow
Ctrl
TC Group 1
DWRR
Strict Priority
Flow
Ctrl
TC Group 0
DWRR
TC Group 7
DWRR
TC0
TC1
Flow
Ctrl
TC3
Flow
Ctrl
© 2014 Mellanox Technologies 36
TC7
Flow
Ctrl ...
L H
...
0
Priority
1
Priority
2
Priority
3
Priority
7
Rate Limiter
Rate Limiter
Function Rate Limiter Enhanced ETS Packet Pacing
37. Record Breaking 195Gbs for Guest VM over DPDK
High performance 10/40/56Gbps
SR-IOV enabled
OpenFlow enabled eSwitch
OpenStack Neutron Plug-in
PMD for DPDK: VM OS bypass
Multi cores and RSS support
Delivering bare-metal performance
Hypervisor
Legacy Software
vSwitches
Neutron Plug-in
OpenStack integration
VM
OS
VM
OS
High-performance packet processing solutions for
• Gateways
• Security appliances
• UTMs
• Virtual appliances
• etc.
• Data Plane libraries
• Optimized NIC drivers
Client’s Application Software
librte_pmd_mlx4 librte_crypto_nitrox VMware … 6WIND addons
……
Multicore Processor
SR-IOV eSwitch
Hardware Offload
VM
6WIND or Intel® DPDK
OpenFlow enabled
10/40/56Gbps
© 2014 Mellanox Technologies 37
38. Remote HWA as a Service in NFV Cloud Model
Fat-tree SDN Switch Network
40GbE
56Gbps
IB FDR
Ethernet Ethernet Ethernet
Platform 1
40Gbps
DPI
BRAS
SGSN
GGSN
PE Router
STB
Firewall
RDMA / RoCE
CG-NAT SBC
RDMA / RoCE
RDMA / RoCE
Nx40Gbps Nx40Gbps Nx40Gbps
SX1024 Ethernet Switch
Fabric
HWA /
Signal
Processing
40Gbps
Platform 2
SX1024 Ethernet Switch
Fabric
HWA /
Signal
Processing
40Gbps
Platform X
SX1024 Ethernet Switch
Fabric
HWA /
Signal
Processing
© 2014 Mellanox Technologies 38
39. iSCSI SAN/NAS Storage Architecture in an NFV Cloud Model
Fat-tree SDN Switch Network
Aggregation
ToR
Compute Storage
10/40/100Gbps
Compute Storage
10/40/100Gbps
RDMA / RoCE
RDMA / RoCE
Rack 1 Rack 2 10/40/100Gbps
12x10/40/100Gbps 12x10/40/100Gbps 12x10/40/100Gbps
Ethernet Switch
SAN/NAS Storage
Compute Storage
10/40/100Gbps
10/40/100Gbps
Ethernet Switch
SAN/NAS Storage
10/40/100Gbps
Rack n
RDMA / RoCE
Ethernet Switch
SAN/NAS Storage
iSCSI SAN/NAS Storage over Standard Ethernet Network: Shared Resource
© 2014 Mellanox Technologies 39
41. CloudX Delivers Unbounded Cloud Performance
4X Faster Runtime!
Benchmark: TestDFSIO (1TeraByte, 100 files)
2X Higher Performance!
Benchmark: 1M Records Workload (4M Operations)
2X faster runtime and 2X higher throughput
2X Faster Runtime!
Benchmark: MemCacheD Operations
3X Faster Runtime!
Benchmark: Redis Operations
© 2014 Mellanox Technologies 41
42. Accelerating Cloud Performance
SCSI Write Example, Linux KVM
6200
1200 800
8000
6000
4000
2000
0
I/O Size - 64 [KB]
Bandwidth
[MB/s]
iSER 16 VMs Write
40GbE VMs Write
10GbE
Fiber Channel - 8Gb
Migration of Active VM
38
10
50
40
30
20
10
0
Time [s]
Fibre 10 GbE
40 GbE
10GE-A 40GE-A
Storage
Migration
Virtualization
6X
3.5X
20X
VM-to-VM Latency Performance
40
50
40
30
20
10
2 0
Message Size - 256 [bytes]
Latency [us]
TCP ParaVirtualization
RDMA Direct Access
© 2014 Mellanox Technologies 42
46. まとめ – Mellanox CloudXで効率の良い先進クラウドが構築可能
• メラノックスが最先端の大規模クラウドシステムでグローバルに培ったテクノロジーを
• リファレンスアーキテクチャのドキュメントを無償提供、プロフェッショナルサービス
• 世界最先端のクラウドシステム構築を実現
• エンドトゥエンドの高性能ネットワークを実現する
ハードウェア
• Mellanox ネットワークアダプタ
• Mellanox ネットワークスイッチ
• Mellanox ネットワークケーブル
• ハードウェアサポートサービス
• CloudXシステム構築プロフェッショナルサービス
• CloudXシステム運用・保守テクニカルサポート
パッケージ化
メニューも用意
© 2014 Mellanox Technologies 46