More Related Content Similar to Managing Heterogeneous Cloud Infrastructure with OpenStack, Alex Glikson & Ezra Silvera (20) More from Cloud Native Day Tel Aviv (20) Managing Heterogeneous Cloud Infrastructure with OpenStack, Alex Glikson & Ezra Silvera1. 1
Managing Heterogeneous Cloud Infrastructure
with OpenStack
Example:
Hosting ‘Hybrid’ Applications Comprising Bare-Metal and Virtualized Resources
OpenStack Israel, June 2nd, 2014, Hertzliya
IBM Research – Haifa
Alex Glikson
Ezra Silvera
2. 2 © 2014 IBM Corporation
Outline
Heterogeneous Clouds
‒ Background & Motivation
‒ Opportunities and Challenges
Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications
‒ Goals
‒ Scenarios
‒ Solution Approach
‒ Technical Challenges and Design Considerations
‒ Example
Summary, Q&A
4. 4 © 2014 IBM Corporation
Evolution of IaaS Usage Patterns and Offerings
Large variety of configurations
‒ E.g., AWS now offers >60
instance configurations
Rise of bare-metal providers
‒ IBM/SoftLayer, Internap, etc
Small set of
standardized (virtual)
hardware configurations
‒ e.g., 1 instance type
in AWS EC2 (2006)
Commonly
Offered
Hardware
Configurations
Growing spectrum of workloads
‒ E.g., ‘legacy’ business
applications, HPC, analytics, etc
Dev & test
Web applications
(‘cloud-native’)
Common
Usage Patterns
Agility, efficiency, elasticity, self-service, pay-per-useValue
proposition
NowEarly Days
Trend: Increased Cloud Infrastructure Heterogeneity (including bare-metal)
to Support Larger Variety of Applications with Special
Requirements
5. 5 © 2014 IBM Corporation
Embracing Infrastructure Heterogeneity
for the Benefit of the Application: Example
Virtualization
Network
Storage
Compute
CPU/mem/disk
balance
CPU-rich mem-rich disk-rich …
CPUs GPGPUs Accelerators
Local HDD SSD SAN NAS
…
…1 Gbit shared 10 Gbit shared 10 Gbit reserved
…
bare-metal KVM LXC VMware …
Tier-1: web app
Tier-2: 3D rendering
Tier-4: Designs DB Tier-3: Hadoop
balanced
Example: Application for Collaborative 3D Design
6. 6 © 2014 IBM Corporation
Embracing Infrastructure Heterogeneity
for the Benefit of the Application: Example
Virtualization
Network
Storage
Compute
CPU/mem/disk
balance
CPU-rich mem-rich disk-rich …
CPUs GPGPUs Accelerators
Local HDD SSD SAN NAS
…
…1 Gbit shared 10 Gbit shared 10 Gbit reserved
…
bare-metal KVM LXC VMware …
Tier-1: web app
Tier-2: 3D rendering
Tier-4: Designs DB Tier-3: Hadoop
balanced
Example: Application for Collaborative 3D Design
4
4
4
4 3
3
3
3
2
2
2
2
2
3
41
1
1
1
1
8. 8 © 2014 IBM Corporation
Opportunities and Challenges
Business opportunities enabled by infrastructure heterogeneity
‒ Service/application providers:
• Cloud adoption in ‘traditional’ market segments agility and efficiency
– E.g., manufacturing, finance, scientific, etc
– IDC: the (extended) HPC market alone is larger than IaaS ($15B vs $9B)
‒ Cloud/infrastructure providers:
• Ability to better fit to specific classes of applications differentiation
– Especially relevant given the overall commoditization of the IaaS market,
recent price wars, etc
9. 9 © 2014 IBM Corporation
Opportunities and Challenges
Challenge: cloud infrastructure natively designed for:
‒ Underlying infrastructure heterogeneity
‒ Ability to seamlessly host applications with special requirements
‒ Still preserve the characteristics of the cloud model
• Agility, efficiency, elasticity, etc
10. 10 © 2014 IBM Corporation
Heterogeneous Cloud Requirements: Cloud User
Cloud user perspective
‒ Access to a wider range of offered HW/SW configurations
• Performance, security/isolation
• Compute, storage, network
‒ Same or better user experience as with ‘regular’ cloud, e.g.:
• Same: single/unified catalog of configuration ‘flavors’, images, etc
• Same: seamless network connectivity, storage access, etc
• Same: higher-level services (orchestration, LBaaS, DBaaS, etc)
• Better: higher-level application abstraction, hiding the details of the
underlying HW (critical with increased complexity of underlying HW)
– Specify goals/requirements, rather than specific resource types/models etc
11. 11 © 2014 IBM Corporation
Heterogeneous Cloud Requirements: Cloud Provider
Cloud provider perspective
‒ Minimize management cost by unified processes and APIs
• Provisioning, inventory, monitoring, life cycle operations, visualization, etc
– Unified resource model
‒ Secure multi-tenancy across heterogeneous resources
‒ Elasticity & efficiency
• Smart scheduling, to overcome resource fragmentation
• Dynamic balancing between HW-compatible pools by “repurposing”
(subject to compatibility)
– Capacity monitoring & prediction
– Automated evacuation & re-provisioning
12. 12 © 2014 IBM Corporation
Heterogeneous Cloud: OpenStack Perspective
OpenStack is following a similar pattern
‒ Started primarily with dev & test and cloud-native applications
• Simplistic approach to flavors, scheduling, performance optimization, etc
‒ Extensive efforts to enable adoption by additional applications, e.g.:
• Hadoop (Sahara)
• DBaaS (Trove)
• Bare-metal (Ironic, TripleO, Tuskar)
• Scheduling (Gantt)
• Reservations (Climate)
• Policies (Congress)
• Etc
‒ Growing community interest in supporting heterogeneous clouds, e.g.:
• Multi-Tenant Bare Metal Provisioning (Mon, 11:15, B206; Tue, 2:50pm, B301)
• Deploying OpenStack in a Multi-Hypervisor Environment (Wed, 11am, B101)
• IBM, SoftLayer and OpenStack - Present and Future (Wed, 11:50am, B312)
• Many more related sessions…
13. 13 © 2014 IBM Corporation
Heterogeneous Cloud: OpenStack to the Rescue!
Claim: OpenStack provides a good basis for
building heterogeneous clouds
• But few gaps yet to be addressed
14. 14 © 2014 IBM Corporation
Hosting ‘Hybrid’ (Bare-Metal + Virtualized)
Applications
15. 15 © 2014 IBM Corporation
Hybrid Applications – Background and Goals
Hybrid environment – includes both physical and virtual resources
‒ Pools: Resources are grouped into bare-metal and virtual pools
Hybrid application – a composite (e.g., multi-tier) application that
spans both virtual and physical server(s)
Goals
‒ Admin: Minimize management complexity and overhead. Unified
management across bare-metal and virtualized pools.
‒ User: Simplified ongoing management and application life cycle.
Design guideline: If possible try to use OpenStack native solution
‒ E.g., no external coordinators, orchestration, management
16. 16 © 2014 IBM Corporation
Pool 3: Virt Type 2 Pool 4: Bare metalPool 1: Spare (BM) Pool 2: Virt Type 1
Tenant ATenant B
Hybrid Applications – Background and Goals
18. 18 © 2014 IBM Corporation
Scenario 1: Node Repurposing
Support native policies that dynamically balance the virtual and
physical pools (e.g., moving servers from one to another)
Repurposing Virtualized node (Hypervisor) Physical node
1. Detect resource congestion in the physical pool
2. Identify a candidate host to be re-purposed into physical
3. Trigger “evacuate“ for all VMs on the selected host
4. Assign physical host to the BM pool
‒ Server is ready to be deployed as bare-metal application (later on)
Bare metal pool
Bare metal pool
19. 19 © 2014 IBM Corporation
Scenario 2: Runtime Decision on Target Pool
Scenario: decide on the server type (BM/VM) upon deployment
E.g., according to performance requirements
1. User deploys an application without specifying the server type
2. The system automatically choose server type
3. Automatically choose/construct actual image for deployment
‒ Option 1: dynamic adaptation
‒ Option 2: Maintain multiple ‘versions’
4. Deploy using corresponding provisioning mechanism
‒ leveraging Ironic
‒ Apply adaptation upon boot (e.g., leveraging Heat)
20. 20 © 2014 IBM Corporation
Solution Approach
21. 21 © 2014 IBM Corporation
Management Approaches
Possible solutions range from 'share nothing to 'share everything'
‒ 'share nothing’ - dedicated services for each pool
‒ ‘share everything’ - Use single copy of each service
‒ Some of the services are shared (e.g., regions, cells)
OpenStack can natively handle bare metal and virtual hosts, so
why not combining them together in a single OpenStack?
We focused on the 'shared everything' architecture
‒ Defined as 'Integrated Management‘
22. 22 © 2014 IBM Corporation
Integrated Management: Basic Architecture
Special resource types, mapped to specially designed resource pools
Use multiple host aggregates, including ‘bare-metal’
‒ Use aggregate filters for scheduling
Networking
‒ Admin network for provisioning and management of compute nodes
‒ Separate management network for BM machines hosting user applications
‒ Data network combining both VMs and BM nodes
Controller node
mysql-server
rabbit-server
neutron-server
glance
Network node
Neutron.*-plugin-agent
Neutron-l3-agent
Neutron-dhcp-agent
Neutron.*-plugin-agent
Neutron-l3-agent
BM computeCompute node
nova-compute
Neutron.*-plugin-agent
Compute node
nova-compute
Neutron.*-plugin-agent
Compute node
nova-compute
Neutron.*-plugin-agent
BM
node
BM
node
BM
node
External
Data/Guests
Management
API
Internet
BM Mgt
BM
node
BM
node
BM
spare
23. 23 © 2014 IBM Corporation
Advantages of Integrated Management approach
Native and simple OpenStack-based solution
‒ Doesn’t require external orchestration
‒ No need to coordinate between services of different pools
Simplified administration
‒ Controlling all pools/resources from one place in a unified manner
‒ Reduce configuration complexity and overhead
‒ Better diagnostics and root cause analysis
Improved user experience and manageability
‒ Maintain/expose combined topology view for different server types
‒ Hide the underlying infrastructure details from the user (if needed)
24. 24 © 2014 IBM Corporation
Technical Challenges and
Design Considerations
25. 25 © 2014 IBM Corporation
Challenges Examples
OpenStack has some gaps when managing bare-metal and
virtual machines simultaneously
Networking: Simultaneously manage both virtual and physical
networks
‒ Network isolation for tenants should combine both BM and VMs
‒ E.g., Use tunnels/overlays connecting physical nodes and VMs
‒ E.g., Use multiple plugins (e.g., OVS, openflow) simultaneously to
gain advantages for each pool (if possible)
Managing Bare-Metal workloads
‒ Security in multi-tenant environments (e.g., device firmware)
26. 26 © 2014 IBM Corporation
Challenges Examples (Cont.) - Compute
Enhanced scheduler to better support heterogeneity
‒ Use different placement policies for different pools (e.g., look at
GPU consumption rather then CPU/memory)
Support instance management for BM
‒ Actions for BM instances may differ from actions on VMs
‒ E.g., implement: Console, Explore usage of: Suspend, Resize
Add hierarchical relations between BM instance and the
compute node running on it
‒ E.g., Propagate BM delete command to the VM level
Support aggregates for specific BM nodes
27. 27 © 2014 IBM Corporation
Challenges Examples (Cont.)
Image Management
‒ Enhance access control for images in Glance
‒ Support run-time image selection
‒ Support dynamic adaptation of images
UI: extend UI to support Hybrid applications/instances
‒ Add bare-metal node management in Horizon
‒ Physical machines are shown as Hypervisors
28. 28 © 2014 IBM Corporation
Hybrid Application example
29. 29 © 2014 IBM Corporation
Example – Deploying Hybrid WordPress
Apache+WordPress on VM and MySQL on physical node
Setup and configuration
‒ A single HEAT template used for the two nodes
‒ Using host aggregates for scheduling
‒ Used diskimage-builder to build the baremetal images
• cloud-init element installed
30. 30 © 2014 IBM Corporation
Example – Deploying Hybrid WordPress
Example gaps reflected in Horizon:
‒ BM nodes are shown as Hypervisors
‒ Deployment images are exposed to users
32. 32 © 2014 IBM Corporation
Summary
Heterogeneous cloud environments are gaining momentum
‒ Critical in order to host a broad spectrum of applications
OpenStack is a promising solution to manage hybrid clouds
‒ By using integrated management we can have
• Simple native OpenStack solution
• Simplified administration and enhanced user experience
Need a careful design to ensure all requirements (e.g., security,
isolation) are maintained across the hybrid environment
Some gaps need to be addressed in OpenStack