Software is eating infrastructure. Migrating reliability and
scalability responsibilities up the stack from specialized hardware to software, cloud orchestration platforms such as Apache CloudStack (ACS) and object stores such as Riak CS increase the utilization and density of compute and storage resources by dynamically shifting workloads based on demand. Together, these platform can saturate compute and storage of 1000s of commodity hosts with strong operational visibility and end-user self-service.
This presentation explores cloud design strategies to achieve high availability and reliability using commodity components. It then applies these strategies using Apache CloudStack and Riak CS.
2. Who Am I
• Senior Software Engineer @ Basho Technologies
• Member, Apache CloudStack PMC
• Ran operations and designed automated provisioning for hybrid
analytic/virtualization cloud infrastructures
• Led architectural design and server-side development of a SaaS
physical security platform
4. Better adjust to shifting requirements
Deliver resources and recover from failure faster
Utilize cheaper commodity components
without compromising reliability
8. Software Is Eating Hardware
• Moore’s Law has pulled fault tolerance up the stack
• Commodity Hardware
• Reduced cost
• Services wider range of workloads
• Dynamic resource allocation
Specialized Hardware Reduces Agility
12. Storage
• Reliable
• Fault tolerant
• Specialized hardware
• IT managed provisioning
• Static scale out model
Focused on resource consolidation not agility
13. Cloud Workload
• Reliable
• Fault Tolerant
• Commodity Hardware
• Self-Service provisioning
• Control Plane - Dynamic
scale out model
SDN
Rack Rack
App App
Object
18. What Is Apache Cloudstack?
§Open source software that deploys
and manages large networks of
virtual machines, as a highly
available, highly scalable
Infrastructure as a Service (IaaS)
cloud computing platform.
!
§CloudStack is a Top-Level Project at
the Apache Software Foundation.
19. ~300: Acknowledged production implementations
~43,000: Hosts in a single CloudStack instance
~100: Project committers
20. Anatomy Of CloudStack
• Management Server Farm
• Management and provisioning tasks
• Zone
• Collection of pods, network offerings and secondary storage
• Pod
• Collection of clusters in the same failure boundary
• Cluster
• A grouping of hosts and their associated storage
• Hosts
• Servers onto which services will be provisioned
• Primary Storage
• VM disk storage
• Network
• Logical network associated with service offerings
• Secondary Storage
• Template, snapshot and ISO storage
!
Host
!
Host
Cluster
CloudStack Pod
Zone
Cluster
Network
Primary
Storage
VM
VM
CloudStack Pod
Secondary
Storage
21. Primary Vs Secondary Storage
• Random access read/write
• Changes often
• Zone Scoped
• Focused on throughput
• Write once/read many
• Append only
• Zone or Region Scoped
• Focused on reliability
Secondary Storage is the compute lifeboat.
22. Secondary Storage Vm
• Required for secondary storage operation
• Separates data transfer operations from the control plane
• Transfers data and manipulates secondary storage as directed
by the management server
• Dynamically scaled by the management server
• An SSVM template is required and its installation is an
additional step
23. What Is Object Storage?
• Stores large, immutable, chunks of unstructured data
referenced through a unique key
• Objects live in a unique global namespace (e.g. bucket)
• Simple interface that provides uniform access to distributed
storage
• Typically provide redundancy and replication in software
allowing the use commodity JBOD storage
24. What Is Riak CS?
• Simple, reliable storage of large objects accessed using the S3 or Swift
API
• Multi-tenant security model with per object ACLs
• Usage Accounting and Reporting
• Open Source (Apache2 Licensed)
Your Own Private S3
25. Anatomy Of Riak CS
• S3/Swift API Proxy + Riak key-value store
• Peer-to-Peer Design: Any node can service any read or accept
any write request
• Maintains multiple replicas (default 3) spread across multiple
nodes
• Ops Friendly
Horizontally scalable storage on commodity
hardware
26. Riak CS Use Cases
• Secondary Storage for VM snapshot and template data
• Backup Storage
• Serving Static Content
• Application Log Storage
27. Using Riak Cs With CloudStack
• Create a user account and bucket for CloudStack — the user’s
ACLs should be restricted to the CloudStack bucket
• In CloudStack, define an S3 secondary storage resource using
the Riak CS cluster’s end point, the CloudStack account’s
credentials, and the CloudStack bucket name
• Create an NFS share to stage files sized to meet peak snapshot
and VM creation load. This share is scratch space to support
hypervisors that can not directly stream objects.
28. Achieving Commodity Reliability
• Create a zone per power source/backbone network connection
• Establish independent primary storage in each zone with block-level
replication between zones
• Utilize Riak CS for secondary storage — evenly distributing
nodes across zones
• Distribute application instances across all zones
38. Up And Running With Riak CS
• Learn more @ http://basho.com/riak-cloud-storage
• Grab it @ http://docs.basho.com/riakcs/latest/riakcs-downloads/
• Follow us @basho
• Help from the riak-users@lists.basho.com mailing list or on
FreeNode @ #riak
• Contribute @ http://github.com/basho/riak_cs
39. Join Our Community
• Learn more @ http://cloudstack.apache.org
• Grab it @ http://cloudstack.apache.org/downloads.html
• Follow us @cloudstack
• Help from the cloudstack-users@cloudstack.apache.org mailing
list or on Freenode @ #cloudstack
• Contribute on the cloudstack-dev@cloudstack.apache.org
mailing list or on Freenode @ #cloudstack-dev