2. Outline
● Background and motivation
● Evaluation of available options
● CloudBridge
○ Design philosophy
○ Testing philosophy
○ Structure
○ Code Samples
● Demo
○ CloudLaunch
● Using
● Contributing
● Questions
3. Why?
● We needed to make the Genomics Virtual Lab/Galaxy CloudMan
cloud independent
○ Original version written against Amazon’s EC2 APIs
○ Ran on OpenStack using OpenStack’s EC2 Compatibility Layer
● OpenStack EC2 compatibility layer had limitations
○ Stability issues
○ Lacked features: Cannot name instances, cannot specify root volume sizes etc.
etc.
○ Deprecated - being spun off as separate project
● Go Native! - native APIs were
○ More stable
○ Cutting edge
○ Not likely to be dropped
4. The Available Options
● Apache Libcloud
○ The obvious candidate
○ Supports more than 30 providers
○ Supports Python 2.5 - 3
● HTTP abstraction layers
○ e.g. Apache DeltaCloud, Dasein Cloud
○ Need to run a separate HTTP service
○ Service would abstract away the specific clouds
● Build your own abstraction
○ NIH syndrome?
○ Differences between clouds can be complicated
○ How do you get access to so many clouds anyway?
5. Apache Libcloud Design
● Offers a lowest-common-denominator approach
○ Pros
■ Widest possible compatibility
■ Over 30 providers
■ Supports many versions of Python
■ Basic abstraction standardised
○ Cons
■ Everything else exposed as a provider specific extension (“ex_”)
■ Lots of methods/properties are “ex_”:
● ex_list_availability_zones
● def create_node(ex_keyname, ex_userdata..)
● ex_create_security_group
● ex_create_floating_ip etc.
○ In practice, requires special-cased code, unless requirements are very modest.
6. Apache Libcloud Design (contd…)
● Wraps libraries at a ReST/HTTP level, instead of using native SDKs
○ Pros
■ Minimises dependencies
■ Gives control over client-side implementation
○ Cons
■ Slower to incorporate features over native SDKs
■ Significant duplication of work
● Boto is over 77,000 lines of code
● Libcloud weighing in at 250,000+
7. Apache Libcloud Design (contd…)
● Testing is largely provider specific
○ This is because the exposed methods tend to be provider specific
○ Does not guarantee a write-once-run-anywhere experience
○ Makes it difficult to support a large number of clouds in practice
■ How do you test against many clouds yourself?
■ How do you even get access to so many clouds?
8. Apache Libcloud Design (contd…)
Does not offer a way to determine provider feature sets/profiles at runtime
Makes it difficult to write cross-cloud code
9. Meanwhile...
Mature clouds are converging in terms of functionality and scope
Making a single, uniform interface possible
10. Desirable requirements
● Need a write-once, run-anywhere experience
● Should not need to special-case code on a per cloud basis
○ Cannot obtain access to clouds easily
○ Don’t have the time to test them on each
● Simple and Pythonic
Tradeoffs
● Skip on widest possible compatibility
● Fine-grained feature determination
12. CloudBridge Design Goals
1. Offer a uniform API irrespective of the underlying provider
a. No special casing of application code
b. Simpler code
2. Provide a set of conformance tests for all supported clouds
a. No need to test against each cloud
b. Goal of “write-once-run-anywhere”
3. Focus on mature clouds with a required minimal set of features
a. Enables 1 and 2
4. Be as thin as possible
a. Keep the code-simple and reuse existing work ⟶ greater reliability, reduced development time
13. CloudBridge Testing Goals
1. Write one set of tests that all provider implementations must pass
2. Make the tests a ‘conformance’ test suite
● Validate that each implementation correctly implements the CloudBridge specification
3. No provider specific testing required
● A provider which passes the test suite can be used safely by an application with no additional
provider specific testing
4. Tests must pass against real infrastructure
● Mocks used to speed-up feedback
● But same tests must pass on real infrastructure
5. Aim for 100% code coverage
14. Automated testing using Travis
Testing py27, py35,
pypy
On OpenStack
(DevStack) and Mock
AWS (moto)
15. CloudBridge Design
API revolves around 3 concepts
a. Providers
An entry point for a service, encapsulating a connection
b. Services
Exposes provider functionality/feature sets
c. Resources
A remote cloud resource, such a Machine Instance, Volume etc.
17. Service name list get find create
InstanceService yes yes yes yes
VolumeService yes yes yes yes
SnapshotService yes yes yes yes
ImageService yes yes yes no
NetworkService yes yes yes yes
SubnetService yes yes no yes
ObjectStoreService yes yes yes yes
KeyPairService yes yes yes yes
SecurityGroupService yes yes yes yes
InstanceTypeService yes yes yes N/A
RegionService yes yes no N/A
Service feature set
Services have consistent interfaces:
<ServiceName>.[list|get|find|create]()
Resource feature
set
Resources have a common set of basic
properties: id, name
Plus fields and methods appropriate for
that resource type
22. A quick demo: CloudLaunch
CloudLaunch is a general-purpose cloud launcher
○ Multi-cloud
○ ReST API driven with Angular 2 front end
○ Built on CloudBridge and Django Rest Framework
http://beta.launch.usegalaxy.org/