Over the last few years, we have seen a dramatic increase in the use of open source projects as the mainstay of architectures in both startups and enterprises. Many of our customers and partners also run their own open source programs and contribute key technologies to the industry as a whole (see DCS201). At AWS we engage with open source projects in a number of ways. We contribute bug fixes and enhancements to popular projects including our work with the Hadoop ecosystem (see BDM401), Chromium (see BAP305) and (obviously) Boto. We have our own standalone projects including the security library s2n (see NET405) and machine learning project MXnet (see MAC401). We also have services that make open source easier to use like ECS for Docker (see CON316), and RDS for MySQL and PostgreSQL (see DAT305). In this session you will learn about our existing open source work across AWS, and our next steps.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adrian Cockcroft - AWS VP Cloud
Architecture Strategy
December 1, 2016
Open Source at AWS
Contributions, Support and Engagement
ARC213

2. What to Expect from the Session
• Examine how leading AWS customers use open source
• See the value of an open source program
• Understand the trends in open source adoption
• Baseline existing contributions from AWS
• Begin the discussion of next steps in engagement

3. What do customers want?

4. Capital One use of open source

5. Capital One
https://developer.capitalone.com/open-source/
“We’re an open source first organization —
actively using, contributing to and managing
open source software projects.”

6. Capital One Open Source Projects
https://capitalone.github.io
Cloud Custodian – AWS Lambda based policy engine – ARC312
Hygieia – DevOps delivery pipeline dashboard
OAS-nodegen – Open API Specification support
Ops-Pipeline – scripts for creating environments
Hydrograph – User interface for ETL management
Giraffez – Client library for Teradata
React Native PathJS Charts
SWHttpTrafficRecorder – Swift mobile library
CQRS Manager for Distributed Reactive Services

7. Netflix open source program

8. From 2012 NetflixOSS Presentation by @adrianco

12. Current NetflixOSS Program
http://www.slideshare.net/aspyker/netflix-open-source-building-a-distributed-and-automated-open-source-program

14. NetflixOSS in 2016
http://netflix.github.io
Around 200 projects currently on Github
Deep integration into SpringCloud Java framework
Hystrix circuit-breaker ported and adopted widely
Curator migrated from NetflixOSS to Apache Curator
New: Titus team working with AWS ECS – See CON313
Startups inspired by NetflixOSS:
• DynomiteDB supporting and extending Dynomite Redis clustering
• Armory providing Spinnaker deployment pipeline as a service
• Gremlin implementing Chaos Monkey testing as a product

15. Many Benefits of Open Source Programs
Give back to the communities that you depend on
Attract and curate better developers
Improved code quality, docs, modularity and re-use
Pooled development and testing resources
Validate architectural decisions in public
Enable fast-followers and remain centered in an ecosystem
Build trust and basis for collaboration

16. Open Source is the Default
Leading customers have their own open source programs
Many AWS products have an open source component
Blackduck Annual Future Of Open Source survey
“The survey reports that 78 percent of its respondents are now running their businesses with open
source software, and two-thirds are building software for their customers that’s based on open source
software. More significant, the percentage of respondents actually participating in open source
projects has increased from 50 percent to 64 percent, and 88 percent say they expect to contribute
to projects within the next three years.”

17. Baseline current status

18. Memberships
Code contributions
Projects hosted on GitHub
Open source based services

19. Open Source Memberships
Linux Foundation Silver Member
• Founding advisory member of Xen @ Linux Foundation
• Founding member of the Core Infrastructure Initiative
• Founding member of the Open Container Initiative
ToDo Group - Companies committed to open source
Alliance for Open Media – Video format
Open-MPI - consortium for high performance computing

20. Amazon & AWS contributions

21. Bug Fix and Feature Contributions
Linux & Drivers
Xen
Apache Tomcat
PostgreSQL
Docker
Boto
Apache Hadoop
Apache Hive
Apache Bigtop
Apache Oozie
Apache Drill
Apache Zeppelin
Apache Pig
Cloudera HUE
Apache Lucene
Apache Solr
Kuromoji
ElasticSearch
CBMC
MXNet
Moses
Apache Joshua

22. Highlights for Bug/Feature Contributions
Linux Contributions
• ENA – network driver
• Annapurna – ARM64 chipset support
• AWS recently hired Linux committer David Woodhouse
Boto – Python SDK – AWS hired creator Mitch Garnaat
Hadoop ecosystem projects – over 100 contributions

23. MXNet – Deep Learning Framework
http://www.allthingsdistributed.com/2016/11/mxnet-default-framework-deep-learning-aws.html
Created at CMU
Major investments by AWS
Scalable, high performance
Supported alongside many other deep learning frameworks
Contributions to developer experience, documentation
Tools for visualization, development, migration
Supporting academic contributions with AWS credits
See MAC401 - Scalable Deep Learning Using MXNet

24. Repositories Owned by AWS & Amazon
https://github.com/aws https://github.com/awslabs https://github.com/amznlabs https://blox.github.io/
Blox – Container orchestration for ECS – CON316
s2n – Secure replacement for openssl – NET405
Chalice – Python serverless microframework for AWS – DEV308
AWS-CLI – Built by Mitch Garnaat – DEV402
AWS-shell – Autocomplete based user interface for AWS-CLI
AWS SDKs
AWS Mobile SDKs
Cloud9 Ace – Cloud based interactive development editor
Cfncluster – Build and manage HPC clusters
ION – Amazon data serialization libraries

25. Blox – Container Orchestration Framework
https://blox.github.io/
New governance model for Blox – built in the open
All development and roadmap in public
Provides a venue for collaboration with AWS customers
Intent to build an ecosystem
Contribute/port schedulers
See more details in CON316

26. Contributing to Blox
https://blox.github.io/
• Blox is licensed under Apache 2.0
• Open an issue or pull request
• Watch our roadmap on GitHub
• Check out our Gitter channel

27. Open Source Made Easy by AWS Services
Amazon EMR – Elastic Hadoop
Amazon ElastiCache - Memcached
Amazon RDS – MySQL
Amazon RDS – PostgreSQL
Amazon RDS – Scalable Aurora back-end for MySQL & PostgreSQL
AWS OpsWorks – Chef Automate Server
Amazon ECS – Docker Orchestration
Amazon CloudSearch
Amazon Elasticsearch

28. Let AWS Do The Undifferentiated Heavy Lifting
Amazon EMR updates & patches are frequent and automatic

29. Engagement and next steps

30. Program Team
Represents all of Amazon
Manage consumption of projects
Manage engineering contributions
Represent AWS (foundations etc.)
Work with legal on licensing
Assist with standalone projects
Operate GitHub accounts
Did everything to this point
AWS Teams Managing Open Source
New – Outbound Programs Team
Engage with AWS customers
Support academic credit program
Create content: blogs, talks etc.
Represent AWS (foundations etc.)
Present at industry events
Manage example code projects
Measure sentiment/engagement
Starting now!

31. AWS Cloud Credits
for Research

32. AWS has been supporting global research projects through the
Cloud Credits for Research program since 2009.
 AWS believes that researchers are key drivers of technological innovation
 AWS Cloud Credits for Research is aimed at helping researchers move their
research endeavors to the cloud so they can innovate more quickly and at
lower cost

33. AWS has been supporting global research projects through the
Cloud Credits for Research program since 2009.
We support researchers who seek to:
 Build reusable tools to facilitate their future research and the research of
their community
 Perform pilot, proof of concept, or benchmark tests evaluating the efficacy
of moving research workloads or open data sets to the cloud
 Train a broader community on the usage of cloud for research workloads via
workshops or tutorials

34. AWS Cloud Credits for Research Recipients
https://aws.amazon.com/research-credits/2016-recipients/
Categories
Proof of Concept
Development of Open Source Software
Development of a Cloud Service
Open Dataset Development
Benchmark for Comparison
Workshop or Tutorial

35. AWS Credits for Open Source Projects
Awarded on a case-by-case basis:
Berkeley AMP Lab – Founding Sponsor
Apache Mesos, Spark and other open source tools
Open-MPI Consortium support
PostgreSQL project was given credits to support testing

36. How can we help?

37. Adrian Cockcroft
@adrianco
Experience:
Startup Open Source Ecosystems
NetflixOSS
New - Outbound Team
Contact us via AWS account management, or in person at #reinvent
Zaheda Bhorat
@zahedab
Experience:
UK Gov Open Document Format
Salesforce Open Source Program
Google Summer of Code
Sun OpenOffice & Netbeans

38. First steps, we’re listening…

39. Thank you!

40. Remember to complete
your evaluations!

41. Related Sessions
DCS201 - Open-Source Resources and Certifications
BDM401 - Deep Dive: Amazon EMR Best Practices & Design Patterns
BAP305 - Zero to Google Chrome in 60 Minutes: Lightweight and Inexpensive
Client Devices for Amazon WorkSpaces
NET405 - Amazon s2n: Cryptography and Open Source at AWS
MAC401 - Scalable Deep Learning Using MXNet
CON316 - State of the Union: Containers
CON313 - Netflix: Container Scheduling, Execution, and Integration with AWS
DAT305 - Deep Dive on Amazon Relational Database Service
DEV308 - Chalice: a Serverless Microframework for Python
DEV402 - The Effective AWS-CLI User