2. About eBay Inc.
⢠eBay Inc. enables commerce by delivering flexible and scalable solutions that foster merchant growth.
⢠eBay Marketplace: 128 million active users
⢠More than 500 million items are listed on eBay Marketplace
⢠PayPal: 143 million active accounts
⢠Processing almost 8 million payments every day
⢠193 markets and 26 currencies around the world
Continuous Integration and Deployment using OpenStack 2
OpenStack at eBay Inc.
⢠One Cloud
⢠Multiple Regions (global)
⢠Multiple Availability Zones within Regions
⢠Global Engineering and Operations Teams
⢠Internal customers: developers, R&D, projects, business units (everyone)
3. Agenda
⢠Enter Fluo (CI/CD)
⢠Fluo Flow
⢠Fluo Code Replication
⢠Packages, Artifacts, <your term here>
⢠Distribution of Packages
⢠Infrastructure as Code
⢠Deployment to Production
⢠Screenshots
⢠Roadmap
Continuous Integration and Deployment using OpenStack 3
5. At the beginning
Basic system running
⢠Gerrit
⢠Jenkins
⢠Web server
Requirements
⢠Single Interface
⢠Scalable
⢠Simple to use
⢠Developer Friendly
⢠Generic
Continuous Integration and Deployment using OpenStack 5
6. Fluo and the CICD configuration file
⢠What is Fluo?
â WebUI
â Single pane of glass
â Cloud instance provisioner
â Allows userâs to configure the system
â RBAC
⢠How it works?
â Jenkins through API
â Gerrit through API
â Uses ZMQ to communicate with workers which do non-API actions
â Recycle Jenkins slaves or provisions them on the fly at the moment of job execution.
⢠What is the cicd configuration file?
â Specification and requirements files
â How the developers instruct the system what to do (run unittests, how to build packages, install
dependencies, moreâŚ).
⢠Example
Continuous Integration and Deployment using OpenStack 6
10. Basic Workflow
Continuous Integration and Deployment using OpenStack 10
⢠Fluo basic workflow has 6 different stages/step
â Review
â Approval
â Build
â Integration
â Release
â Periodic
⢠At each stage/step Fluo will:
1. Build a cloud instance
2. Read the configuration file
3. Execute the scripts defined
4. Report back status
5. Destroy the instance
⢠Customs workflows can be created by users.
Developer Fluo
Commit and request
review
Execute Review Scripts
Approve the code change Execute Approval Scripts
Merge and replicate code
Execute Build Scripts
Adds âRun Integrationâ to
comment history on
specific change
Execute Integration
Scripts
Tagâs a specific commit Execute Release Scripts
At XY time of day Execute Periodic Scripts
11. Code Quality = Must have⌠Review, Approval, Build
Continuous Integration and Deployment using OpenStack 11
The rest of the flow⌠Integration⌠ReleaseâŚ
Periodic.
Review
⢠Code change submitted
for review
Approval
⢠Executed before
merging the code when
code change is
approved by someone
Build
⢠Executed after the code
change has been
merged
Integration
⢠Activated by user through
Fluo by commenting âRun
Integrationâ on a specific
code change.
Release
⢠Activated by user
adding tags to a
specific commit.
Periodic
⢠Activated by at a
specific time of the day,
week. Similar to a cron
job.
13. Github, Internal Github and your own Git repository.
Some points to consider:
⢠Categorize your code
â Will it be open sourced?
â Is it private for our team?
â Is it private for our company?
â Do we need mirrors?
⢠Always use ssh keys
⢠Standards
â User which will be replicating
â Replicate Branches or maybe notâŚ
Continuous Integration and Deployment using OpenStack 13
Fluo
Github
Local
Github
Private
Github
15. When and where do I create my packages?
Things to consider.
⢠Jenkins slaves will build the packages.
⢠Make sure all your dependencies are in your cicd config file.
⢠Prepare your environment with the package_install and before_{stage} sections.
⢠Build your package at the {stage}_script section.
⢠Define a versioning standard (0.0.x for test packages, x.y.0 for production packages)
Two main stages were created for build purposes.
⢠Build stage
â Use commit number + time in secs as package version.
⢠Release stage
â Use git tag as package version.
Continuous Integration and Deployment using OpenStack 15
17. Shipping Packages to a Central Point
⢠Use simple and secure methods.
â Secure Object storage
â Rsync + SSH
â SCP
⢠Mrepo (if you are RPM Based)
⢠Replication is done every 5 mins.
Continuous Integration and Deployment using OpenStack 17
Replicate⌠Replicate and Replicate.
Fluo
Package
Repository
Cloud A
Datacenter
2
Cloud B
Datacenter
1
19. Configuration Management
⢠At the moment we use Puppet.
⢠Puppet Code will also go through Fluo.
Continuous Integration and Deployment using OpenStack 19
Some guidelines to our puppet coders.
⢠Create your CM code in a service oriented way.
⢠Use virtual resources.
⢠Create within a module sections for each type
of OS you work with.
⢠Create a common module.
⢠Use parameterized classes.
⢠Puppet parser validate filename.pp at the
review/approval stage.
⢠Puppet apply âvd --noop --
modulepath=/modules filename.pp at the
review/approval stage.
Example:
⢠virtual.pp
⢠classes
⢠someclass.pp
⢠modules
⢠Fluo
⢠files
⢠templates
⢠manifests
⢠init.pp
⢠Redhat
⢠install.pp
⢠config.pp
⢠service.pp
⢠Ubuntu
⢠install.pp
⢠config.pp
⢠postconfig.pp
⢠service.pp
21. Itâs GO time
⢠Centralize your configuration management code
⢠Keep it in sync (pull from the repository every 2 or 5 mins).
⢠Tag your Releases with a chronological meaning and keep a standard
â OK
⢠1.1.0
⢠1.2.0
⢠1.10.0
â NOT OK
⢠V1.0
⢠Beta1.0.1
⢠1.2
⢠Remember the TAG goes into the name of the package and its how your package manager identifies
which package should be installed on an upgrade.
⢠Automatic Deployment or Scheduled Deployment?
Continuous Integration and Deployment using OpenStack 21
28. The Future of Fluo at eBay Cloud.
⢠CICDaaS (CICD as a Service).
⢠Integrate with other CM tools.
⢠Completely control of CM tools through Fluo.
⢠Container support (OpenVZ/Docker).
⢠Add swift as an option to store packages.
Continuous Integration and Deployment using OpenStack 28