Presentation given at OSCON 2015 Open Cloud Day on 7/21/2015
For open source projects such as OpenStack, being able to effectively build, release, and deploy upstream code into a production environment is as much art as science. Using the experience gained from more than three years in OpenStack operations, we will share best practices and tools to create a sustainable and repeatable release engineering process for your own open source development needs.
http://www.oscon.com/open-source-2015/public/schedule/detail/44790
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Release Engineering Downstream of an OpenStack Project
1. Release Engineering Downstream
of an OpenStack Project
Best Practices for the Path from Idea to Production with
an Open Source Code Base
Rainya Mosher
@rainyamosher | www.rainyamosher.com
2. People don’t want…a quarter inch
drill. They want a quarter inch hole.
Clayton M. Christensen
The Innovator’s Solution
3.
4.
5. About Rainya
• 10+ Years Web App Devoplment
(ASP & .NET C#)
• 3+ Years OpenStack Operations
• Master of Science in Technology
Commercialization
• Musical Theatre Nerd, Singer, and
Actor
• Twitter: @rainyamosher
• Blog: rainyamosher.com That about sums it up.
ENFP
7. Release Engineering (RelEg / RE) is the sub-
discipline of software engineering concerned
with the compilation, assembly, and delivery of
source code into finished products or other
software components.
https://en.wikipedia.org/wiki/Release_engineering
9. What is your relationship with the
upstream codebase?
10. Current Dates & Milestones: https://wiki.openstack.org/wiki/Releases
11. http://docs.openstack.org/infra/manual/developers.html
• Clone the code from github
• Branch your change
• Make fixes
• Submit review
• Submit patch revisions until you
get passing tests and a +2
• Change is merged
Average Review Wait Times
as of 7/20/2015
• Nova – 22 days
• Neutron – 9 days
• Glance – 26 days
http://stackalytics.com
12. Pull
OpenStack
Master
Consume
Third-Party
Distro
Internal Integrations & Configs
More ownership
over release
cadence
More in-house
resources for
packaging
More flexiblity
with to
integration of
internal
changes into
python source
code
Third party
guides release
cadence
Fewer in-house
packaging
needs
Less flexibility in
integrating
internal
changes into
python source
code
No matter the upstream relationship,
you’ll need to account for your
internal configurations and system
integrations!
13. Even understanding
the upstream
landscape of the
project, there will be
times when someone
– usually a product
or engineering
manager – will say “I
don’t care about the
community, just ship
the feature/bug
fix/widget.” What can
you do?
14. How will you manage the product
and software development lifecycles?
15. PDLC
SDLC
AGILE
Scrum,
Kanban,
XP, etc.
PDLC is the management of a
good or service from inception,
design, engineering, build, sale,
maintenance, and disposal
through the integration of
people, data, process, and
business systems to make an
optimal experience for a
customer (internal or external).
SDLC is a framework of steps
and phases providing a model
for development teams to follow
when creating a piece of
software to improve the quality
of the software and the overall
process.
AGILE SDLCs are a collection
of methodologies valuing
flexibility, collaboration,
communication, and rapid
iterations in the development
process.
16. Simple Easy
One role / task /
concept / dimension
vs Complex
Familiar, part of our
skillset, near at hand
vs Hard
17. Source Image: Rich Hickey, “Simple Made Easy”, 11/2012
Early on, the easy path gives
you speed and a sense of doing.
Over time, easy ignores
complexity, slows you down, and
leads to painful technical debt.
18. Being intentional with your PDLC & SDLC approach can help
you get from mad baby to successful sand eating baby.
19. What tools will you use for build,
release, and deploy activities?
21. BUILD
RELEASE
DEPLOY
SYSTEMS
Pipeline
• Invest in your non-production environments
• Engineers cannot deliver stable software to production if the places
they have to develop and validate are unstable
22. Dev
• Prod-like
functionality
(as much as
possible)
• Unit tests at a
minimum
Integration
• First place for
Dev to
integrate code
with upstream
packages
• Unit tests
• Basic smoke
tests
Test
• Full smoke
tests
• High level
integration
testing with
tightly coupled
services
Pre-Prod
• Mirrors Prod
• Full
integration
testing with all
services
• Load and
performance
testing when
major
changes
Production
• Run baseline
tests prior to
an upgrade
• Automate the
deploy to
keep human
error out of
the way
• Don’t touch it!
Speed: Find failures quickly Accuracy: Find all failures Stability!
23. BUILD
RELEASE
DEPLOY
SYSTEMS
Config
Management
• Make all system and software
changes through a config
management system
• Start with immutable servers and
services (it’s harder to give up root
later)
• Ansible, Puppet, Chef, Salt
all have strengths &
weaknesses – take the
time to determine which is
best for your situation!
24. BUILD
RELEASE
DEPLOY
SYSTEMS
Packaging
• Successful packaging has been
done with RPMs and DEBs on
physical hosts and VMs
• Containers are rapidly maturing into
a viable approach for the control
plane
• Finding an approach everyone
can agree to is perhaps the
biggest challenge!
• Configs can be bundled
with the source code or
distributed separately
based on frequency of
updates and scale of
system
25. BUILD
RELEASE
DEPLOY
SYSTEMS
Orchestration
• Thoughtful orchestration for deploy-
time activities can overcome many
shortfalls of software
• Dedicate time and energy to find the
best order of operations for
introducing changes to services
• Deploy patterns - blue-green, rolling,
shotgun – are limited by the
architecture of the software being
deployed, and that’s ok!
26. How will your organization adapt to
the changes necessary for success?
27. We need to create a culture that reinforces
the value of taking risks and learning from
failure and the need for repetition and
practice to create mastery.
Gene Kim
The Phoenix Project
28. CI &
DevOps
Release
Management
Engineering Practices
Always code as if the person who
ends up maintaining your code will
be a violent psychopath who
knows where you live.
~ John F. Woods, c. 1991
Until zero-downtime deploys are a
reality, CI/CD may as well stand for
continual impact and constant
downtime. Adopt a release
management process to stay sane!
Automation first! This is difficult for many
organizations to do. It often means slowing
down, taking the long way rather than the
easy way, and telling people “not right now.”
30. References
Images
- Gripp & Drouhard. Space Unicorn. (2012). http://youtu.be/17o1OlroNSE
- Miltonious Blog Unicorn of Technical Difficulties, Source Unknown
- Random Internet Meme’s found via Google Search
- Hickey, R. 2012. Simple Made Easy. http://www.slideshare.net/evandrix/simple-made-easy
Quotes
- Christensen, C. M. (2003). The Innovator’s Solution: Creating and Sustaining Successful Growth
- Kim, G., Behre, K., Spafford, G. (2013). The Phoenix Project: A Novel about IT, DevOps, and
Helping Your Business Win
32. Release Engineering Downstream
of an OpenStack Project
Discovering the Path from Idea to Production with an
Open Source Code Base
Rainya Mosher
@rainyamosher | www.rainyamosher.com
Notas do Editor
References:
http://www.keirsey.com
Identifiability
Being able to identify all of the source, tools, environment, and other components that make up a particular release.
Reproducibility
The ability to integrate source, third party components, data, and deployment externals of a software system in order to guarantee operational stability.
Consistency
The mission to provide a stable framework for development, deployment, audit and accountability for software components.
Agility
WHAT methodology you choose to follow is far less important than adopting a framework everyone can get behind, from the entry level software devs to the senior leader of the organization.
SDLC Frameworks are a starting point, not the end result. Whichever you chose probably won’t be a perfect fit out of the box. Expect to iterate and tweak the framework to find the best fit for your organization and team.
Unless it’s throw away work, push for simplicity over easy first and foremost. Ease will come with time and the technical debt you DON’T incur.
intro (why i am qualified to talk about this shit); SDLC -- what the hell does "agile" even mean?; BRDS -- why you need build release and deploy systems; Culture - DevOps, release management, change