1. Sam Brown
samuel.brown@excella.com
November 7, 2012

2. Thanks to Mike McGarr and
Excella Consulting for hosting!!

3. Sam Brown
 11+ Years as a Java developer with commercial and
federal clients
 Practicing continuous integration/continuous delivery
for ~6 years
 DevOps Evangelist at Excella (www.excella.com)
 Certified Scrum Master
 Puppet Certified Professional

4. Basic components of an automated enterprise
 Continuous integration
 Dependency management
 Automated build tools
to build...
 Shared API libraries
 Custom web applications
 Products

5. “The purpose of a pipeline is to transport some resource
from point A to point B quickly and effectively with minimal
upkeep or attention required once built” – me
So how did „pipelines‟ get applied to software? Let‟s
try a few changes to this statement...
“The purpose of a pipeline is to transport from
to quickly and
effectively with minimal upkeep or attention required once
built” – me

6. Build pipelines require measurements and verification of
the code to ensure:
 Adherence to standards
 Quality proven through testing
 A product that meets user‟s needs
The purpose is not just transport, but to ensure that our
product is high-quality, prepared for the environment it will
reach, and satisfies the end-user.

7. “An automated manifestation of the process required to get your
team’s application code to the end-user, typically implemented via
continuous integration server, with emphasis on eliminating
defects” – me (again)

8.  …in fact, NONE ARE!
 Build pipelines will vary as much as
applications
 Different teams have different needs
 Simplicity is key
One Size
Fits All!

10. Repeatable, automated, process to ensure that application code is
tested, analyzed, and packaged for deployment.

11.  System of record
 Just do it!
 Take advantage of commit hooks
 Build from trunk and reduce server-side
branches
 Tag often
 Don‟t check in broken code!

12. Purpose: Integrate, build and unit test code for quick
feedback
 Best Practices
 Runs in under 10 minutes (rapid feedback)
 Unit tests do not require external resources
 Run on EVERY developer check-in
 Fixing broken builds is the top priority!
 Gamification to drive adoption
 80% test coverage or BETTER
 Challenges
 LOTS of builds
 False sense of security
 Writing tests is hard

13. Purpose: Test component and/or external resource
integration
 Best Practices
 Test connectivity with external resources
 Test frameworks load correctly
 Test application components work together
 Test configuration
 Fewer integration tests than unit tests
 Challenges
 External resources may not be available in all environments
○ Mock locally
 Can be time consuming
○ Use local resources
○ Separate short/long running tests

14. Purpose: Use automated tools to inspect code
 Best Practices
 Check syntax
 Find security vulnerabilities
 Record test coverage
 Discover complexity
 Optional: Fail based on a metric
 Optional: View technical debt
 Challenges
 Not all code analysis tools are free
 Learning/installing new tools

15. Purpose: Label code and package as
deployable
Best Practices
 Labeling allows you to go back in time
 Packaging code for deployment
 Reduce complexity by combining steps
 NO configuration in package -> Package once,
deploy multiple
 Challenges
 Labeling can be resource intensive
 Many packaging options

16. Purpose: Make artifacts available for
deployment or available to other teams
 Best Practices
 Publish a versioned artifact
 Make repository available
 Reduce complexity by combining steps
 Challenges
 Requires initial complex setup
 Security requirements around exposing artifacts
○ Use a tool with security built-in like Nexus

18. Repeatable, automated, process to ensure that our target environment
is properly constructed for our application(s).

19. Purpose: Check syntax and compile prior to
application
 Puppet Lint – Static format checker for
Puppet manifests
 No-op Test Run – Ensure that manifest
compiles
 Challenges
 Puppet-lint requires a ruby-based environment
 No-op test needs production-like VM
 Long feedback loop

20. Purpose: Test infrastructure in a prod-like environment
 Puppet Apply –Puppet application against VM that
mimics DEV/TEST/PROD
 Infrastructure Tests – Test your environment!
 Example tests:
 Users and groups created
 Packages installed
 Services running
 Firewall configured
 Challenges
 Long feedback loop
 Yet another language (cucumber/rspec/other)
 VM must be up to date with DEV/TEST/PROD

21. cucumber-puppet rspec-puppet
Feature: Services require 'spec_helper'
Scenario Outline: Service should be running and bind to describe 'logrotate::rule' do
port let(:title) { 'nginx' }
When I run `lsof -i :<port>`
Then the output should match /<service>.*<user>/ it { should include_class('logrotate::rule') }
Examples: it do
| service | user | port | should contain_file('/etc/logrotate.d/nginx').with({
| master | root | 25 | 'ensure' => 'present',
| apache2 | www-data | 80 | 'owner' => 'root',
| dovecot | root | 110 | 'group' => 'root',
| mysqld | mysql | 3306 | 'mode' => '0444',
})
end
http://projects.puppetlabs.com/projects/cucu end
mber-puppet/wiki
http://rspec-puppet.com/

23. Repeatable, automated, process to ensure that our application is
properly installed in the target environment and that the application
meets acceptance criteria.

24. Purpose: Test acceptance criteria in a prod-like
environment
 Puppet Apply – Apply Puppet manifests including
deploying application
 Run Acceptance Tests – “End-to-end” testing
 End-user perspective
 Meets user-defined acceptance criteria
 Possible tools: Cucumber, Selenium, Geb, Sikuli
 Challenges
 Maintain a production-like VM
 Acceptance tests brittle
○ Test at the right level
 Acceptance tests long running
○ Run nightly

25. Purpose: Label application and infrastructure
code, deploy to DEV environment
 Label Release Candidate – Known
“accepted” versions will be deployed
together
 Deploy to DEV – Automated deployment
 Infrastructure AND application
 Challenges
 DEV updating, not deployed from scratch
○ Create tests for ALL possible scenarios
 Security
○ Work with security early and often!

27. Simplified process to support streamlined deployments to TEST
and PRODUCTION

28. Purpose: Enable the test team to pull the
latest code
 Pull-based deployment
 Manual Testing/Approval
 Challenges
 Enabling test team is a paradigm shift
 Producing changes too fast
○ Create good release notes
○ Not every build needs manual testing

29. Purpose: Enable operations team to pull the
latest code into production
 “Push-button” deployment to production
 Requires testing approval
 Challenges
 Audit/security check before deployment
○ Discuss with operations
○ Automate as much as possible and prudent
 Paradigm shift for operations, TOO EASY!
○ Engage the operations team as early and often
 Rollback/Roll forward strategy
○ Easier with RPM‟s, I prefer roll forward

30.  Remove human error
 Repeatability tests and improves the
process
 Visibility from code to deployment
 Baked-in quality
 Metrics, metrics, metrics
 Rapid and constant feedback
 Releases are non-events

31. Why do we store old/obsolete versions?
 Rollback
 Auditing
 History?
 Any other reason?
My view: Store only the latest build and current production release
 Bugs fixed in latest version
 (Almost) impossible to reproduce environments
 Version control has history
Exception: Other teams dependent on a previous version
 Store major/minor revisions
Reasoning: In a continuous delivery environment, delivering frequently
allows you to keep moving forward with new features AND bug fixes!

32.  Put EVERYTHING in version control
 Start simple, up your unit test coverage.
 Analyze your code in order to focus
 Install CI and start with two build steps
 Start and maintain a wiki
 And lastly…

34.  samuel.brown@excella.com
 @SamuelBrownIV
 http://github.com/samueltbrown
 http://www.linkedin.com/pub/samuel-brown/3/715/352