The value of software is only potential value until it is in users’ hands. There can be many roadblocks to software getting into those hands. These roadblocks tend to revolve around elaborate deployment pipelines stemming from Configuration Management Debt: * Over-burdened release engineering and operations teams * High coupling with centrally managed architecture element/component * Source control practices that impact delivery velocity * Too many variations/versions of the software supported in production * Poor integration processes across architecture components and scaled team delivery * Too many hand-offs between teams in order to release software to users * Code changes feel too risky and takes too long to validate before releasing into production * Poor documentation practices In organizations that have effective configuration management practices it is common to see deployment pipelines that have a smaller number of hand-offs between teams, architectures that tend to be more malleable, and efficient validation processes. By focusing on reducing Configuration Management Debt it is simpler to identify aspects of the integration and release management process that need to be tackled in order to get working software in the hands of users sooner while reducing the bottlenecks in the organizational processes and practices. In this session we will discuss specific approaches and examples on how reducing Configuration Management Debt leads to reducing other forms of software debt including: * Smaller number of hand-offs: Platform Experience Debt * Malleable architectures: Design Debt * Efficient validation processes: Quality Debt * More testable software: Technical Debt

1. Reduce Time to Value
Focus First on Configuration Management Debt
Chris Sterling, Product Owner, CenturyLink Cloud

2. Bio - Chris Sterling
2
Director of Product Management at
CenturyLink Cloud
Recently Launched AppFog:
https://www.ctl.io/appfog
Author of book “Managing Software Debt:
Building for Inevitable Change”
Supported change efforts to adopt Lean,
Agile & Continuous Delivery behaviors in
organizations of 10 up to 800+ people
Entrepreneur & Lean Startup Practitioner
Blog: managingsoftwaredebt.com

3. Social
3
@csterwa
#swdebt
@appfog
@centurylinkcld

4. Types of Software Debt
Configuration Management Debt: Integration and release management become
more risky, complex, and error-prone.
Platform Experience Debt: The availability and alignment of people to business
objectives that involve software changes is becoming more limited or cost-prohibitive.
Design Debt: The cost of adding features is increasing toward the point where it is more
than the cost of writing from scratch.
Quality Debt: There is a diminishing ability to verify the functional and technical quality
of software: the “Break/Fix” mentality.
Technical Debt: These are the activities that a team or team members take shortcuts
on now that will impede future development if left as is.
4

5. Reducing Time To Value
5
Focusing on Configuration Management Debt first leads
to opportunities for reducing all forms of software debt
Reduce hand-offs and dependencies in Org to reduce
Platform Experience Debt
Malleable architectures with Pluggable UI,
Microservices and APIs to reduce Design Debt
Increase efficiency of validation processes to reduce
Quality Debt
Make software more testable to reduce Technical Debt

6. Configuration
Management
Debt
“If releases are like giving birth,
then you must be doing
something wrong.”
— Robert Benefield, CTO - Evolve Beyond
6

7. Case Study: Web Property
7
180+ person “Web 2.0” product organization
Waterfall SDLC that development uses to deliver in 6 month release
cycles
Want to use Agile methods to be more responsive to users and keep up
with other “Web 2.0” companies
Transitioned to Agile methods on 15 teams in 3 months
Changed release management strategy, added XP technical practices,
and implemented Scrum product development framework for scaled
coordination
Able to release every week to users within 4 months
Used streamlined deployment environment process to validate product
changes daily using Continuous Integration and automated promotions

8. Case Study: Ad Platform
8
700+ person Ad Platform organization
Extended dependencies out to 2000+ people throughout company
Millions of lines of code + Hundreds of unique apps & services
Continuous Integration server involved thousands of jobs & 1M+
builds per year
9-18 month overlapping release cycles
Found opportunity to not branch for each overlapping release
After 2 years can release to 20+ data centers globally every day
4 years later they release to production with no humans for nearly
100% of their company application & service deployments

9. Principle:
Focusing on Configuration
Management Debt first leads to
opportunities for reducing all
forms of software debt
9

10. Where Does CM Debt
Source control practices that impact delivery velocity
Too many variations/versions of the software supported in production
Over-burdened release engineering and operations teams
High coupling with centrally managed architecture element/
component
Too many hand-offs between teams in order to release software to users
Poor integration processes across architecture components and scaled
team delivery
Code changes feel too risky and takes too long to validate before
releasing into production
Poor documentation practices
10

11. Traditional Source Control
11

12. Traditional Source Control
11
Main Branch

13. Traditional Source Control
11
Main Branch
Version 1
Branch
Integrate for
Version 2
Code
Complete

14. Traditional Source Control
11
Main BranchDebt
Death March
Version 1
Branch
Integrate for
Version 2
Code
Complete

15. Traditional Source Control
11
Main BranchDebt
Death March
{
Debt accrues quickly within stabilization periods
Version 1
Branch
Integrate for
Version 2
Code
Complete

16. Worse Source Control
12
Main Branch
Version 1
Version 2 Version 3 Version 4

17. Worse Source Control
12
Main Branch
Version 1
Version 2 Version 3 Version 4
{
We wish branch points were this clean

18. Flexible Source Control
13

19. Flexible Source Control
13
Main Branch

20. Flexible Source Control
13
Main Branch
Version 1

21. Flexible Source Control
13
Main Branch
Version 1 Version 2

22. Flexible Source Control
13
Main Branch
Version 1 Version 2
{
Not Easy! Must have proper infrastructure to do this.

23. Scaled Continuous Delivery
14
Component
Validation
Integrated
Component
Validation
End-to-End &
Load/Stress

24. Canary Deployments
15

25. Embrace
DevOps
Culture
Cultures, for better or worse, are
very stable.
— Jeff Bezos, CEO Amazon
16

26. Principle:
Teams own software delivery
from solution to deployment to
operations.
17

27. Monitor App Performance
18

28. Assess Anomalies
19

29. Monitor Valuable Trends
20

30. Platform Experience
Debt
“As in Nature, if an organization is too inflexible
or stands still too long it will get eaten.”
- James Burke

31. Principle:
Rather than creating teams to
work on projects, let’s find ways
to give work to cross-functional
teams.
22

32. Feature Team
“Feature Team” structure
Uses common Product Backlog
Integration is done in parallel
Requires high levels of communication
across teams to resolve integration
issues
Forces Teams to be more coordinated
Sprints should be synchronized
Cross team fertilization is a
requirement to successfully
deliver in parallel
23

33. Design Debt
Technical features that involve improving
software quality attributes can be prioritized
based on the cost of not addressing them.

34. Design Debt
Technical features that involve improving software
quality attributes can be prioritized based on the
cost of not addressing them.

35. 26
Malleable architectures and design
enable nimble business capabilities.

36. Feature Toggles
Problem: Need to deliver changes to production on master
with features that are not sufficient or validated with customers
Introduce feature behind toggle
▶ ON: Show and allow access to feature
▶ OFF: Don’t show or allow access to feature
Should be secondary option to introducing smaller
capabilities that lead to sufficient feature
27

37. Microservices
28
Pluggable UI
API Routing
Service 1 Service 2 Service 3 Service 4
Identity Management
Authorization Authorization Authorization Authorization
Data Data Data Data
Messaging Platform

38. Application Programming
29
API

39. Deploy to a Platform
30

40. Quality Debt
“Promises make debt, and debt makes promises.”
— Dutch proverb

41. The Three Amigos
Quickly get testers,
coders, and business on
the same page before
building based on a
requirement
32
* The Three Amigos pattern originally coined by George Dinwiddie
http://www.stickyminds.com/s.asp?F=S17232_COL_2

42. The Three Amigos Pattern
33
As a Shopper I want to
receive updates on incredible
deals that are located near
my home so that I can save
money on my purchases
Acceptance Criteria:
Save Shopper’s location
Ask Shopper if they want to receive
localized deals daily
Send notification of incredible deals to
Shoppers located within 10 miles each
morning
Allow Shopper to stop receiving localized
deals

43. The Three Amigos Pattern
34
As a Shopper I want to
receive updates on incredible
deals that are located near
my home so that I can save
money on my purchases
What areas of the application will this
affect?
What is the overall design? (UI, API, UX,
etc…)
What are the details test cases for this user
story and it’s acceptance criteria?
What about negative test conditions?
What about boundary conditions?
How might we put existing functionality at
risk?

44. The Three Amigos
At minimum include tester,
coder & business rep in
discussion
Should only take 30 minutes to 1
hour for user stories
Focus on clarification and design
through testable inputs/outputs
Extend to Operations to reduce
Configuration Management Debt
35

45. Acceptance Test-Driven
36

46. ATDD Case Study
Test Automation Reduces Cost of Change

47. Manual Regression Testing
Testing was taking 75 person hours during 2 full test
runs consisting of:
▶ Comprehensive manual regression testing
▶ Data conversion and validation
Cost for testing was $17,000 each iteration
38

48. Introducing Fit into Testing
After 8 iterations team had introduced healthy
amount of Fit fixtures and automated tests
Reduced 70+ hour test runtime down to 6 hours which
now included:
▶ Fit automated regression testing
▶ Data conversion and validation automated with Fit
fixtures
Reduced cost of testing each iteration from $17,000 to
$7,000
39

49. The Agile Regression Testing
40
* The Agile Triangle has been modified from Mike Cohn’s original version

50. The Agile Regression Testing
40
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests
Make up largest portion of
regression tests and are
developed by programmers

51. The Agile Regression Testing
40
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests
Make up largest portion of
regression tests and are
developed by programmers
Integration Tests
Automated &
Exploratory

52. The Agile Regression Testing
40
* The Agile Triangle has been modified from Mike Cohn’s original version
Automated Unit Tests
Make up largest portion of
regression tests and are
developed by programmers
Integration Tests
Automated &
Exploratory
Smoke++ Tests
Risk-based UI &
API Automated
Tests

53. Technical Debt
“For every [dollar] of competitive advantage gained by cutting
quality, it costs $4 to restore it; and software is an organizational
asset and decisions to cut quality must be made by executive
management and reflected in the financial statements.”
— Ken Schwaber, co-creator of Scrum
Source: http://www.infoq.com/presentations/agile-quality-canary-coalmine

54. Continuous Integration
42

55. Early Warning Signs
43
Early Warnings:
• Broken Builds
• Broken Automated Tests
• Broken Custom Thresholds

56. Quality Dashboard - Sonar
44

57. Early Warning Dashboard
45
Measuring Early Warning Signs:
• Design Debt in Duplication (DRY)
• Technical Debt in Code Complexity
• Quality Debt in Bug DB (Break/Fix)
• Other Custom Thresholds

58. Reducing Time To Value
46
Focusing on Configuration Management Debt first leads
to opportunities for reducing all forms of software debt
Reduce hand-offs and dependencies in Org to reduce
Platform Experience Debt
Malleable architectures with Pluggable UI,
Microservices and APIs to reduce Design Debt
Increase efficiency of validation processes to reduce
Quality Debt
Make software more testable to reduce Technical Debt

59. Q&A