This presentation deals with agile and lean support and maintenance of IT services. Agile pproaches are considered as software development approaches mostly, but the usage of agile principles and techniques also brings a lot of benefits to the support and maintenance. Common support and maintenance standards, recommendations and methods are process oriented and omit human aspect that leads to low innovativeness and lack of proactive approach. Another current challenge perceived by companies and researchers is connection between IT and business, namely how IT contributes to business. This presentation takes a critical viewpoint of current approaches supported by facts and evidence. To solve mentioned issues, we define agile and lean support and maintenance principles and control framework for improvement and implementation of these principles in IT services. This framework respects human nature with cognitive science, motivation and typology research as solid basis. The described approach was piloted in 13 IT services. Achievements are enclosed as verification of practical results and benefits.
Agile and Lean support and maintenance of IT Services and Information systems
1. Agile and Lean Support
and Maintenance of IT
Services
Jaroslav Procházka, Ph.D.
Extended version of ISD 2010 presentation
Tieto / University of Ostrava
www.differ.cz
2. Your feedback is
invaluable!
Please share with me, why
you looked for Agile and
Lean Support and
Maintenance topic.
I have e-book on this topic
in progress and would like
to provide the best value ;)
Take short survey:
What exactly do you
look for here?
3. Terms definition
¢ Maintenance (IEEE 610.12 Glossary of SW Engineering):
l “Software maintenance is the process of modifying a software system
or component after delivery to correct faults, improve performances or
other attributes, or adapt to a changed environment.”
l Types: corrective, perfective, adaptive
¢ IT Service (ITIL v3):
l “A service is a means of delivering value to clients by facilitating
outcomes clients want to achieve without the ownership of specific
costs and risks.”
¢ Support (ITIL v3)
l Incident, Event and Problem management process and Service Desk
function
5. XXL team …
¢ Mark, Petr, Dan and Rachel
work as L2 support and
maintenance of legacy
batch processing system
that generates customer
invoices
¢ They struggle with incidents
caused by poor architecture
and long processing time of
legacy system and
technologies
¢ Manual checks need to be
done (physical files copied
in server folders to avoid
batch failure)
¢ Insufficient hardware
requires ongoing
investments
¢ Person on call needed in
8/5 no shift mode (not very
favorite rotating
responsibility)
6. Current state
(measure perspective)
Category of measure Current value
Mean processing runtime (median) 04:49:21
Cumulated monthly runtime 154 h
Runtime per 1000 invoices 00:06:12
Invoices processed per second 2.69
7. It causes many issues
¢ Millions of invoices per week
l Total invoices count is still increasing
¢ Legacy technology, architecture and batch
processing model cause:
l common rework if processing failed
l higher maintenance costs
(because of rework)
l decreasing team motivation
(no creative work, manual checks in error-prone
environment, rework)
8. Current status of
mainstream support and
maintenance approaches
(generalization based on our research,
experience and observations)
9. Current state in IT Service
Support and Maintenance
¢ Support and Maintenance approaches nowadays exist in
form of rigorous process descriptions with required
comprehensive (paper) documentation, e.g.
l IEEE-STD-1219
l ISO 12207 Maintenance sub-process
l Mantema
l ITIL v2 and v3
l ISO 20000
10. Some numbers …
¢ Support and Maintenance – the longest phase of IT service
lifecycle
l Intended period of use about 5-10 years (Application software)
l Real period (mainly telco, banking, government sector): 15+ years
¢ It consumes large part of overall lifecycle costs
• Bennett, Rajlich (2000) Software Maintenance and Evolution: A Roadmap
¢ Poor support and maintenance approach can decrease the cost
and quality of operated and maintained service
l Currently more than 50% of IT budget spent on maintenance:
• Boehm B (1981) Software Engineering Economics
• Hopkins R, Jenkins K (2008) Eating the IT Elephant
11. Budget
Year
Administration, planning, etc.
Wan, Voice, Mainframe, Help desk,
end user support
Operations and maintenance
New development
Significant changes and improvements
Source: Gartner, IT spending and Staffing Report, 2007, 2008 and 2009
12. Statements and discussion I.
1. Process orientation of support and maintenance approaches
(prescribed phases, activities, documentation, measuring process
over measuring value to the customer) omitting human aspect by
assuming the same result with different people, consequences:
1. detailed procedures reducing errors but also creativity, proactive
behavior and learning (affects people motivation)
2. comprehensive (paper) documentation as the main form of
collaboration and knowledge sharing contributing to more expensive
maintenance; this approach omits different context, language and
typology of people involved in communication
3. inability to change quickly to respond to market changes;
4. low motivation of maintenance teams and high fluctuation;
13. Statements and discussion II.
2. Measuring process performance (KPIs) rather than business
value delivered to the customer/end users
l measuring productivity with no respect to nature of creative work à
lacking focus on solving root causes
l utilization as key measure hinders learning, innovative ideas, proactive
behavior (lack of slack space)
Business value: money saved or earned by delivering new feature (new
revenue, revenue increase or operational efficiency; Total Cost of Ownership;
decreased maintenance costs) -> Bills M (2004) ROI: Bad Practice, Poor Results
14. Statements and discussion III.
3. Often lacking practical implementation approach and answers
to typical practical questions
l How to adjust the process/framework under our constraints?
l Shall I use every activity and artifact?
4. Missing connection to software (IS/IT service) development
process or if it mentions, it does not reflect best practices and
latest evolution in SWD domain
l E.g. ITIL recommending waterfall approach
15. Statement evidence I.
¢ Traditional maintenance approaches quite expensive
l Currently more than 50% (even up to 95%) of IT budget spent on
maintenance:
• Boehm B (1981) Software Engineering Economics
• Hopkins R, Jenkins K (2008) Eating the IT Elephant
• Our and community experience: customer IT budget details
l Documentation overhead as part of traditional approaches
• IEEE-STD-1219 recommends 41 different IEEE standards to document software
maintenance lifecycle!
• de Souza et al. (2005, 2007)
l Low contribution to quality (documentation, process)
• Prechelt, Unger-Lampert, Philippsen, Tichy (2002) Two controlled experiments…
• Svensson & Host (2005) Introducing agile process in SW maintenance…
l Error corrections consuming around 21% of maintenance effort
• Bennett, Rajlich (2000) Software Maintenance and Evolution: A Roadmap
16. Statement evidence II.
¢ Poor results of traditional approaches in various types of projects
• Standish Chaos research 1999, 2001, 2004, 2006, 2009 (only 28-35% success rate)
¢ Inability to change software quickly – business opportunity losses
• Bennett, Rajlich (1981) Software Maintenance and Evolution: A Roadmap
• Forrester (2010) Application Outsourcing Clients Are Satisfied, But Want More
¢ Low customer satisfaction (if considering innovativeness, connection
to business goals)
• Forrester (2010) Application Outsourcing Clients Are Satisfied, But Want More
• Our internal customer satisfaction surveys’ comments
¢ Low maintenance team members’ motivation and productivity
• Boehm (1981) Software Engineering Economics
• Our internal team satisfaction survey (max results 2.8 out of 5)
17. Statement evidence III.
¢ Low specialist’s control over final product/service quality: only
5-20% defects come from specialist’s sphere of control, the rest
is system or management sphere of control
• Deming (1986) Out of the Crisis
• Liker (2003) The Toyota way
• Juran, Defeo (2010) Juran’s Quality Handbook, 6th edition
¢ Low maintenance team members’ motivation and productivity
• Boehm (1981) Software Engineering Economics
• Our internal team satisfaction survey (max results 2.8 out of 5)
18. Good stuff in support and
maintenance approaches
¢ There exist many respected and used approaches
l But it causes its usability and relevancy issues at the same
time
¢ Some approaches widely implemented
l e.g. Materna research declares about 60% enterprises in
Middle Europe and above 70% in Scandinavia following
ITIL®
¢ Visible attempts to connect IT and business and also
SWD with support and maintenance
l See ITSM/ITIL, CobiT, Enterprise Unified Process
20. Milestones in SWD approaches
¢ From chaos or ad hoc approaches in 50th and 60th
¢ To detailed rigorous processes in 80th
¢ To iterative & incremental (Agile and Lean) in late 90th
Ad hoc,
chaotic Rigorous,
process oriented Agile (iterative &
incremental) and Lean
21. Agile widely accepted in SWD
¢ Agile widely accepted and used as Software
Development (SWD) approach for more then 10 years
even in complex and traditional environment, see
adoption surveys
l Ambler 2005-2009
l VersionOne 2009, 2010
l Forrester 2008, 2009
23. And it works!
Agile and Lean success evidence
¢ Agile has better success rate then traditional approaches
• Standish Chaos research 1999, 2001, 2004, 2006, 2009
• Adoption surveys: Ambler 2007, 2009, VersionOne 2009
¢ Agile maintenance empirical studies, experiment evaluations
l Simplified source code by over 40% and improved software quality
(defect reduction) by 67%
• Poole, Huisman (2001) Using XP in a maintenance environment
• Poole, Murphy, Huisman, Higgins (2001) Extreme maintenance
• Alshayeh & Li (2005) An empirical study of system design instability metric…
• Dinh-Trong, Bieman (2005) A replication case study of Open Source development…
l Reduced traditional maintenance stuff by 40%
• Poole, Huisman (2001), Poole, Murphy, Huisman, Higgins (2001)
¢ Toyota (lean): people centric approach to car manufacturing and
Software development
• Liker (2003), Ohno (1988), Poppendieck (2006)
25. Let’s apply the lessons from SWD
to support and maintenance
Support & Maintenance status
Software Development status
Ad hoc Rigorous Agile/Lean
26. Short Agile Introduction
¢ Agile manifesto
1. communication and cooperation among all people (business and IT;
within IT teams);
2. focus on continuous delivery of valuable software;
3. working software as the primary measure of progress;
4. change tolerance understood as an advantage for the customer
¢ Agile approach:
l Applies those common principles
and values
28. Agile Maintenance:
State-of-the-art
¢ Maintenance in-built in Agile
l if Agile development is followed: ongoing development and
maintenance à no need for separate maintenance processes, teams
à better architecture and business knowledge à quicker ability to
change, cheaper maintenance
¢ Two levels of application and research
l Level 1: Methods – mostly XP, Scrum, Lean/Kanban
• XP, Scrum: Poole & Huisman (2001), Svensson & Host (2005), Alshayeh & Li (2005)
• Open Source Agile methods: Berglund, Priestley(2001)
l Level 2: Practices
• Rudzki, Hammouda, Mikkola (2009)
• de Souza et al. (2007)
l Our approach: customizable framework, focus also on support, goes
beyond software only
¢ No papers on application of Agile in support (SD, IM, PrM) found
29. Let’s get back to our
story …
¢ Team sits together and think during regular
retrospective
l How to get more sophisticated development tasks, not
just simple ones?
l How to improve current situation?
l What should we do differently?
l What can we do proactively without customer order?
30. Investments and
proactive approach
Actions team performed
¢ Measure existing and predicted trends
¢ Proposal of Java rewrite
¢ Comparison of both solutions
(technical one: performance, reliability, HW fit, …)
¢ Presentation to the customer
l Failed!!!
l Customer did not understand the value and reason why to invest in
rewriting
31. Investments and
proactive approach (round 2)
Actions performed in next round
¢ Better monitoring à upcoming issues predicted based on real,
more exact data
¢ Proof of Concept in Java (100 man-hours investment)
l Real time processing
l No new HW needed, less HW requirements for this technology
l More error resistant
¢ Measuring current solution and Java pilot
l Comparison prepared
¢ Presented to customer second time in money speech
l Customer got the point finally, but put the solution in backlog
l Soon accepted without any push when the first predicted issue
become true (in approximately 6 weeks)
32. Summary
¢ Proposal accepted in deep financial crisis in 2009
¢ Benefits for the customer:
l Customers core business process not affected
l Maintenance costs lowered
l Less incidents in few weeks
¢ Benefits for the team and vendor
l 160000 EUR new development project (higher vendor Net Sales)
l Higher team motivation: they prepared proposal and implemented
the solution (creative work), maintaining their work
l Incident decrease – less firefighting, no on call disturbances (out of
office hours)
33. Results
0
1
2
3
4
5
0
50000
100000
150000
200000
bře.09 dub.09kvě.09 čvn.09 čvc.09 srp.09 zář.09
Delepomentprojectcost
Motivation
bře.09 čvn.09 zář.09
Development projects 8300 7500 160000
People motivation 2,7 2,7 4,1
Service - telco
Category of measure Current value Proof of Concept
value
Mean processing runtime (median) 04:49:21 01:41:16
Cumulated monthly runtime 154 h 54 h
Runtime per 1000 invoices 00:06:12 00:02:10
Invoices processed per second 2.69 4.44
35. Framework goals and
structure
¢ Holistic approach
l Connected to Agile and non-agile development framework
• Extended OpenUP/RUP framework by Production phase
• Kroll, Kruchten (2003), Kroll, MacIsaac (2007)
l Focus on maintenance and support
l Full set of Agile and Lean practices as baseline for future process,
not only adjusted Scrum or XP to fit maintenance and support needs
l Respecting human nature
¢ Consists of:
1. principles we defined based on existing research and empirical
experience
2. set of (common) practices helping to reach each principle in practice
3. control framework with measuring principle
4. implementation approach
36. Framework respecting
human nature
¢ Cognitive, motivation and typology research as a basis
1. Building block 1: Context is important
• Leadership and communication
Tabaka (2006) Collaboration Explained, Aldair (2009) Leadership and Motivation
2. Building block 2: People are different
• Typology: Mayers-Bryggs Type Indicator (MBTI)
Keirsey, Bates (1984) Please Understand Me I. and II
3. Building block 3: Internal motivation matters
• Herzberg (2003) One More Time: How Do You Motivate Employees?
Hackman, Oldman (1980) Work Redesign
Melnik, Maurer (2006) Comparative Analysis of Job Satisfaction in Agile and Non-agile
Software development Teams
Melnik, Maurer (2007) Job Satisfaction and Motivation in a Large Agile Team
Whitworth, Biddle (2007) Motivation and Cohesion in Agile Teams
Nordström, Ridderstrale (2008) Funky Business forever
37. Framework respecting
human nature: key features
¢ Incremental adoption of Agile/Lean practices to avoid change
resistance
¢ Small Kaizen steps implemented towards ideal solution to overcome
amygdala warning reaction
l E.g. short pair work intervals, regular attempt to write source code
documentation, two times per day help others (leaving our comfort
zone)
¢ Change covers vendor, team but also personal vision
(manager’s role is to ensure this)
¢ Simple mechanism for immediate stop and root cause identification to
fix quality issues
l E.g. pair work or 5 whys
¢ Automation as key support for learning of new practices
l E.g. Continuous integration, Test driven development, pair work
¢ Learning and feedback mechanism to achieve mastery
l E.g. retrospective, rotation, slack space
39. Principle 1: More discipline,
less bureaucracy
¢ Benefit: ability to react; commitment to rules; team’s contribution;
motivated people; fun at work
¢ Pattern: rules defined with cooperation of teams; no detail
activities, room for creativity; automated basic rules; regular
retrospectives and rule changes; flat hierarchical structure
¢ Anti-pattern: detailed activities mandatory; leave the rules without
changes for years; add new rules continuously; do not leave any
decision power on the team but give them all the responsibilities.
¢ Symptoms: long lasting approvals; not followed enterprise
business system (processes); a lot of regular reports; process
oriented metrics; faking reports; unproductive meetings;
decreasing technical team motivation
¢ Practices: iterative approach, kanban, retrospectives, rotation,
visualization, continuous integration
40. Principle 2: Internal and cross
team cooperation and systematic
view
¢ Benefit: fulfilled needs; informed users; meeting business goals;
contribution to the business value creation; fun at work
¢ Pattern: share goals; map goals to IT; define metrics from the
business goals; rotate people; regular demonstration
¢ Anti-pattern: defy business side, they do not know programming;
carry off information; establish different teams for development and
maintenance of the services
¢ Symptoms: role oriented way of working; knowledge and
information transfer via documents; unknown business goal in the
teams; sub-optimization; unknown service or delivery status; 90%
done syndrome; IT understood as the cost
¢ Practices: visualization, business scenarios, pair working, rotation,
on-job learning, retrospective, fight with ambiguity
41. Measuring principle example
Business
objective
Contributing
operational IT
objectives
Operational objective
measures
Set of relevant
agile practices
Increase the
turnover per one
customer by 10%
until the end of
this year
Improve the quality
Number of critical defects
(trend!)
Ratio open/closed incidents
Trend of repeating incidents
Test coverage in %
Test driven approach
Iterative approach
Continuous integration
Pair working
Increase productivity
Team velocity (trend, not
exact number!)
Mean time to repair (MTTR)
Retrospective
Rotation
Pair working
On-job learning
42. Principle 3: Proactive
behavior and learning
¢ Benefit: satisfied customer; business supported by appropriate IT
services; new business opportunity, new business orders from the
customer
¢ Pattern: understand the business, propose future evolution
according to the business scenarios; analyze, simulate and
propose solutions; prototype and measure proposals
¢ Anti-pattern: only solve incidents and minor problems when
occur; do not ask for business evolution; do not propose solutions,
customer will not pay them
¢ Symptoms: unfamiliarity of customer; fire fighting; no time to do
things right; vendor does not solve (propose solutions to)
customer’s daily problems
¢ Practices: iterative approach, retrospective, rotation, visualization,
business scenarios, on-job learning, fight with ambiguity
43. Principle 4: Risk driven
approach
¢ Benefit: mitigate risks; plan the service according the existing
constraints
¢ Pattern: fact identification; risk identification and given action
proposal including assignee and deadline (actions performed as
real work - integration, development, prototyping) instead of a
detailed theoretical analysis or monitoring only
¢ Anti-pattern: ignore surroundings, changes; solve problems and
risks when they appear; conduct detailed analysis before real
implementation, integration, simulation, tests; only monitor the
appearance of risk
¢ Symptoms: reactively driven service; surprises during operations;
facts interpreted as risks; no time to do things right
¢ Practices: iterative approach, retrospective, business scenarios
44. List of the practices to support
implementation of principles
¢ Iterative approach/kanban
¢ Pair working
¢ Rotation between development and
maintenance/support teams
¢ Test driven approach and unit testing
¢ Refactoring to incrementally clean and
improve the code and architecture
¢ Daily meetings to synchronize in the
team, share solutions, share service
status and identify problems early
¢ On-job learning to gain knowledge and
experience quickly
¢ Business scenario simulation – to
adapt IT services to evolving business
and be ready for cost efficient future
solutions
¢ Retrospective for learning
¢ Continuous integration to get quick
feedback and improve the quality of
the code/service
¢ Defensive programming to avoid
common defects and have easier
maintenance
¢ Visualization to uncover problems
or trends, e.g. burn down chart,
kanban, value stream analysis
¢ Fight with ambiguity (memory
heuristic, keyword technique, fuzzy
sets and fuzzy logic) to deal with
different languages of business and
IT people.
45. Implementation approach
1. Analysis – kaizen workshop,
Theory of Constraint Current
Reality Tree (CRT), questioning
2. Selecting practices based on
measuring principle
3. Roadmap definition based on
priorities and risks
4. Hands-on daily support
5. Regular objectives check
46. Example results
Service X – forest
Service Y – telecommunication (our story)
Short
background
Data warehousing and ETL processing (200
workflows) supporting billing business
process;
4 people in service, 5 maintaining
Informatica platform (mixed teams in Czech
and Finland)
L2 and L3 support and maintenance team, also
developing small change requests; invoice
processing in batches (XML and various
scripts); 4 people in Czech, 2 people in Sweden;
millions of invoices processed weekly
Perceived
problems (by
customer and
teams)
(1) Unknown SLA targets
(2) Recurring incidents
(3) Critical incidents caused by platform and
database
(1) Invoices processed in batches, failure
required restart of the whole batch
(2) Increasing (geometrically) processing time
causing problems with invoicing
(3) Very low motivation of the team
Achievements
after applying
our approach
(1) Identified dependencies and SLA times
thanks to early piloting in transition period
(2) 75% decrease of incidents in 5 months,
changed nature of incidents (to low impact
and priority ones)
(3) Doubled end user satisfaction
(1) New 160 000 EUR business for our
company (development projects for rewriting to
real time processing)
(2) Customer’s daily critical business problem
solved
(3) Increased team motivation (team having
also development projects)
49. Conclusion I.
¢ Agile and Lean approach – current cure for experienced problems
l Lack of proactive, innovative behavior
l Customer and team motivation
l Quality improvements
l Responding to changing environment
¢ Holistic approach
l Connected to Agile and non-agile development framework
l Focused on operations, maintenance and support
l Full set of practices as building blocks for future process
• Chosen based on business goals
l Respects human nature
• Approach designed based on cognitive science, typology and motivation
research
50. Conclusion II.
¢ Framework consists of:
1. principles we defined based on existing research and empirical
experience
2. set of practices helping to reach each principle in practice
3. control framework with measuring principle
4. implementation approach
¢ Verified in different industry domains and customers
l 13 services and projects in past three years
l 10 successful implementations
l 3 unsuccessful implementations
(not achieved milestones, agreed not to continue during regular check)
l Telecommunications, public, energy, forest industry
51. e-book to be published
¢ In 2011 published “Operate IT differently. Agile and Lean
operations, support and maintenance of information systems and
IT services” in Czech
¢ Currently in progress e-book on this topic in English
¢ Help me to put in needed topics by answering this simple form.
52. About
author
• Freelance Agile and Lean mentor
• 13 years in IT (developer, support, project manager, consultant)
• Also teaching and researching experience at University of Ostrava
• Presenting at international conferences
• E.g. ISD, Lean IT Summit, ICGSE
• Blogs and free e-books at www.differ.cz (in Czech and English)
• Linked in contact