DevOps 2021 Research

IT & DATA MANAGEMENT RESEARCH,
INDUSTRY ANALYSIS & CONSULTING
Stephen Hendrick
Research Director, Application Development & Management
shendrick@emausa.com
DevOps 2021
Integrating Dev and Ops Through Analytics

Watch the On-Demand Webinar
Slide 2
• DevOps 2021 On-Demand Webinar:
https://info.enterprisemanagement.com/devops-2021-webinar-ws
• Check out upcoming webinars from EMA here:
https://www.enterprisemanagement.com/freeResearch

Featured Speaker
Slide 3 © 2020 Enterprise Management Associates, Inc.
Steve Hendrick, Research Director
Steve has over 20 years of experience as an industry analyst and 10 years
of experience across a variety of application development roles. He
specializes in application development and deployment topics, including
DevOps, lifecycle management, application platforms, decision analytics,
middleware, and the broad spectrum of tools and technologies that support
professional developers and IT managers. Steve brings advanced
quantitative and qualitative research techniques as an industry analyst,
providing vendors with deep insight into the markets where they compete.

Sponsors
A Thank You to Our Sponsors.
Sponsors make this research possible.

Agenda
• Research Objectives & Methods
• DevOps Maturity, Objectives, and Challenges
• VSM Adoption, Impact, and Vendor Leadership
• OA Adoption, Challenges, and Vendor Leadership
• TEM Use, Impact, and Vendor Leadership
• Conclusions and Recommendations

About DevOps 2021 Research
Project Objectives
Primary Objectives
• Understand current and planned DevOps tool category use across 20 DevOps market
segments
• Identify DevOps maturity, objectives, and challenges at large enterprises
Secondary Objectives
• For select DevOps markets, identify products in use, product penetration, feature
effectiveness, satisfaction, and enterprise value
• Leverage quantitative and structured customer feedback to identify the Top 3 vendors and
the leading innovators in specific DevOps markets

About DevOps 2021 Research
Survey Parameters
• Sample Size: 400
• Geographic Scope: Worldwide
• Survey Fielding: August 2019
• Target Respondent: IT professionals with experience in the SDLC, DevOps, app
release, dev/test environment mgmt, app development, & operations mgmt
• Enterprise size: 2,500 employees or more
• Application development participation: Material enterprise involvement
• Vertical industries: All other than education and IT vendors
• Respondent expertise: Strong familiarity and involvement with DevOps
• DevOps market focus: Value stream management, operational analytics, test
environment management, software testing

INDUSTRY ANALYSIS & CONSULTINGSlide 8 © 2020 Enterprise Management Associates, Inc.
Total Enterprise Employees
Q4: Please estimate the total employees your organization has worldwide
1,000 to 9,999, 39%
10,000 to 19,999,
27%
20,000 to 29,999,
22%
30,000 or more,
14%
DevOps 2021 Survey, September 2019, N=400

Application Development Maturity
Q5: What response best describes your enterprise’s approach to application development?
We develop custom
applications to
support our internal
informational needs
only, 21%
We develop a
moderate number of
custom applications in
support of internal
and external
informational needs,
40%
We develop many
custom applications
that impact nearly
every activity in our
enterprise, 39%

Involvement in DevOps
Q11: How would you describe your enterprise’s involvement in DevOps?
6%
22%
30%
28%
14%
0% 5% 10% 15% 20% 25% 30% 35%
We have not started to automate any activities in our
DevOps pipeline
We are just beginning to automate one or two activities in
our DevOps pipeline
We have automated some of the key activities in our
DevOps pipeline
We have automated many of the activities in our DevOps
pipeline
We have automated nearly all activities in our DevOps
pipeline
Percent responding
52%, Foundational and Mainstream
42%, Modern and Optimized

DevOps Pipeline Deployment
Q13: Where have you deployed your DevOps pipeline? (single response)
44%
22%
14%
11%
8%
0% 10% 20% 30% 40% 50%
In a hybrid combination of a public cloud and our own data
center
In a public cloud using dedicated resources
In a public cloud using multitenant resources
As a hosted service though a managed service provider
In our own data center
Percent responding

IT Tool Use
Q17: What is your organization’s approach to IT tool use? (select all that apply)
39%
38%
31%
29%
28%
24%
19%
14%
11%
9%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
We have a high level of visibility into the time and cost to deliver
a feature or address a defect
There is some integration between our portfolio of IT tools based
on highest need
We have limited visibility into the time and cost to deliver a
feature or address a defect
We proactively define the interfaces between most of our IT tools
Our mix of IT tools is often changing as we position to address
new business opportunities
There is little integration between our portfolio of IT tools
We rely on a heterogeneous best-of-breed mix of tools
Our mix of IT tools remains mostly stable
We have no real visibility into the time and cost to deliver a
feature or address a defect
We rely on a more homogeneous best-of-suite collection of tools
Percent responding
42%
39%

Top Three DevOps Objectives
Q22: What are the top three objectives that your enterprise wants to accomplish through DevOps?
4%
5%
14%
15%
19%
21%
24%
4%
10%
11%
22%
12%
18%
23%
15%
12%
5%
16%
13%
21%
19%
0% 10% 20% 30% 40% 50% 60% 70%
Improve the performance of applications
Improve the quality of applications
Improve IT productivity and reduce cost
Improve the customer experience
Improve our agility in addressing changing market needs
Improve the reliability of applications
Improve the speed of software delivery
Percent responding
First
Second
Third

Top Three DevOps Challenges
Q23: What are the top three DevOps challenges that your enterprise now faces?
3%
3%
6%
7%
7%
7%
8%
9%
10%
19%
21%
7%
2%
2%
2%
8%
8%
17%
8%
6%
18%
7%
8%
8%
8%
7%
8%
11%
8%
8%
8%
2%
12%
7%
8%
8%
8%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Figuring out how to add or integrate new tools into our toolchain
Determining when to implement a DevOps for database strategy
Making a commitment to DevOps because other business commitments get in the way
We don’t have the expertise needed to pursue additional IT automation and DevOps
activities
Finding a way to fully test applications given the faster speed of application development
Knowing where to start in building a DevOps toolchain
We are using DevOps to deliver applications to staging, but legacy governance and
compliance policy prevents us from moving to production in a timely manner
Deciding when to throw out our cobbled-together toolchain for something more
comprehensive and better integrated
Deciding what knowledge and tools developers and IT Operations should share
Getting Ops staff interested in helping define and configure our DevOps strategy
Changing the way our staff does application development and deployment
Setting up and managing Dev and Test environments
Dealing with the impact of project scope changes on our pipeline
Percent responding
First
Second
Third

DevOps Maturity by Market Segment
Comparing DevOps 2020 and DevOps 2021 Tool Use
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentresponding
2019 Q3 2018 Q2
Release and Ops focused segments,
lower penetration, higher growth: 50%-
250% growth between Jun 2018 and
Aug 2019.
Dev focused segments, rapid SW
development, higher penetration, lower
growth: 4%-28% growth between Jun
2018 and Aug 2019.

INDUSTRY ANALYSIS & CONSULTINGSlide 16
Value Stream Management

VSM Involvement
Q44: To what degree is your organization involved with VSM?
3%
1%
29%
25%
13%
18%
13%
1%
0% 5% 10% 15% 20% 25% 30% 35%
Our IT organization has no interest in value stream management
(VSM)
Our IT organization is not familiar with the term
Our IT organization is familiar with the term, but we have not
begun thinking about what kind of solution we want
Our IT organization is beginning to evaluate what VSM can do for
us
Our IT organization has some aspects of VSM in place and is
working on building out a more complete solution
Our organization has enough experience with VSM to understand
its value and is using it for some products
Our IT organization is fully onboard and using a value stream
management approach for most products
Don’t know or not sure
54% Just Starting
31% Beginning
13% Advanced

Why Use VSM Tools
Q46: Why does your enterprise use or plan to use VSM tools? (select all that apply)
55%
54%
50%
48%
45%
45%
38%
35%
34%
33%
33%
32%
32%
31%
30%
30%
29%
29%
0% 10% 20% 30% 40% 50% 60%
It helps us eliminate delays
It helps us improve product quality
It helps us standardize how we design, develop, deploy, and operate products
It provides us with a way to standardize the governance of our pipeline(s)
It forces us to focus on how best to support our customers
It helps us improve product performance
It helps us understand where steps in our value chain can be improved
It helps us improve staff productivity
It provides a single unified view of our value streams
It enables us to connect features to builds to environments
It helps us improve product support
It helps us manage the explosion of tools that we are using
It helps us correlate & synchronize data across the tools in the toolchain
It helps us provide transparency and visibility into our development activities
It helps us reduce product cost
It provides a better way to manage & govern product dev & app delivery
It helps us manage digital transformation and show progress being made
It makes our heterogeneous use of tools and methodologies easier to manage
Percent of current or planned VSM tool users

Impact of VSM on the Enterprise
Q52: What impact is value stream management having on your company?
2%
0%
7%
39%
35%
16%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
No impact
Minimal impact
Some impact
Moderate impact
Significant impact
Transformative impact
Percent of VSM Users

Current/Planned VSM Tool Use
Q49: What VSM tools does your company use or plan to use?
42%
26%
21%
19%
16%
14%
13%
12%
11%
11%
9%
9%
9%
8%
6%
5%
4%
4%
3%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
An internally developed solution
Microsoft/Excel
XebiaLabs/DevOps Platform
Plutora
CollabNet VersionOne
CloudBees/Flow (from Electric Cloud)
CA Technologies
IBM/UrbanCode Velocity
GitLab/GitLab
Tasktop
Jama Software/Connect & Analyze
CloudBees/DevOptics or Flow
Targetprocess
Panaya/Release Dynamix
Intland Software/codeBeamer ALM
Blueprint Software Systems/Storyteller
Numerify
ServiceNow
Atlassian/JiraAlign
Don't know or not sure
Percent valid cases of current or planned VSM tool users

Primary VSM Tool Use
Q50: What primary VSM tool does your company use?
22%
22%
16%
11%
8%
5%
5%
3%
2%
2%
1%
1%
1%
1%
0% 5% 10% 15% 20% 25%
Plutora
XebiaLabs/DevOps Platform
Microsoft/Excel
CollabNet VersionOne
CA Technologies
Targetprocess
CloudBees/DevOptics or Flow
Jama Software/Connect & Analyze
GitLab/GitLab
IBM/UrbanCode Velocity
Intland Software/codeBeamer ALM
Panaya/Release Dynamix
Blueprint Software Systems/Storyteller
Percent of current VSM tool users

Methodology – Empirical Product Analysis
1. Vendor evaluations were 100% empirically driven by a worldwide sample of 400 large
enterprises
2. Products evaluated must have at least a 1% product penetration
3. Products evaluated were identified as the primary product in use by the enterprise
4. Four criteria factored into product evaluations
 Enterprise value
 Product capabilities (7 components)
 Product satisfaction
 Product penetration
5. A product’s raw score for each criteria is the average of valid responses except for
product penetration which is a summation
6. A product’s raw score for each criteria is expressed as a market percentile score
7. A product’s overall market score is the average of their four percentile scores

Vendor Recognition
1. EMA supports two award categories: Top-3
awards and Innovation awards
2. All scoring is based on empirical evidence
from large enterprises as defined by our
methodology
3. A Top-3 award is granted to the three
market-specific products with the highest
overall scores
4. An Innovation award is granted to the
leading market-specific product in each of
these four categories:
• Enterprise value
• Product capabilities
• Product satisfaction
• Product penetration
Market Name
Vendor Name/Logo
INNOVATION AWARD
DEVOPS2021 VENDORNAME
MarketName
Measure

Value Stream Management Market: Vendor Awards
Top-3 Awards
• Plutora
• XebiaLabs
• CollabNet
Innovation Awards
• XebiaLabs: Enterprise Value
• CollabNet: Product Capabilities
• Plutora: Product Capabilities
• Targetprocess: Product Satisfaction
• Plutora: Product Penetration
Market Name
Vendor Name/Logo INNOVATION AWARD
MarketName
Measure
A VSM Empirical Product Analysis report
discussing the market, award winners,
and scores to follow in February 2020

Operational Analytics

Reasons for Using an OA Tool
Q62: Why does your enterprise use or plan to use operational analytic tools?
49%
47%
46%
45%
45%
35%
35%
35%
34%
33%
33%
31%
30%
2%
0% 10% 20% 30% 40% 50% 60%
It helps us avoid hardware and software failures
It helps us improve performance
It helps us improve quality
It helps us eliminate delays
It helps us improve staff productivity
It helps us reduce cost
It helps us understand where steps in our value chain can be improved
It helps us standardize how we design/develop/deploy/operate products
It provides visibility into the operation of our infrastructure hardware
It provides visibility into the operation of our software processes
It is a necessary step in moving us to the use of AI and ML
It enables us to better support customer needs
It provides a single unified view of our machine data
Percent valid cases of current or planned OA tool users

Current/Planned OA Tool Use
Q64: What OA tools does your company use or plan to use?
25%
25%
24%
18%
18%
13%
9%
8%
8%
8%
7%
6%
6%
6%
6%
5%
5%
4%
0% 5% 10% 15% 20% 25% 30%
Splunk
Sumo Logic
AppDynamics
XebiaLabs/XL Impact
Prometheus
New Relic
ITRS Group/Geneos
Moogsoft
Nagios
Dynatrace
Elastic
Loggy
Numerify
Zabbix
Datadog
Logstash
Fluentd
Valid cases of current or planned OA tool users

Primary OA Tool Use
Q65: What primary OA tool does your company use?
23%
20%
19%
9%
8%
5%
3%
3.0%
3%
2%
2%
2%
1%
1%
1%
0% 5% 10% 15% 20% 25%
Sumo Logic
Splunk
Prometheus
XebiaLabs/XL Impact
Moogsoft
Datadog
Elastic
Dynatrace
New Relic
Loggy
AppDynamics
ITRS Group/Geneos
Fluentd
Zabbix
Valid cases for current and planned OA tool users

Operational Analytics Market: Vendor Awards
Top-3 Awards
• Splunk
• Sumo Logic
• XebiaLabs
• Prometheus
Innovation Awards
• XebiaLabs: Enterprise Value
• Prometheus: Product Capabilities
• XebiaLabs: Product Satisfaction
• Sumo Logic: Product Penetration
Market Name
MarketName
Measure
An OA Empirical Product Analysis report
discussing the market, award winners, and
scores to follow in February 2020

Test Environment Management

Why Use a TEM Tool?
Q28: Why does your enterprise use or plan to use a TEM tool?
49%
29%
28%
27%
27%
26%
24%
24%
18%
2%
0% 10% 20% 30% 40% 50% 60%
We need to reduce the amount of time spent on coordinating and
communicating schedules
We need to improve the availability of our test environments
We need a system of record for managing our test environments
We need to accelerate the handoff from dev to test
We need cost controls
We need to improve the reliability of our test environments
We need to know who was using our test environments
We need to move beyond spreadsheets or a homegrown solution
We need more transparency into test environment usage
Percent responding

Distribution of Outage Root Causes
Q33: Across these outages, what is the average distribution of root causes?
33%
31%
18%
12%
7%
0% 5% 10% 15% 20% 25% 30% 35%
Configuration problems and software stack
Deployment failures (wrong builds or incorrect deployment)
Coding or programming bugs
Configuration problems with hardware
Data or database access problems
Root cause averages

Primary TEM Tool Use
Q30: What is the primary TEM tool that your company uses?
26%
20%
17%
16%
10%
7%
3%
2%
0% 5% 10% 15% 20% 25% 30%
Plutora/Environment Management
TEMS/Omnium Lite
Apwide/Apwide
Other
Microsoft/Excel
Micro Focus/Quality Center
Percent of current TEM tool users

Test Environment Management Market: Vendor Awards
Top-3 Awards
• Plutora
• Apwide
• TEMS
Innovation Awards
• Apwide: Enterprise Value
• Micro Focus Quality Center: Product
Capabilities
• Apwide: Product Satisfaction
• Plutora: Product Penetration
Market Name
MarketName
Measure
A TEM Empirical Product Analysis report
discussing the market, award winners,
and scores to follow in February 2020

Conclusions and Recommendations

Conclusions
The 1st 10 years of DevOps:
• Accelerating software delivery was a key objective (slide 13)
• Implementing the MVP (minimum viable pipeline) (slide 15)
• High penetration of SCodeM, MT, DTE, TEM, SCM, CI/CD, ARA, CTP, CDI,
BM*suggests the right tools are in place to achieve this (slide 15)
• Fast software delivery now occurring at 40% of large enterprises (slide 10)
Visibility into time/cost to address features/defects remains an issue
• Enterprises split 50/50 on whether they have high or limited visibility (slide 12)
Three most important DevOps objectives
• Improve SW delivery speed, improve app reliability, improve the UX (slide 13)
• Dealing with scope change and dev/test issues persist (slide 14)

Conclusions
The next 3-5 years of DevOps: putting the Ops in DevOps
• Growth in app RM, DB RM, testing @ scale, OA, ARRM, & VSM are all data-driven
methods to better enable SW delivery, deployment, and operation.
• Portfolio capabilities, where supported, could drive risk management and optimization
Value Stream Management
• Only 13% of enterprises have fully embraced VSM (slide 17)
• VSM expected value: eliminates delays, improve quality, standardize processes,
demands a customer focus, improves performance (slide 18)
• 51% of large enterprises say VSM impact is significant or transformative (slide 19)
• Penetration of internally developed solutions (42%) and Excel (26%) exists because of
missing market-specific tool capabilities. (slide 20)
• VSM tools beginning to offer functional depth, integration, scale, and SoR features
(slide 21)

Conclusions
Operational Analytics
• Expected benefits: avoid software failures, improve performance, improve quality,
eliminate delays, improve staff productivity, supports value chain improvements (slide 26)
• Increased maturity of market-specific tools limits appeal of internally developed solutions
(slide 27)
Test Environment Management
• Managing TEs can be enormously resource intensive (slide 31)
• The root cause of most outages is due to configuration problems or deployment failures
not coding or data access issues (slide 32)
• Primary TEM tool use shows some Excel but almost no internally developed solutions
(slide 33)

Recommendations
Building out your DevOps strategy
• Determine where you are in tool adoption
• Most enterprises plan to have a comprehensive tool chain in place by 2023
• Consider tool acquisitions relative to pain points and opportunities for greatest net
benefit gains
• VSM and OA share instrumentation and analytic objectives. This provides an
opportunity to bring the SDLC and ITSM together
• Use VSM as a lever to make IT more product and customer focused
• Use OA to better manage production apps and systems
• Engage Ops to incorporate OA activities into VSM-based streams
• Analytics provides an opportunity to make IT customer and data-driven which
mitigates risk and facilitates management, process improvement, & optimization

Questions?
• Get the DevOps 2021 Research
https://bit.ly/2BwATRO

DevOps 2021 Research

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a DevOps 2021 Research

Semelhante a DevOps 2021 Research (20)

Mais de Enterprise Management Associates

Mais de Enterprise Management Associates (20)

Último

Último (20)

DevOps 2021 Research