We all know AI has a key role to play in analyzing and drawing insight from the vast amounts of data in our increasingly complex, interconnected, software-dependent world. This playbook explores how.

1. We all know AI has a key role to play in analyzing and drawing insight
from the vast amounts of data in our increasingly complex,
interconnected, software-dependent world, this playbook explores how.
The Ultimate DevOps
Playbook
The Voice of the User and the Future of AI in DevOps

2. Introduction
Introduction
Page 002
DevOps is an abbreviation for
Development and Operations. The
idea is that the people responsible
for creating software (Development)
collaborate more closely with the
people responsible for deploying and
maintaining it (Operations).
The aim is to ensure that software
can be delivered in a more agile,
responsive way, continuously
improving, based on user feedback
and changing market conditions.
But while the theory sounds great,
actually making it happen can be
incredibly complex. This is not least
because for DevOps to be truly
effective, there is an argument that it
needs to be all-encompassing; that it
needs to involve the entire business,
C suite to Marketing to Customer
Service. In other words, DevOps
needs to be regarded as a business
process, not purely a technical one.
Customer feedback and market
opportunities are key factors
in defining business strategy, but
DevOps is a technically oriented term
and focused on the teams who
build products and ensure they’re
available. On top of this, there may
be procedural and political barriers
within organizations that limit the
communication and effectiveness of
data flow between the two worlds of
Dev and Ops. And perhaps more
than anything, poor or no
communication is why any Agile or
DevOps methodology breaks down.
Another issue to consider is that
while the DevOps loop is great for a
single release in isolation, most
software is created through off-the-
shelf software or with integrators via
a REST API. Anyone testing in a
DevOps world needs to understand
that the third-party programs or
integrations they use may be using a
better or worse DevOps process,
Agile, Waterfall, or no methodology.

3. Why do we need DevOps?
In some ways, the need for DevOps
simply reflects the fact that software
has become such an integral part of
our everyday lives. We all expect new
features that address our needs at
an ever-increasing pace; and if we
can’t get that from one vendor, we
will find one who can.
Why do we
need DevOps?
Page 003
What’s more, that software is
becoming more interactive. Even a
simple website is now less likely to be
a repository for information and
more likely to be something that end
users engage with. Furthermore, the
market for smartphone apps has
become ever more competitive and
saturated. Consumers are fickle and
liable to uninstall apps that don’t
meet their needs. The ability to react
fast is therefore essential.
Finally, we live in a much more
connected world. Applications are
less self-contained, and increasingly
need to work with other
technology - think the Internet of
Things. Applications therefore need
to be updated as the technology
around them changes.
"The key to following the continuous delivery path is to
continually question your own assumptions about what’s
possible.” - Jeff Sussna

4. Challenges in making DevOps Work
Challenges in
making
DevOps Work
Page 004
QA is another area that feels the heat
when trying to adopt DevOps
practices, and testing is fundamental
to realizing the benefits promised by
DevOps.
All software needs to be tested
before it can be released. And when
release cycles are accelerated, this
can place enormous pressure on QA
teams. This is especially true given
the proliferation of ways in which
end users interact with software. For
example, a service may be available
as a website or an app. Both will
have to work on multiple platforms
on thousands of different devices.
They may also have to interact with
other websites, apps, or services,
such as payment gateways,
geolocation services, or review
services. The need to test more
permutations more frequently
We’ve already talked about the need for DevOps to be regarded
as a business process, rather than a purely technical one, and
there are challenges in making that happen. Getting buy-in from
marketing or customer service teams, for example, may require
organizational or structural change that will require C level
input to achieve.
means the number of possible test
cases increases exponentially.
Similarly, as we add more features,
we need to add more test cases and
also maintain an ever-increasing set
of tests—as the product line
matures, the load on the QA team
increases significantly.

5. Challenges in making DevOps Work
Automation is therefore key, with testing of functional and non-
functional requirements built into the development and release
process. The trouble is that this doesn’t always allow for
comprehensive testing. Exploratory testing can be a challenge, for
example. Exploratory testing involves using the software in ways
that weren’t necessarily anticipated by developers. Deviating from
the script in this way can be a useful route to uncovering bugs, as
well as being more representative of real user behavior, which
frequently fails to conform to what the developers originally had in
mind.
While automating exploratory testing
by taking random paths through an
application may be relatively
straightforward, a human tester
would do something more. On
finding a bug, a person would actively
look for common factors, trying to
identify something that would help
developers pinpoint the cause. They
would also be able to identify
whether something is truly “correct”
from the user’s standpoint, over and
Page 005
above a simplistic value returned
from an API.
This is where AI comes into play. If
automated testing can actively search
for bugs with a view to helping people
to identify root causes, it can both
take the brakes of the DevOps
process and make it much more
effective.
"AI is everywhere.
It's not that big,
scary thing in the
future.
AI is here with us."
- Fei-Fei Li

6. Shift Left
Shift Left
Page 006
The concept of “shift left” is
fundamental to DevOps. It means
that operations move closer to
development. The idea is that it
becomes possible very early in the
development lifecycle to understand
how the application will behave in
the real world. It also means that
bugs can be found and fixed as early
as possible. This is important
because it’s generally cheaper to fix
errors early in the development
lifecycle.
Continuous testing is key to shifting
left and involves running automated
tests that deliver immediate
feedback throughout the
development lifecycle. It is an
important part of continuous
integration, continuous delivery and
continuous deployment.
Continuous integration
In continuous integration, multiple
developers working on different
components are regularly
committing code changes to the
main branch, possibly several times a
day. The advantage is that each
change is validated in an automated
unit test as it is committed, allowing
any problems to be ironed out
straight away, rather than creating a
potential bottleneck on release day.
Continuous delivery
Continuous delivery takes
continuous integration a step further
by automating deployment to a test
or staging environment.
Continuous deployment
The holy grail of DevOps, continuous
deployment cuts out all human
intervention and, as long as a release
passes all tests, it is deployed to
production automatically.

7. Page 007
The importance of continuous
testing in DevOps has led to
the coining of the term
DevTestOps.
At the same time, another
term, DevSecOps, has also
gained some currency. This
reflects the need to embed
sound security practices into
the DevOps process.
It is no coincidence that both
new terms relate to potential
Shift Left
According to an IBM study, it is
100 times more expensive to fix
a defect that’s been found after
software has been released.
bottlenecks. DevOps is
all about speed, and the
temptation is to fail to pay
adequate attention to those
areas that have the potential
to slow things down. We’ve
already seen how automation
can help break through these
bottlenecks in testing. But it is
not just about technology. It is
also about culture and
ensuring that the right lines of
communication are in place.

8. Shift Right
Shift Right
Page 008
“Shift right” means taking certain
aspects of pre-production testing
and applying them post-production.
This is really about making sure that
the left hand is talking to the right
hand. Most live systems will employ
some kind of monitoring or analytics,
but this is not always closely tied in
with how applications are developed
and tested.
Shifting right addresses this through
innovations such as real user
monitoring data to create functional
test models, so that testing can be
based on how people actually use
the software, not how analysts think
they will use it or how developers
intend them to use it. Essentially,
then, part of shifting right involves
leveraging a huge, but under-used
resource in software testing – the
end user population.

9. Shift Up!
Shift Up!
Page 009
That brings us on to an alternative
concept—shift up—which has been
pioneered by Michael Giacometti,
Director of Partners and Alliances at
Eggplant. Shift up advocates bringing
the voice of the customer into the
testing and development process.
The point is that software testing
often fails to consider customer
experience, with tests tending to
focus on function and performance.
Crucially, an application could pass
such tests with flying colors and still
fail in terms of user experience.
Part of the issue is that software
doesn’t exist in a vacuum. Indeed, it’s
possible to view the end user as an
integral part of the end product. Each
one has different needs, wants, and
experiences that shape their view of
that product. And while this has
always been an issue, the
pervasiveness of software and the
fragmentation in its delivery have
conspired to make it harder to
develop applications that work for all
users all of the time.
One example of this is the explosion
in the number and type of mobile
and tablet devices, running different
software on different platforms and
with different capabilities, viewports
and processing power. At the same
time, the diversity in the range of
people using these devices has also
seen huge growth, both
demographically and geographically.
The upshot is that it is becoming less
and less possible to make
assumptions about the way people
will use your software. This means
that Shift Left and Shift Right are no
longer enough in and of themselves.
You need AI and predictive analytics to
help you understand how people are
using your software and feed that
insight into the development and
testing process, so you can focus on
and optimize business performance.
This is very much the new frontier of
DevOps. Because while it is possible
to speed up release cycles by
introducing automation, feeding the
voice of the voice of the customer
into that process is much harder. As
such, if you get everything else in
DevOps right, there’s a chance that
customer input becomes the new
bottleneck.

10. Page 010
Embracing shift up through AI-driven testing
“A major publishing company responsible for multiple
print and digital publications delivered to subscribers
around the world needed a new approach to QA. It
wanted to offer self-service features to its readers, as
well as customize offers based on reading habits and
locales. The system needed to be elastic enough to
quickly and seamlessly add new offerings.
“This required a way to test builds with new and
updated security features—alongside performance
testing of the site and digital content access—in a non-
invasive manner. By deploying an AI-driven test engine,
the company increased test execution by 300% and
conducted long-term testing across multiple platforms,
channels, and technologies. Optimizing the site resulted
in improvements in revenue and in customer
acquisition and retention.”
Why software development needs to shift up
(not left or right), Michael Giacometti,
TechBeacon, 19 Feb 2019
Fortunately, there are some things
we can automate. For example, we
can analyze how different groups of
customers use software. We might
not have demographic data, but we
might know what device they’re
using and where they’re located. And
we can correlate these with
demographics. We can also look at
the behavior of the user populations
that deliver the greatest business
value. Or those that don’t, but
should. And we can then start to use
that data to inform the development,
test and release process, even
automating test case production and
maintenance.
For example, can we predict how
changing an element in a key user
journey will impact business KPIs?
If so, we can begin to use AI and real
customer data to
drive development choices
automatically, prioritizing fixes and
enhancements that will deliver the
greatest business value.
We can argue, then, that the final
piece in the DevOps puzzle is big
data and using AI to analyze it.
DevOps was always supposed to be
about delivering more value to the
business. Up until now, this has
largely been achieved by adopting
agile processes and faster release
cycles. These are prerequisites but
they’re not enough on their own. It’s
like buying a faster car - it’s great,
and it does get us to our destination
more quickly. It’s just that we would
also really benefit from improved
navigation
Shift Up!

11. Continuous Feedback
Continuous
Feedback
Page 011
Earlier on, we looked at continuous
integration, delivery and deployment
in DevOps. There is also the concept
of continuous feedback, where the
impact of releases on customer
experience is regularly measured
and fed back into the development
process.
Customer feedback comes from
various sources. Some, such as
surveys and focus groups, tend to be
expensive and time consuming, and
don’t lend themselves easily to
automation. Others, such as social
sentiment and app ratings, are more
susceptible to automated analysis.
Indeed, given the volume of data and
the number of possible
sources, some kind of automated big
data analysis is almost required to
make sense of it all.
The challenge is automating this with
the help of AI in the same way that
we’re automating testing in
development. It’s arguable that
continuous deployment isn’t truly
continuous unless and until that loop
is closed.
It is possible to see a future where AI
analyzes real user interactions with
an application and automatically
prioritizes updates and fixes based on
business outcomes.
The example on the following page
shows how this sort of approach
could work.

12. Continuous Feedback
COOK, a business that delivers high-
quality frozen meals, wanted to
understand how the performance of its
website was influencing consumer
behavior. It then wanted to use that
insight to inform its development
priorities.
COOK used Eggplant’s synthetic
monitoring solution to identify a slow
server response time that could reach
six seconds, impacting several key
customer journeys.
At the same time, COOK was using
Performance Impact Prediction in
Eggplant’s real user monitoring solution,
which took all the data about visits to the
website and built a model of the
relationship between performance and
visitor behavior.
By entering a new load time based on
Case Study
COOK
Page 012
Analyzing the relationship between application
performance and business outcomes in order to inform
development priorities
what it could achieve by fixing the slow
server response, COOK was able to
predict changes in conversion rate,
bounce rate, and revenue for its entire
user population or individual segments
(e.g. Android users). This suggested that
fixing the problem would result in a
seven percent increase in conversions.
Based on what it learned from real user
monitoring, COOK prioritized fixing the
server response problem ahead of
introducing new functionality.
The resulting performance improvement
delivered exactly what was predicted: a
seven-percent increase in conversion
rate. At the same time, bounce rate fell
by seven percent and engagement
(pages per session) increased by 10
percent.

13. Continuous Feedback
Page 013
Monitoring identifies a
slow piece of
functionality in an app
It also shows that 3% of
iPhone users uninstall the
app after encountering it
...and 5% of Android users
uninstall the app after
encountering it
It also shows that 3% of
iPhone users uninstall the
app after encountering it
Those users account for
$150k lost revenue
Prioritize iPhone app
Fix Android app next
Taken to its logical conclusion, this
effectively leads to ‘self-healing
applications’. In some ways, this isn’t
such a great leap from where we are
now. Releases aren’t pushed to live
unless they pass automated tests. It’s
not going that much further to see
releases rolled back automatically in
response to poor user feedback.
Beyond that, it’s possible to envisage
systems that automatically identify
and fix the offending piece of code,
running simulations of all possible fixes
and anticipating real-user impacts.
None of this takes away from the fact
that most products have a start period,
a sustainment period, and an end-of-
life period. DevOps doesn’t change
that. Part of the communication and
analysis that goes into DevOps is about
ensuring that software is always being
tested for both quality and relevance.
One thing that’s clear is that AI has a
key role to play in analyzing and
drawing insight from the vast amounts
of data in our increasingly complex,
interconnected, software-dependent
world.
How automated analysis of real user behavior can help drive decision making in development

14. About eggplant
Shift Right
At Eggplant we help businesses to
test, monitor and analyze their end-
to-end customer experience and
continuously improve their business
outcomes. We provide business with
award winning software such as the
winner of Codie's Best DevOps tool
2019 - eggplant Digital Automation
Intelligence Suite.
Companies worldwide use Eggplant
to surpass competitors, boost
productivity, and delight customers.
How? By dramatically enhancing the
quality, responsiveness, and
performance of their software
applications across different
interfaces, platforms, browsers, and
devices—including mobile, IoT, and
desktop—in agile, DevOps, and
innovative application and data
environments.
We are a global company serving
more than 650 enterprise customers
in over 30 countries. Sectors include
automotive, defense and aerospace,
financial services, healthcare, media
and entertainment, and retail.
Eggplant is backed by The Carlyle
Group (NASDAQ: CG).www.eggplant.io sales@eggplant.io