White paper on the impact of DevOps principles on the data delivery pipeline and continuous data testing. Read about:
- DevOps tools
- Terms in the DevOps world
- Test automation in the DataOps pipeline
How the DevOps Movement Can Impact your Company's Data
1. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 1
How the DevOps Movement
Can Impact your Company's Data
Published on July 24, 2019
DevOps is one of the hottest trends in the software industry and is the goal of most progressive IT
organizations (see chart below, courtesy of Google Trends). DevOps (short for development and
operations) is a set of automated software practices that combine software development (Dev), testing
and IT operations (Ops) to shorten the software development life cycle while delivering features, fixes,
and updates frequently in alignment with the business’ objectives.
Bill Hayduk
Founder & CEO of RTTS
2. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 2
DevOps is typically cross-functional (people from different areas) and uses different software tools.
These tools usually fit into one or more of the following categories:
• Coding – code development and review, source code management tools, code merging
• Building – continuous integration tools (like Jenkins), build status
• Testing – continuous testing tools (like QuerySurge, Selenium, Cucumber, JMeter) that provide
feedback on business risks
• Packaging – artifact repository, application pre-deployment staging
• Releasing – change management, release approvals, release automation
• Configuring – infrastructure configuration and management, infrastructure as code tools
• Monitoring – applications performance monitoring, end-user experience
While we're at it, let's add a couple more terms to the DevOps movement:
Continuous Integration (CI) Continuous Integration is about automating build and test processes to make
sure the resulting software is in a good state, ideally every time a developer changes code. CI helps
development teams avoid integration issues where the software works on individual developers’
machines, but it fails when all developers combine their code.
Continuous Delivery (CD) Continuous Delivery goes one step further to automate a software release,
which typically involves packaging the software for deployment in a production-like environment. The
goal of CD is to make sure the software is always ready to go to production, even if the team decides not
to do it for business reasons.
Continuous Deployment (also CD) Continuous deployment goes one step further than continuous
delivery. With this practice, every change that passes all stages of your production pipeline is released to
your customers. There's no human intervention, and only a failed test will prevent a new change to be
deployed to production.
Continuous Testing One of the hottest buzz terms in the testing world, continuous testing is the process
of executing automated tests as part of the delivery pipeline to obtain immediate feedback on the
business risks associated with a release candidate. Continuous testing cannot be implemented without
test automation.
DevOps principles demand strong interdepartmental communication and rely heavily on automation
tools.
And now the movement to incorporate a DevOps-type of automated process for data has grown
stronger. These practices are often referred to as Data DevOps or DataOps and apply DevOps tools and
techniques to data. Data is growing geometrically and applying automation to develop, deploy and
validate/test the data is becoming more critical, as businesses are implementing BI & Analytics to make
sense of their data and to leverage it in hopes of providing a competitive advantage.
4. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 4
Tests in DataOps have a role in both the Value and Innovation Pipelines. In the Value Pipeline, tests
monitor the data values flowing through the data factory to catch anomalies or flag data values outside
statistical norms. In the Innovation Pipeline, tests validate new analytics before deploying them.
DataKitchen
Comparing DevOps to DataOps process - from DataKitchen
Many of our QuerySurge customers have been requesting that we provide more DevOps-type features
for validating their data pipeline and fitting into their CD automation processes. So we just released our
new DevOps solution, QuerySurge for DevOps, which now automates data validation of Big Data, Data
Warehouses, Enterprise Apps and Business Intelligence reports with full DevOps functionality for
continuous testing.
5. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 5
With our new QuerySurge for DevOps release:
• Testers can now dynamically generate, execute, and update tests and data stores utilizing API
calls
• Teams have access to 60+ API calls with almost 100 different properties
• QuerySurge integrates with virtually all DevOps solutions in the marketplace
QuerySurge for DevOps' API features include the ability to create and modify:
• source and target test queries
• connections to data stores
• tests associated with an execution suite
• new staging tables from various data connections
• custom flow controls based on run results
Below are some sample Use Cases that can be utilized in a DataOps pipeline with QuerySurge for
DevOps:
7. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 7
The use cases are practically endless as QuerySurge for DevOps provides the flexibility to integrate your
data testing process into your existing DevOps implementation.
We are now in a full Data DevOps or DataOps movement to validate and automate the testing of the
data pipeline. We're just at the beginning of this movement but we are all-in on building the right
solution into QuerySurge to automate the continuous testing process.
8. copyright Real-Time Technology Solutions, Inc. August 2019 www.rttsweb.com page 8
About RTTS - The Software Quality Experts
RTTS, the developer of QuerySurge, is the
premier pure-play QA & Testing
organization that specializes in Test
Automation.
Headquartered in New York, RTTS has had
1,000+ successful engagements at over 700 corporations since 1996.
RTTS focuses on improving software & data quality through test automation. We create
QA strategies, provide software (QuerySurge) and implementation services targeted at
the data space.
To learn more about RTTS, click here>>
About QuerySurge
QuerySurge is the software division of RTTS.
QuerySurge is the smart Data Testing solution that
automates the data validation & testing of Big Data, Data
Warehouses, and Business Intelligence Reports.
QuerySurge will help you:
• Improve your data quality at speed
• Dramatically increase data validation coverage
• Detect data issues early in the delivery cycle
• Leverage analytics to optimize your critical data
• Provide a huge ROI
But don’t believe us (or our clients). Try it for yourself.
Check out our free trials and great tutorial