1. DevOps for the Mainframe
Leveraging Continuous Integration, the cloud, and beyond to
deliver z/OS applications
Session SC-2402
David Myers, Product Manager, Cross-Brand DevOps
myersda@us.ibm.com
@Dave_Does
Rosalind Radcliffe, Distinguished Engineer, Enterprise
Modernization Solution Architect
rradclif@us.ibm.com

2. Please note
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract. The development, release, and
timing of any future features or functionality described for our products remains at our sole
discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,
and the workload processed. Therefore, no assurance can be given that an individual user
will achieve results similar to those stated here.

3. 3
Social, Local, Mobile
Smart Infrastructure Analytics
ERP
Legacy
DB
CRM
HR
Manage workloads
and maintain security
Rapid innovation
in the cloud
User experience and
mobile management
New era systems integrate existing operational
systems with rapid delivery of new client-facing apps

4. Evolving customer and market expectations
Capabilities and User Experience Today Emerging
Primary Workload Types
Systems of Record
Transactional
Systems of Engagement (+ Record)
Big Data, Analytics, Mobile/Social Channels
Time to Value Planned Opportunistic
Delivery Model Planned Incremental (DevOps)
Development and Operations Team Sizes 100s and Costly 10s with built-in DevOps automation
Release Frequency Months to Years Hours to Days, based on business opportunity
Integration Frequency Weeks Continuous
Service Sourcing Develop Consume and Assemble (Public and Private)
Operational Model Systems Management
Built in to application, Recovery Oriented Computing, Continuous
Availability
Infrastructure Deployment Days Minutes
Risk Profile Big-Bang (High Risk) Incremental

5. DevOps is…
A set of principles and values that facilitate collaboration
across disciplines to…
1. Enable rapid evolution of deployed business services
2. Reduce risk, decrease cost, and improve quality across the
portfolio
DevOps Principles
§ Collaborate across disciplines
§ Develop and test against a production-like system
§ Deploy frequently using repeatable and reliable processes
§ Continuously monitor and validate operational quality
characteristics
§ Amplify feedback loops
People
Process
Tools

6. Solution: A Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
• Validate on more production-like
conditions earlier
• Automate hand-offs/promotions to
increase velocity through the
different stages
• Standardization on processes and
assets between Dev and Ops
• Automated monitoring and
dashboarding of quality and
performance against service level
agreements at multiple stages
Ensure applications are production-ready throughout the lifecycle
and can be released at any time while minimizing rollback due to quality issues

7. Mainframe Delivery Pains…
§ Multiple teams working across
restricted dev and test capacity
lead to conflict, delays, or bad
test results in shared
environments
§ Complex and manual
management and configuration
tasks result in errors and delays
§ Too much bad code going into test
and production causes crit sits
and emergency fixes
§ Bottlenecks due to inefficient
communications between
disparate platforms and teams
(Dev/Test - System Programmers;
mobile – distributed-mainframe)

8. Go on Offense
Play Defense
…solutions from IBM
§ Provide cheap, isolated,
development and test environments
for project teams
– Rational Development and Test
Environment
– Rational Test Virtualization Server
– SmartCloud Provisioning
– Cloud Ready for Linux on System z
§ Automate consistent build,
configure, and deploy processes
across all stages
– Rational Team Concert
– UrbanCode uDeploy
– SmartCloud Orchestrator
§ Enforce base quality standards
automatically prior to promotion
– Rational Test Workbench
– Rational Quality Manager
– SmartCloud Application Monitoring
– Omegamon
§ Improve communication and
collaboration with cross-platform
release planning
– IBM Collaborative Lifecycle
Management
– Smart Cloud Control Desk
– UrbanCode uRelease

9. Test
Test
Dev
Dev
1
DevOps for the common mainframe delivery cycle
Dev Test Staging Prod
Development" Test" System
Programmer"
3
2 2
4

10. 10 10
Test LPAR
z/OS
…
Typical z/OS Testing Architecture
Organized by project team, vertically scaled, sharing resources, limited automation
Project
Team
[April Maintain]
Project
Team
[Prototype SOA]
Project
Team
[June New Func]
Project
Team
[Dec Sys Upgrade]
Test
Data
App
App
App
Problems Encountered
1. Shared resources combined
with overlapping schedules can
elicit conflicts, impede
innovation and slow code
delivery
2. Coordination of environmental
changes and releases cause
bottlenecks, delays and
additional overhead
3. Shared test data is difficult to
manage and can lead to over
testing or incorrect test results

11. The ultimate in modern application development for System z
§ Liberate developers to rapidly prototype new applications
§ Develop and test System z applications anywhere, anytime!
§ Free up mainframe development MIPS for production capacity
§ Eliminate costly delays by reducing dependencies on operations staff
Note: This Program is licensed only for development and test of applications that run on IBM z/OS. The Program may not be used to run production workloads of any kind, nor more
robust development workloads including without limitation production module builds, pre-production testing, stress testing, or performance testing.
RDz & ISPF user
ISPF user
RDz user
RDz user
RDz user
COBOL, PL/I, C++, Java, EGL, Batch,
Assembler, Debug Tool
x86 PC running Linux
RDz user
IMS
z/OS
WAS
DB2
MQ
CICS
Rational Development and Test Environment for System z

12. Simulate subsystem dependencies
– Test impact of latency
– Test application response to unresponsive
services
Create a stable test environment
– Simulate “public” services
– Reduce capacity requirements on
infrastructure
– Improve security by reducing access from
external services
Improve testing with Green Hat service simulation
Decouple subsystem teams using Green Hat
Databases Mainframe
applications
Third-party
Services
Rational Test Virtualization Server
Application changes
being tested
virtualized services

13. Testing with dependency virtualization
Phase 1 Phase 2 Phase 3
App
Test Case
Virtual
Services
3rd Party
Call
IMS
Data
Access
CICS
Commarea
Call
App
Test Case
Virtual
Services
3rd Party
Call
IMS Data
Access
CICS
Commarea
Call
App
Test Case
Virtual
Services
3rd Party
Call
IMS Data
Access
CICS
Commarea
Call
Controlled large system testing by isolating components under test
• Easier problem determination
• Lower test environment capacity requirements
• Improved component quality

14. Provision new dev / test systems in minutes
• Fast VM provisioning for near-instant
deployment of 100s of virtual machines and
scale based on business needs
• Rich set of web interfaces into the cloud
that can be driven by a user or scripts for
complete automation
• Reliable, non-stop cloud capable of
automatically tolerating and recovering from
software and hardware failures
• Near-zero downtime due to faults,
hypervisor/management software upgrades
or addition/removal of hardware
Requested
VMs
will
be
up
and
running
under
a
minute
using
standard
HW
Function
Test
Performance
Test

15. § Fast, dependable, automatic feedback speeds time to market
§ Lower cost of application testing using off-mainframe z/OS test environment
§ Enables confidence by automatically tracking and promoting code health
Continuous Integration for z
Developer
Frequent, rapid
feedback earlier
to improve
quality Rapid Feedback
Code
Check In + Build + Deploy +
Run Automated Tests
Change
Request
Automated,
continuous testing
with isolated
development and
test environment
Reusable
integration
tests simplify
testing across
layers
Promote
To QA
Higher quality
going into QA
Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)
Fail fast and drive better quality downstream

16. System z
Continuous Integration for System z Scenario
16
SCM/CI
Server
(Rational Team
Concert)
Test
Environment
(RD&T or LPAR)
Test
Execution
Manager
(Rational Quality
Manager)
Test
Automation
(Rational Test
Workbench)1. Check-in code
2. Build code and zUnit
tests
3. Deploy build results
and test data to Test
Environment
4. Execute zUnit Tests
<Deploys App+zUnit Tests>
<Drives>
5. Kick-off Automated
Test Plan
6. Run automated
interface tests against
Test Environment
7. Mark execution
records Pass/Fail in
Test Execution
Manager
<Starts Tests>
8. Report test results
in dashboard/build
results/defect
records in CI server.
<Results>

17. § Each tool came with its own
• UI - Web and desktop
presentations of views and
tasks
• Logic – Workflow, process,
search, query, scale, security
and collaboration
• Storage – Availability,
traceability
• Privacy, backup/archive
§ Resulting in...
• Brittle integrations
• Silos everywhere
• High cost to maintain and
administer
• Proprietary API's
Tool E
SCM
UI
LOGIC
DB
UI
LOGIC
DB
UI
LOGIC
DB
UI
LOGIC
DB
UI
LOGIC
DB
Tool B
Release
Planning
Tool D
Incident
Management
Tool C
Reporting
Tool A
Dev work Items
Separate tools makes collaboration inefficient or difficult...

18. Tie development release to operational release
Rational Team Concert Release Plan and work
items
SmartCloud Control Desk

19. 19 19
Delivery Organized for Flexibility and Speed
Organized by application team, horizontally sliced, dedicated resources, highly automated
Problems Encountered
1. Shared resources
combined with overlapping
schedules can elicit
conflicts, impede innovation
and slow code delivery
2. Coordination of
environmental changes and
releases cause bottlenecks,
delays and additional
overhead
3. Shared test data is difficult
to manage and can lead to
over testing or incorrect test
results
4. Provisioning, managing,
and synchronizing project
test environments including
data
QA/IntegrationTestLPAR
z/OS LPAR
…
Project
Team
[April Maintain]
Project
Team
[Prototype SOA]
Project
Team
[June New Func]
Project
Team
[Dec Sys Upgrade]
Integr.
Test
Data
LPAR
Data
RD&T
Data
RD&T
Data
LPAR
Data

20. What are we working on in
the lab…

21. Go on Offense
Play Defense
…solutions from IBM
§ Provide cheap, isolated,
development and test environments
for project teams
– Rational Development and Test
Environment
– Rational Test Virtualization Server
– SmartCloud Provisioning
– Cloud Ready for Linux on System z
§ Automate consistent build,
provision, configure, and deploy
processes across all stages
– Rational Team Concert
– SmartCloud Continuous Delivery
– SmartCloud Orchestrator*
§ Enforce base quality standards
automatically prior to promotion
– Rational Test Workbench
– Rational Quality Manager
– SmartCloud Application Monitoring
– Omegamon
§ Improve communication and
collaboration with cross-platform
release planning
– IBM Collaborative Lifecycle
Management
– Smart Cloud Control Desk
Cloud-style deploy of
zLinux and z/OS dev and
test patterns on RD&T or
zHardware

22. Infrastructure
as Code
Source Control
Management
Source Artifacts
.cbl
.asm
.cpy
CLIST.rexx
Provisioning using Cloud
#!/usr/bin/env ruby
class DevopsDeployer
def initialize(build_url, build_id)
@log = Logger.new(LOG_FILE)
@log.level = LOG_LEVEL
@iaas_gateway = IaasGateway.new(HsltProvider.new(),
LOG_FILE, LOG_LEVEL)
@server_instance = nil
rtc_build_system_provider = RtcBuildSystemProvider.new(
RTC_REPOSITORY_URL, RTC_USER_ID, RTC_PASSWORD_FILE)
@build = rtc_build_system_provider.resolve_build(
build_url, ENV['buildResultUUID'], build_id)
@build_system_gateway = BuildSystemGateway.new(
rtc_build_system_provider, LOG_FILE, LOG_LEVEL)
end
def add_build_stamp
template_file = WEB_APP_ROOT +
"/app/templates/pages/page.html"
@log.info "Adding build ID stamp #{@build.id} to
#{template_file}"
# Read in the file's contents as a string, replace
# the build_id, then overwrite the original contents
# of the file
text = File.read(template_file)
new_text = text.gsub(/{{ build_id }}/,
"<a href="#{@build.uri}">#{@build.id}</a>")
File.open(template_file, "w") { |file|
file.puts new_text
}
end
# ...

23. Delivery Pipeline Fundamentals
Deploy
Environment
Running System
Source Control
Management
Source Artifacts
.cbl
.asm
.cpy
CLIST.rexx
Deployable Artifacts
Build,
Package,
& Unit Test
Application
Binaries &
Platform
Configuration
Library

24. Consolidate, standardize, simplify deployment
COBOL
Java
Copybooks
WSDL
DB2
DDLIMS DBD/
PSB
HTML
MQ
Queue
Def
DB2
Plans
Procs
JCL
Deploy
Environment
Running SystemDeployable Artifacts
Build, Package,
& Unit Test
Application
Binaries &
Platform
Configuration
Library
Enterprise Applications have thousands of disparate parts
• Currently maintained in separate systems, if maintained at all
• Limited linkage between systems for application and configuration
dependencies
• Missing assets and information is rampant
DevOps force linkages, automation, and standardized packaging…
Source Management

25. Create, secure, and deploy test data
2TB
-Subset
Production or
Production Clone
-Compare
-Refresh
Create “right-size”
production-like environments
for application testing
De-identify sensitive information
with realistic but fictional data for
testing & development purposes
De-identify sensitive information
with realistic but fictional data for
testing & development purposes
z/OS
UTRD&T
Automated test data
deployment for each
build and test
1 GB
z/OS
UTLPAR
1 GB
Masked Test Data
Automated Build &
Deploy Test Data
Automated
Testing
Optim RTC
Test Data

26. Standardize z/OS region configurations to ease
deployment
• Standard topologies exist today
(production LPARs)
• Standardized regions which can be
repeatedly deployed are rare
• Standardized/Automated deployment of
COMPLETE system is spotty
To adopt the cloud for testing…
Standardize and automate provisioning of
everything …
Standard Middleware Pattern
CICS 5.1 IMS 11.1
DB2 10.1Z/OS 1.12
Configuration
Data
Application

27. RTC/CIzAvailable
Capabilities
Standardize z/OS region configurations to ease
deployment
Standard Middleware Pattern
CICS 5.1 IMS 11.1
DB2 10.1Z/OS 1.12
Configuration
Data
Application
Pattern/Platform
Application
load modules
Test Data
JCL and REXX

28. Evolving to deploying regions … cloud-style
“Cloud”
RD&T
Or
LPAR
RD&T
Or
LPAR
RD&T
Or
LPAR
RD&T
Or
LPAR
SmartCloud Orchestrator
+
<Deployment Automation
for Mainframe project>
Region/
Middleware
Patterns
CICS+DB2
CICS+ IMS
DB
Batch

29. 29
Daily Apple TV giveaway
§ Complete your session surveys online each day at a conference kiosk or on
your Innovate 2013 Portal!
§ Each day that you complete all of that day’s session surveys, your name will
be entered to win the daily Apple TV!
§ On Wednesday be sure to complete your full conference evaluation to receive
your free conference t-shirt!

30. 30
Acknowledgements and disclaimers
© Copyright IBM Corporation 2013. All rights reserved.
– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products
and services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or
both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™),
these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks
may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright
and trademark information” at www.ibm.com/legal/copytrade.shtml
If you have mentioned trademarks that are not from IBM, please update and add the following lines:
[Insert any special third-party trademark names/attributions here]
Other company, product, or service names may be trademarks or service marks of others.
Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries
in which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided
for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any
participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided
AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise
related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating
any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may
have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is
intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue
growth or other results.

31. © Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
www.ibm.com/software/rational