1. Testing in the Cloud
Our experience of testing software
that is “out there”

2. Introduction
- The Presenters
- Paul Selway – paul_selway@redpathcg.com
- Scott Rostal – scott.rostal@thrivent.com
- What is “The Cloud”?
- What is different / the same for cloud testers?
- Testing Philosophy / Test phases
- What was easier / harder than expected?
- Key success factors
- Questions?

3. What is “The Cloud”?
• Internet (“The Cloud”) accessible solution
• Off-premise multi-tenant architecture
• Accessed via a browser (PC or Wireless)
• Delivered as Software as a Service (SaaS)
Common cloud companies are:
Google, Amazon, WorkDay, and Salesforce

4. What things are the same about
testing cloud solutions?
 Requirements coverage based testing
 Risk-based structured test approach
 Test Plan / Test Phases
 Test Cases / Test Data / Test Automation
 Defect Management / Functional Test
 Use of off-shore resources / Virtualization
 Politics {Quality / Schedule / Resources}

5. What is different about testing
cloud solutions?
Shared multi-tenant test environments
Security (Test User IDs / SSO)
Integration of on/off premise systems
Performance / volume test
Defect isolation
Documentation of “Out Of the Box” reqs
Release Management

6. Project Philosophy
• Risk-based structured testing approach {std}
• “Exploratory Testing” to discover the out of
the box cloud solution features {non-std}
• Data Migration – Moving legacy data off
premise {Non-Std}
• Performance / volume test only our new
infrastructure – not the cloud app {Non-Std}

7. Testing Phases
Inspections End-To-End Data
& Reviews Integration Migration
Unit / Code Exploratory Load & UAT &
Coverage & System Performance Pilot
Overlapping Test Phases
A mix of waterfall (with heavy phase overlapping), and iterative delivery

8. What was harder than expected?
• Isolating the architectural components for a
performance test
• Understanding that a cloud solution vendor
delivers new releases which you have to
take!
• Accepting that you have a “black box” in
your environment
• Determining who owns defects
• Data Migration Mock Runs

9. System Integration Architecture

10. What was easier than expected?
• Getting the IT leaders to embrace change
• Overlapping Test phases
• End User Adoption
• Use of off-shored test resources
– Functional
– Performance
– Data ETL

11. Key Success Factors
• Learn the “Out of the Box” cloud application
features, security model, and configuration
capabilities
• Understand the holistic system architecture
• Understand the migrated test data
• Establish tight communications with the
vendor

12. Questions?

13. Contact Info
• Scott Rostal – scott.rostal@thrivent.com
Performance Lead, Thrivent Financial for Lutherans
• Paul Selway – Paul_Selway@redpathcg.com
President, Red Path Consulting Group