SaaS Testing Overview - Foundation

SaaS Testing Overview

Ram Garg
CEO

Software Performance Assurance (SPA)

Software Performance Assurance Confidential

Outline
 SaaS Introduction
 What is it?
 Why SaaS?

 SaaS Challenges
 Key Challenges

 SaaS Testing
 How to approach SaaS testing?
 How SaaS Testing is different?
 SaaS Test Tools


What is SaaS?

SaaS Introduction


SaaS: What is it?

Software-as-a-Service (SaaS) is the application
layer of the cloud computing model.

Application Software as a Service (SaaS)

Platform Platform as a Service (PaaS)

Infrastructure Infrastructure as a Service (IaaS)


SaaS Definition

“Software as a service (SaaS), sometimes
referred to as "software on demand," is
software that is 100% deployed over the
internet … . With SaaS, a provider licenses
an application to customers as a service
on demand, through a subscription or a
"pay-as-you-go" model.”
– Wikipedia definition


Architecture: Three Key Differentiators

Multi-tenant
Efficient

Scalable Configurable

Key
Architecture
Differentiators


Why SaaS?
SaaS Introduction


The Software Business Model Changes …

non -SaaS SaaS

Software Ownership Customer Provider

Infrastructure
Customer Provider
management Responsibility

Installation Client - Server Central

Upgrades Larger, less frequent Shorter, frequent

Platform Multi-version Single-version


Changing the Business Model cont …

non -SaaS SaaS

Configurability
Low High, Critical
Requirement

Multi-tenancy No Yes

Backward compatibility No Yes

Maintenance Customer Provider


Changing the Business Model cont …

non -SaaS SaaS

Cost, Allocation High, CapEx Low, OpEx

Close the deal, Provide value,
Sales focus / Success
License Revenue Lack of Churn

Profits Initial sale + maintenance Ongoing Subscription


Ecosystem

SaaS (Software)  Starting point to every service within
cloud computing ecosystem
PaaS (Platform)  Highest on the maturity level and
growth curve amongst XaaS
IaaS  Highest revenue generating service
(Infrastructure)
within cloud model for next several
years

SaaS Catalysts
Intrigue of access to unlimited Computing and storage
resources, made available and paid for only as needed (Pay as
you go)

Decreasing cost of bandwidth

Customer frustration with traditional software buying cycle
(license, maintenance contract, upgrades)

Perceived increase in control over the vendor relationship

Early SaaS success stories – Salesforce.com, WebEx, etc.
Demonstrate broader value proposition

You focus on your application and not on infrastructure


SaaS Early Adoption
SaaS Introduction


SaaS Adoption: Verticals
AMI-Partners„ report -
"Software-as-Service Hot Adoption rate of
Spots” : Education, SaaS in SMB is
Healthcare, and Financial" higher than large
enterprises

All research points out that
In-Stat research on SaaS has been adopted
vertical market adoption
shows the healthcare heavily in :
market is currently most  HR applications (ERP,
satisfied with hosted Salesforce automation,
applications, with eRecruiting)
education identified as an  Online Collaboration (Web
important potential target conferencing)
market also.  CRM
 Healthcare


SaaS Adoption Factors
(source: Forrester)

Availability & Reliability (71%) Fast Deployment (45%)

Reduced dependencies on
Flexible consumption (35%)
internal IT (24%)

Simplified pricing model (32%)


The SaaS Market is and will
Continue to Grow…


Today and tomorrow …


Challenges
SaaS Challenges


Primary Concerns with the SaaS Model

Security concerns

Concerns over compliance with standards

Concerns over integration with legacy applications

Concerns over reliability and robustness of service

Not comfortable with putting mission-critical application on shared
infrastructure and have access over internet

SaaS can not be configured to meet my corporation‟s all specific
needs


Numerous Challenges

Security Performance Availability SLA

Compliance
Data Integrity Privacy with Regulatory
Standards

Lack of
Integration Multi-tenancy
confidence


SaaS Testing


Why SaaS Testing?
SaaS Testing


The average enterprise IT
Why SaaS Testing?
department devotes up to 50
percent of its entire technology
infrastructure to development A delay in website page
and test, with up to 90 percent loading by even one second
of the available test can impact conversions by
infrastructure remaining idle. 7% and customer satisfaction
IBM Research by 16%.
Aberdeen Group

Cloud
Testing

Those using cloud facilities to
run applications, 49.8% said
they were doing so
The average online experimentally or for
business had 8 days of prototyping; 28.6% for non-
downtime a year due to critical business applications
performance issues and 21.6% for business
leading to lost revenues critical applications
Aberdeen Group Evans Data Survey


Trend leads to trend …

Overall growth in Cloud Computing (investment,
revenue, ROI, market cap, etc.) is leading to a
marked increase in adoption of SaaS Testing

What is SaaS Testing?
SaaS Testing


Classifying SaaS Testing

“Testing The Cloud”
SaaS”
Testing applications which are in
cloud or migrating to cloud.

Testing Test the performance, reliability &
the SaaS security of applications

Cloud /
SaaS “Testing SaaS Using Cloud”
“Testing Using Cloud”
Testing applications using the
Testing cloud-based hardware, software,
Testing infrastructure and computing
resources.
SaaS
Using Tests performed like performance,
Cloud load & stress, security and
compatibility


How SaaS Testing is Different

Easier Challenges
• No client-server • Frequent live upgrades
installation
• Too many pages
• No multiplatform support
• Multiple privileges
• No multiple version
upgrades • Immature SaaS-specific
• No backward compatibility testing tools
• Access partitions critical


Approach
SaaS Testing


SaaS Testing Preparation

Prepare
Assess Understand Prepare
Test Plan Prepare
Testing Usage Test
and Test Test Data
Needs Patterns Environment
Suite


SaaS Testing Areas

Infrastructure
Testing

Network Application
testing testing

Secured
SaaS
Application

Ensure SaaS testing does not damage
the shared infrastructure

SaaS Functional Testing
• Perform Manual and Exploratory test (similar to traditional
applications)
• Check Browser compatibility
• Perform regression
• SaaS regression must cover significant more functionality as single
defect will impact all the customers

• Functional and regression test automation.
• This is important to comprehensive testing cost-effectively as above
point indicates that, in case of SaaS, regression should cover more
functionality.

• Test SaaS on cloud-based staging environment
• Testing on in-house environment and then directly deploying on cloud-
based production environment may result in chaos

• Test in a multi-user environment where different users are
performing different activities.

SaaS Security Testing

• Test application security
• Cross Site scripting, SQL Injection, HTTP header injection

• Test network security
• Secured communication, Backup and storage policies, Controlled
access to sites and servers

• Test for possible attacks
• Test roles and privileges in multi-tenant environment
• Many roles and privilege levels

• Test all the upgrades to detect broken security,
Ensure security patches are not missed
• Test data security / integrity among multiple tenants


SaaS Security Testing cont …

• Identify scenarios where application might be
vulnerable
• Test PCI compliance
• Test cookie setting, SSL configuration, information
leak, etc.
• Log errors, warnings and suspicious requests
• Tests tampering with the licensing mechanism;
gaining unauthorized access to application data,
editing registry entities or configuration files


SaaS Performance Testing
• Test performance extensively first time and for each upgrade
• SaaS is revenue generating and business impacting applications.
Single performance issue will impact all your customers
• Test performance extensively as SaaS need to handle large number
of users and we don‟t have luxury of re-booting or going down every
once in a while

• Test load characteristics, ramp up rate, protocols, load
origination,
• Establish ability to test performance of your SaaS isolated
fashion so that network related issues can be identified and
passed to network infrastructure and internet providers
• Test key metrics and SLAs
• Test performance for mobile users


In-Service Upgrade Testing
• Ability to upgrade (full release, patches) system with zero-
to-minimum down time is essential
• Global users make this complicated as somewhere in the
world someone might be accessing your application
• Thoroughly test in staging environment before perform
upgrade ass this is one of the most complex features

Remote Accessibility and Usage
• Global access with acceptable response time
• 24 hours availability


Disaster Recovery
• Shutdown to bring up application quickly: server configuration;
install and deploy software ; restore data

• Roll back to last known stable data

Localization
• Make sure it supports international characters as SaaS
application can be accessed from anywhere in the world
• Optimize by supporting target market language


Test Tools
SaaS Testing


Traditional and Cloud-based Test Tools

Cloud-based Test
Traditional Test Tools
Tools

QTP, WinRunner
Rational Robot
Rational Functional Tester
Functional
SilkTest
Gomez
PushToTest,
Selenium, STAFF, Fitness, Watir
SOASTA
CloudTesting
LoadRunner, Rational Robot, uTest
Rational Performance Tester Many others
Silk Performer
Performance
Jmeter, Grinder, WebLoad,
OpenSTA


Traditional Vs. Cloud-based Tools
Testing SaaS Using Testing SaaS Using
Traditional Tools Cloud-based Tools

Many providers who provide A few providers who provide
SaaS testing using traditional SaaS testing using their own
testing tools cloud-based test tools

Many open source matured Limited open source test tools
tools options

Many options for professional Captive professional service by
services by independent test tools provider which is not
service providers even core for their business

Professional service options Captive profession service is
allow cheaper cost sold at very high rate


Conclusion


Conclusion
Customers buy each month – keep them happy

SaaS is only a delivering model, customers will not compromise on
functionality, stability, security, and performance. They will not switch to
SaaS it if not better than traditional systems.

Save time on support, maintenance, and upgrade, etc.

You need to pay more attention to stuff like accessibility, security,
deployments. The way you earn reputation

Don‟t buy cloud-based test solution just because you are testing SaaS. In
most cases, traditional test solutions work just fine and save your 30% –
50% cost.


SaaS Testing Overview - Foundation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to SaaS Testing Overview - Foundation

Similar to SaaS Testing Overview - Foundation (20)

More from Ram Garg

More from Ram Garg (10)

Recently uploaded

Recently uploaded (20)

SaaS Testing Overview - Foundation