This session shares how you can catapult your career with cloud-based test automation and mobile testing techniques that are as exciting to learn and use as they are impactful to your end results and personal success! 1) Become a cloud testing expert 2) Build and articulate a distributed mobile testing strategy 3) Champion a new approach to realistic, repeatable web and mobile performance testing 4) Establish yourself as an agile testing expert with Continuous Testing 5) Defend “Test” in a world heading toward “DevOps” Cloud Computing is a proven advantage for testers and mobile app testing is a veritable testing green field for those willing to charge ahead. Learn the new “arts” - embrace this new world – and become a Renaissance Tester!

1. © 2012 SOASTA. All rights reserved.
1

2. o Brad Johnson - @bradjohnsonsv
• SOASTA (“09-Present)
• Borland (“06-”09)
• Mercury Interactive (“02-”05)
• Compuware (“99-”02)
o I’ve seen a few testing market trends
• Client-server to Web
• Web to BIG Web
• Custom to packaged and back to custom apps
• Waterfall to agile to continuous
• Cloud computing
• Mobile
© 2012 SOASTA. All rights reserved.
May 22, 2012
2

3. o Mobile testing is a vacuum
o Cloud testing is in demand (and
a no brainer)
o Testing in production is an
accepted practice
o Agile testing has come of age…
with Continuous Integration
o DevOps is a movement, just add
the testing and succeed
© 2012 SOASTA. All rights reserved.
May 22, 2012
3

4. What’s Driving the Renaissance?
o Explosion in mobile applications
o The immensity of the web & everything web
o Rapid introduction of new technologies
o Incredible pace of development and delivery

5. o This is the greatest opportunity since
Web
o Every company has a mobile
strategy
o Very few have mobile testers
o Fewer have mobile testing strategies
o Global outsourcers are seeking help
© 2012 SOASTA. All rights reserved.
May 22, 2012
5

6. Keyboards
Mouse
Gestures
Voice
Bendable Phones
Kinetics
3D Navigation
Hovering Gestures
© 2012 SOASTA. All rights reserved.
May 22, 2012
6

7. o Shear Number of Devices (953M
Smartphones)
o Different Operating Systems
o Scale of Global Customers (6B)
o Dynamic Content (Video, Annimation)
o Emerging People to Machine Interfaces
Manual Testing Can Not Deliver Adequate Coverage
© 2012 SOASTA. All rights reserved.
May 22, 2012
7

8. o Nearly 80% of all mobile apps are tested manually (IDC)
Take Action
o Learn and implement effective automation
o Focus on user experience
o Consider the impact of precision and performance
on functionality and satisfaction
o Prioritize and streamline to keep pace with the rate of
change
o Communicate the complexity
© 2012 SOASTA. All rights reserved.
May 22, 2012
8

9. o Read everything you can
o Understand how mobile apps are built and how they are consumed
o Use mobile as an opportunity to “fix” software development
o Join SOASTA’s Early Access Team (see me)
© 2012 SOASTA. All rights reserved.
May 22, 2012
9

10. o Testing with cloud, testing in the
cloud, testing for the cloud…
o Covered by all major (and minor)
analysts: Gartner, Forrester,
Aberdeen, IDC, Ovum…
o Cost and scale benefits are
indubitable
o Expertise is an enterprise QA
requirement
• IBM testers required skills
© 2012 SOASTA. All rights reserved.
May 22, 2012
10

11. What Makes a Prime Cloud Testing Project?
o Critical web launch – mobile support, new website
o Tight timeframes - no room for experimentation
o Web-scale expectations – 100s to millions of users
o Quick start, many test iterations – adapt to flexible dev
schedule
o High cost of tools and infrastructure - artificial ceiling
increases risk
© 2012 SOASTA. All rights reserved.
May 22, 2012
11

12. o Read everything you can
o Open a cloud computing account
o Watch webinars
o Try a cloud testing tool
o Run a cloud test
© 2012 SOASTA. All rights reserved.
May 22, 2012
12

13. Code
From the wall of a very
traditional retailer’s
datacenter
© 2012 SOASTA. All rights reserved.
May 22, 2012
13

14. Latency between systems
Network Network configuration
Capacity limits
& Ops
Network bandwidth
CDN file placement
Conflict with other apps DNS routing
Stage / Team
Load Balancer configuration Firewall max capacity
Auto-scaling failures Unbalanced web servers
Max sockets exceeded
s
Release Global latency variance
& Deploy Slow third-party plug-ins
Security bottlenecks
Caching issues and benefits
Search technology limits
Default configuration settings
Inadequate server resources
Garbage collection
Memory leaks
Memory leaks
Database thread counts
Dev Slow pages
& Test Inefficient database queries
Code bugs
Method-level tuning
Test Lab Staging Production (100% +++)
Scale and Time
© 2012 SOASTA. All rights reserved.
May 22, 2012
14

15. o Read some good whitepapers
o Watch/download Seth Eliot’s session and slides
o Attend a webinar on Oct. 30 (Seth & SOASTA)
© 2012 SOASTA. All rights reserved.
May 22, 2012
15

16. o Still can’t find a good online diagram, though!
© 2012 SOASTA. All rights reserved.
May 22, 2012
16

17. Continuous Testing is Progressive
Dev & Test Lab
Production Environment
Include Hit expected Collect real
Start small, Recheck
functional scale in user data to
early to root Test in regularly for
automation & staging and feed back
out code- Production
regression
performance maintenance
level issues
and capacity
(RUM)
in CI
windows
Keys to agile testing?
SPEED – SCALE – COST
© 2012 SOASTA. All rights reserved.
May 22, 2012
17

18. Many approaches are process and pre-test heavy
Traditional Approaches
TEST BUILDING HW Setup TEST ANALYZE FIX RETEST REPEAT?
TIME
BUILD TEST TEST TEST TEST TEST
Extra
ANALYZE ANALYZE ANALYZE ANALYZE ANALYZE
FIX
time!
FIX FIX FIX FIX
Eliminate constraints and TEST MORE, at SCALE, in LESS TIME
Agile approaches deliver results fast
© 2012 SOASTA. All rights reserved.
May 22, 2012
18

19. Test Types
Test Objective
SPRINT 1 Load Test
Determine Limits
10 hours
Refinement of
Measure User Experience
Performance Test
(Benchmark)
Expectations
SPRINT 2
Test Specific Silos
Targeted Infrastructure Test
8 hours
SPRINT 3
Verify Redundancy
Failover Test
3 hours
SPRINT 4
Stress Test
Exceed Break Points
Hours for 5 hours
example only
SPRINT 5
Capacity Test
Plan for future
8 hours
SPRINT 6
Soak Test
Reliability over time
20 hours
Test
Deploy
Implement
Assess
Design
Define
Delivery Cycle
© 2012 SOASTA. All rights reserved.
May 22, 2012
19

20. Production
Devs
Check in
Or QA
Test Results
✓
☐ Pass
Source Code Repository
(Git, Subversion, Atlassian Jira
☐ Fail
Microsoft VS Team Foundation Server)
Check out
Test Automation
Run Tests
Continuous Integration server
(unit, component, functional)
automates builds, launches
tests, manages results
Dev or Test Lab
(Jenkins, Bamboo, VSTFS,
Anthill,…)
© 2012 SOASTA. All rights reserved.
May 22, 2012
20

21. Beta Users
Devs
Check in
(ad-hoc server)
Test Results
✓
☐ Pass
Source Code Repository
☐ Fail
Check out
Test Automation
Run Tests
Integration Server
i.e. Jenkins Slave
(Mac)
Execute on Devices
Push to Device
Real Devices
© 2012 SOASTA. All rights reserved.
May 22, 2012
21

22. o ALM meets APM
o What’s old is new…just FASTER
o This is Process Compression
Test
Monitor
Build
Deliver
Deploy
Design
Define
Manage
© 2012 SOASTA. All rights reserved.
May 22, 2012
22

23. o ALM meets APM
o What’s old is new…just FASTER
o This is Process Compression
DevOps!
Dev Ops
© 2012 SOASTA. All rights reserved.
May 22, 2012
23

24. o ALM meets APM
o What’s old is new…just FASTER
o This is Process Compression
Dev Psst…
DevOps!
(TEST) Ahhhh.
you forgot
something
Ops!
© 2012 SOASTA. All rights reserved.
May 22, 2012
24

25. o Read everything you can – ALM and APM
o Seek visibility solutions – reporting, collaboration
o Expand your sphere of contact and influence
o Join or run a production performance test
© 2012 SOASTA. All rights reserved.
May 22, 2012
25

26. o Champion a mobile testing team or project?
o Run a cloud test?
o Be the Continuous Integration agent-of-
change?
o Put the “test” in DevOps?
THANKS!
Questions?
(Please turn in the eval.)
© 2012 SOASTA. All rights reserved.
May 22, 2012
26