Testers are under big new pressure to use Agile software methods, Ajax environments, and Mobile environments. In this session, Frank Cohen of Appvance will show how Best Buy and PepsiCo cope with the changes: How they select test tools, test management tools, and create operational test data. See how this all fits together in an Agile environment using Continuous Integration, Source Repository, Test Scripting, and Agile Test Management for Agile Stories. Frank will show practical examples - to the code level - to deal with desktop browsers, iOS and Android native apps, WebKit and Mobile Web browsers.

1. Testing Ajax, Mobile Apps
the Agile Way
March 4, 2014
Frank Cohen, fcohen@appvance.com, (408) 364-5508

2. PepsiCo Needs Deep Reach To Consumers
• High User Experience Web App Campaigns
• Do Us A Flavor, Every2Minutes using Ajax, Web, Flash/Flex,
• Broad Reach Mobile App Campaigns
• iOS/iPhone, iPad, Android, Tablets Native Apps
• Time-On-Page Using Publishing Platforms
• Tumblr, WordPress, Google+, Facebook, Twitter
• Partners Via ERP, Workflow, Integration Platforms
• Oracle, TIBCO, IBM, Microsoft, Software AG, Redhat BPM, EMS, MQ

3. Testers Are Under Big New Pressure
• Agile Software Methods
• Ajax Environments
• Mixed Mobile Environments
“The Business Managers Tell Us They Need 1000 VUs in 10 Mins”
“The Devs used JMeter”
“The App Runs on WebKit Browsers and Android Native”

4. How Does It All Fit Together
• Agile Environment featuring Agile Stories
• Story < UseCase < Test < Schedule < Deploy < Build
• Continuous Integration, Source Repository, Test Scripting, Deploy Manager, and Agile
Test Management
• CloudBees, Travis CI+Deploy, JMeter, Rally, Appvance
• Select Test Tools, Test Management Tools, And Create
Operational Test Data

5. Test Runtime Architecture
Continuous
Integration
Developers
Jenkins
Appvance Performance Cloud
Appvance
Controller
Test Injector
Test Management
Clear
Instance 1
Map Story to Tests
Deploy to Cloud
Log Results
Test Repository
Clear
DPL Repository
Oracle
Script Runner, Monitor, DPL
Browser
Driver
Chrome, IE
SOAP, REST
Instance 2
Test Injector
Script Runner, Monitor, DPL
Aric New User SLA Warning
Average perf: 32 Seconds
Business,
DevOps
Results Repository
Oracle
Dashboard
Driver
Chrome, IE
Scalability Index
Browser
SOAP, REST
Shared Services
Environment
Service Under Test
App Under Test
Virtualized
Environment
Service Under Test
App Under Test

6. Modern Deployment Requires Modern Testing
• Desktop browsers, iOS and Android native apps, WebKit and
Mobile Web browsers
• End-To-End means HTTP Protocol level testing
• Beginning-To-End means Driving The Entire App
• Combination Testing Surfaces Root Case To Bottlenecks

7. The Hybrid Model
• End-To-End Component Testing at API/Protocol Level
• Beginning-to-End App Integration Testing
• Scalability Testing of End-User Experience
• Data Driven Tests from live operational data sources
• Functional Tests Repurposed To Performance Tests
• Validate Digital Agency Tests (normally not LoadRunner)

8. Typical Solution Combines Process and Teams
• LoadRunner Best Practices for Ajax, Mobile
• Tool and license for LoadRunner team to do its own testing
• Train testers and business managers on Test Methodology
• Consulting Practice, Integration Expertise
• Build upon the tools
• Identify Requirements for Digital Agencies, Product Teams
• 10-15 best practices required by digital agencies for app development
• Server Configuration Requirements Document
• SLA Requirements Definition for Publishing PAAS Providers

9. Missing Integration Expertise
• App Integration Bridges Digital Agencies
Agency
Architecture
Diagram
Host
Implements
Frank:
“Where’s the
Node.js load
balancer?”
Host: “Oh, We
missed that.”

10. Digital Agencies Making Same Mistakes
• Frank asks: “Why is the deployed app not accepting 13 digit
cap codes?”
Agency: “We missed that.”
• Frank asks: “Why is network limiting throughput to PHP
requests to Node.js?”
Host: “The F5 load balancer does that by design. We’ll
remove it.”
• Frank asks: “Does the delivered app have testing APIs?”
Agency: “We will write some.”

11. Mistakes Happen At App Integration Time
• Frank asks: “How could more than 1 person log-in per second
using Gigya?”
Gigya: “Oh, we didn’t think of that.”
• Frank asks: “Why isn’t the Log In service scalable above 100
log-ins per second?”
Agency: “You’re running it on Windows, all our experience is
on Linux.”
Host: “We don’t know.”
• Frank asks: “Why is Apache set to only use 25 worker
threads, when the app needs hundreds?”

12. Example Architecture - Integrated App
Email Services
Utility
Apache, PHP
MySQL Master
MySQL
Read
MySQL
Read
Load Balancer
App Web 1
Apache, PHP, Zend
NodeJS
Bid History
Load Balancer
NodeJS
View Counts
Load Balancer
Distributed Edge
Cache
Browser

13. App Integration Test Readiness Requirements
Number of
Server
Machines
Identified and
Provided
App code
from Agency
Installation at
Host
Last Minute
Changes from
the Business
Voting Service
(SAAS)
Configured
App Integration
Database
Created and
Populated
Preference
Center
Configured
Apache, PHP,
Zend, .NET,
Java optimized

14. Best Practices for LoadRunner on Ajax, Mobile
• Beginning-to-End App Integration Testing
• Run Ajax App In Real Browsers: Ie, Firefox, Chrome, Safari, Webkit, Mobile Web. Tests
100% Of The End-User Experience.
• LoadRunner TruClient does not work reliably with Ajax apps,
does not test on Chrome, Safari, WebKit, and older IE, Firefox
• LoadRunner works great for end-to-end and protocol level tests. For example, making
HTTP/JSON requests to a Node.js server through a load balancer
• Combine Beginning-To-End and LoadRunner
for test of app’s integration at app and protocol tiers

15. Models That May Work
• Product and License - Augment What You Already Have,
Teach and Train your team
• Services, Experts in Load Runner and other products, need
outsource, short staff
• We know your product, we know your tools, we know our
tools, we know what reduces app performance
• Maestro Service between agencies
• App Integration and Performance Test Services

16. Appvance
• When You Build Modern Apps And Wonder
“How Risky Is That App Launch Going To Be?”
• Call Us At (855) 254-1164
Or Email Us Sales@Appvance.Com
And We Will Show You