Testing Web Applications: A Practical Approach Walter Mamed, JWT.com Track 3: 11:00 – 12:00 Web-based applications have become the most widely used form of software, not only for e-commerce, but in our personal lives as well. Whether your spouse is booking your next vacation, or you are scheduling an appointment in an acute care facility, responsiveness and reliability are key to your satisfaction and desire to return. The quality assurance group testing these applications faces many challenges, with shorter test cycle times, fewer resources, constantly evolving technology, and instant world wide exposure. Explore how to plan, test, and deploy new or updated websites with confidence using practical, no nonsense methods. Functional and non-functional testing including configuration, usability, performance, and security will be covered. Learn how to use software tools to improve your testing techniques. Automated testing, mobile browsing, and the future of Rich Internet Applications will also be discussed. Take home a new perspective on testing web applications; implement these solutions and reduce your testing anxiety. About the Speaker… Walter Mamed is Director of Quality Assurance at JWT (Digital Technology) in Irving, Texas. He has over 30 years experience in a variety of quality assurance and software test engineering development positions, focusing on software and hardware test automation. Walt has been building test automation frameworks for GUI testing and web based applications for over 15 years. His web testing experience includes secure Email, On-boarding, ecommerce and lead generation as well as large-scale automated regression test suites. Walt is very active in the professional community as Director of the Board and Secretary for the Dallas/Ft. Worth (HP) Mercury User Group (DFWMUG.com) for the last 7 years. He is an ASQ Certified Software Quality Engineer.

1.
Friday, April 23rd
Testing Web
Applications: A
Practical Approach 2010
INSTRUCTOR: WALTER MAMED
COMPANY: JWT.COM
11:00 AM – 12:00 PM

2.
This page intentionally left blank.

3. Testing Web Sites & Applications
A Practical Approach
Walt Mamed
JWT, Director, Quality Assurance
02/09/2010
News Story ‐ Stray Mouse Click
2
1

4. Double Click of Death
• On November 14, 2007 at 3:30pm one of Credit
Suisse s trading algorithms suddenly went haywire
Suisse’s trading algorithms suddenly went haywire
• Sent hundreds of thousands of bogus requests to the
exchange.
• Acted like a denial‐of‐service attack on the NYSE
• Affected trading of 975 stocks
• Caused by a trader who accidentally double‐clicked
an icon in a trading program’s interface.
• Credit Suisse assessed a $150,000 fine
3
Overview
1. Test Planning
2. Configuration Management
3. Test Execution
4. Projects – CBT, iPhone, Security, and more
5. Test Tools
(Hint: Watch for gold nuggets)
4
2

5. Test planning for a new or existing web site
TEST PLANNING
5
Test Planning
Test Planning is one of the keys to project success:
Gathering Requirements
G th i R i t
Functional Decomposition
Risk Based Testing Analysis
Develop Test Plans and Procedures
6
3

6. Requirements Gathering
Requirements come in many forms:
Project Plan or Business Requirements Spec
P j t Pl B i R i t S
Software Requirements Specification
Functional Design Document
Feature Specification Document
Interface Control Document
Use Cases
Wireframes
7
Functional Decomposition
FD ‐ Breaking it down piece by piece:
Decompose the intended function into sub‐functions
D th i t d d f ti i t b f ti
Divide and conquer (split due to volume of effort)
Top‐down: if system is fully described
FD based on the flow of data or traversal by user
y q
Verify all requirements have been covered.
It’s easier if you have Use Cases
More challenging if you have Business Requirements
8
4

7. Functional Decomposition
Critical or High use of Balancing risk based
subsystem, function testing and repetition
or feature
or feature Booking of tasks
ft k
Booking
Scenarios
Smoke Tests
Searches &
Loyalty Program Filters
Defect Functional 1
Functional 1
Escapes
Business Functional …
Adjustments
Tools supporting
Functional …N
manual testing
9
Test Planning – RBT Analysis
RBT ‐ Risk Based Testing analysis:
High use of a subsystem, function or feature.
Hi h f b t f ti f t
Criticality of a subsystem, function or feature,
including the cost of failure.
Prioritize what should be tested first.
Not doing so explains why big bugs are found at the
end of a test cycle; its human nature to test the easy
functionality first.
10
5

8. Test Planning – RBT Analysis
Test Design Techniques using software models:
Equivalence partitioning
E i l titi i
− Breakdown elements into classes
− Perhaps use a mind map
Boundary value analysis
− Identify edges or end‐points
Decision tables
State transition diagrams
− Will also help define your negative tests
11
Test Planning ‐ Mindmap
12
6

9. Test Planning – Test Plans
Develop Test Plans and Procedures:
Test plans are usually in Word.
Test plans are usually in Word
Detailed test procedures are usually in Excel.
Quality Center – Requirements and Test Plan
modules (great for confirming all requirements
covered).
Writing these documents should be easier if the
Writing these documents should be easier if the
previous steps (FD & RBT) were performed.
IEEE 829 defines many types of test specifications
– “If it’s not written down, it didn’t happen.”
13
Test Planning
Practical Suggestions for Test Planning :
Web Analytics (for existing websites)
W b A l ti (f i ti b it )
Planning Test Automation? – Testability as a
requirement for Development
Production Monitoring
14
7

10. Test Planning Suggestions
Web Analytics (for existing websites)
Online Business Optimization (Tealeaf, Omniture)
O li B i O ti i ti (T l f O it )
– Exit rate, average time on page, contribution to revenue.
– Where and why are visitors leaving.
– Know how customers are using your site.
Browser usage (Cross Browser Test planning)
– Browser type used, what version, mobile user type
d h bl
– Use what you know your customers/visitors use.
Behavior Map (page hit frequency)
15
Test Planning ‐ Behavior Map
16
8

11. Test Planning ‐ Testability
Planning Test Automation?
Testability
T t bilit as a requirement for Development:
i tf D l t
Provide a unique and meaningful name property for:
– Every actionable html object on the page. (entry‐fields,
buttons, radio buttons, dropdown list boxes, images, links,
etc.)
– Every table object that requires testing
Every table object that requires testing.
– Every response that requires testing. The response may be
in tables, spans, divs, lis, etc.
17
Test Planning ‐ Testability
Testability as a requirement for Development:
Populate the ‘id’ and ‘alt’ tags to give QA more
P l t th ‘id’ d ‘ lt’ t t i QA
alternatives to identify an object during scripting.
– SEO and 508 Compliance contribute to this
recommendation as well.
Use a naming convention that includes the function
or purpose of the given object.
or purpose of the given object
Do not change any HTML element property name
(including id & alt tags) from release to release.
18
9

12. Test Planning ‐ Monitoring
Production Monitoring:
Ensure your site and applications are performing.
Ensure your site and applications are performing
Identify, resolve and prevent issues.
Develop an escalation policy, triage, remediate, and
confirm resolution.
Use automated daily smoke tests to supplement
monitoring from a customer or partner perspective.
monitoring from a customer or partner perspective.
Discuss this during the requirements phase
– What, how, where and who?
– The wrong time is the day of deployment.
Make sure you know what you are testing.
CONFIGURATION MANAGEMENT
20
10

13. Configuration Management
Manage software configurations:
Audit configuration after push to QA/Prod
A dit fi ti ft h t QA/P d
– Use mySite.com/revision.txt to confirm
– Output contains Build Version, Date & Time
Establish method to directly access web servers
– Avoid round‐robin approach behind load balancers.
– h // b
http://web#‐www.mySite.com/revision.txt
/
21
Let’s get to it!
TEST EXECUTION
22
11

14. Functional Testing
Functional testing:
Run Smoke, Sanity, Critical Path tests
R S k S it C iti l P th t t
Check all links and web pages
– Site spider
• Start at Home page and traverse whole site
– (exclude external links)
• Check HTTP status 2xx, 3xx, 4xx & 5xx
• View pages for gross or cosmetic failures. (more later)
– Xenu link sleuth
23
Functional Testing
Functional testing:
Forms submittal
F b itt l
Email
User profiles
Role based access
, j
Flash, Ajax
Back office testing
Examine server side logs
24
12

15. Examine Server Side Logs
25
Usability Inspection
Usability Inspection:
Navigation
N i ti
Page Content
Intuitive
508 Compliance ‐ accessibility
Search
Sitemap
Help
26
13

16. Usability – User Experience
Page Download Times and Browser Rendering:
No one likes a slow website
N lik l b it
Load testing and performance usually done late in
the test cycle
Measure web page download performance early
– Part of Sanity/Smoke test script.
– Run multiple times and average.
Track page download trends from release to release.
– Test script writes download times to csv.
27
SEO Dated Page Download Trend XPIE6 (Single User)
25
38b5
R6
20 42b2
44b4
47b4
15
49b2
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
28
14

17. Page Download and Render Time
29
Usability – Drilling Down
Page Download Time and HTML elements:
Measure HTML element download times
M HTML l td l d ti
– HttpWatch (works with both Firefox & IE, has an API)
– tools.pingdom.com (to demo object downloads)
– Yslow (Firefox addon)
30
15

18. HttpWatch
31
Cross Browser Testing, iPhone automation, Security, Defect Life Cycle
TESTING PROJECTS
32
16

19. Cross Browser Testing
Cross Browser Testing project:
Created and used a CBT lab.
C t d d d CBT l b
– Various combinations of FF, IE and Windows OS
• XP/IE6, XP/IE7
• Vista/IE7
• XP/Firefox
– Ran automated regression tests on each combination.
– Discovered many cosmetic defects.
– No functional errors found.
– Many companies use Selenium.
33
Cross Browser Testing
Cross Browser Testing project (next steps?):
Considering an HTML/CSS syntax checker / W3C
C id i HTML/CSS t h k / W3C
validator
– Many online tools generate considerable output.
– HTML Validator (Firefox Add on)
– Total Validator (Firefox Add on)
– Litmusapp.com is another consideration
Litmusapp com is another consideration
34
17

20. Automated testing on the iPhone
Automated testing on the iPhone:
A Hotels.com website was created for the iPhone
AH t l b it t d f th iPh
Examined test tools to automate testing of iPhone
web site.
Tried SafariWatir on the Mac without success.
As a reasonable alternative I used FireWatir (Watir
for Firefox) on a PC and ran automated regression
tests for the iPhone web site.
– Minor visual differences vs. Safari
35
Security Testing
Security – start simply (perhaps you already do?):
Invalid inputs in text entry fields and forms
I lid i t i t t t fi ld df
SSL– https is used where appropriate (e.g. forms)
Internal URLs not accessible (unless logged in)
Confirm no access to web server directories
XSS – Cross Site Scripting
p g
36
18

21. Security Testing
Security:
Set everyone's expectations
S t ' t ti
Gather good tools
Look at your application from every perspective
Test for underlying weaknesses
yy g
Go back and verify your scanner findings
Manually check for weaknesses
Test your source code
37
Security Testing
Security – Captcha:
Captcha – detects automated scripts in the wild and
C t h d t t t t d i t i th ild d
blocks them
– Verifying the detection of automated scripts is easy if you
have automated tests
– In order to run automated tests in Production, plan to have
a means to disarm Captcha
p
– Timed re‐arming is preferable such that Captcha is enabled
automatically to protect the site in case you forget.
38
19

22. Defect Lifecycle
Managing the defect lifecycle:
Issues are detailed, descriptive, and concise.
I d t il d d i ti d i
Ensure severity and priority are appropriate.
Ensure there are no unassigned issues.
Hold weekly mandatory review meetings between
QA & stakeholders
Write a defect, write a test case (if none exists)
– Copy steps to reproduce into a new test case. ^C^V
– Great way to “beef up” regression test suite.
39
What’s in your QA Tool Belt?
TEST TOOLS
40
20

23. Test Tools – Browser Add‐ons
Useful Firefox Add ons:
Firebug
Fi b
FormSaver
FireCookie
tamperData (view/modify HTTP/HTTPS)
g
Screengrab
Xpather
41
Test Tools – Browser Add‐ons
Useful Internet Explorer Add ons:
Developer Toolbar
D l T lb
IECookiesView
Fiddler (Watcher – Passive Security Auditor)
Webcollect (screen capture)
y
Web Accessibility Toolbar
Mathon (Swiss army knife)
42
21

24. Test Tools ‐ WATiR
Automated Test Tool:
WATiR – Web App Testing in Ruby
WATiR W b A T ti i R b
– Supports your web app no matter what it is developed in
– Full featured modern scripting language
– Supports multiple browsers on different platforms
– It is powerful and easy to use, yet beautifully lightweight
– There is an active and growing community behind it
There is an active and growing community behind it
– It is free Open Source tool. There are no costs to use the
tool
– User for five years (solid, stable, growing functionality)
43
Test Tools ‐ WATiR
Automated testing results using Watir:
Fully automated the Hotels.com testing
F ll t t d th H t l t ti
– Sanity testing (page download times too)
– Regression Testing (a deployment every week)
– Booking tests were data driven (Excel spreadsheet)
– Three day test cycle (2 resources) reduced to two hours
end to end. Exploratory testing added to process.
end to end Exploratory testing added to process
– Data Center Consolidation (15 app servers, 8 instances on
each, 120 total instances)
44
22

25. Test Tools ‐ Justification
Tips when using automation:
Use a widescreen monitor in portrait mode to
U id it i t it d t
maximize visibility of the whole page.
Use automation to scroll to the bottom of the page.
Record all defect #’s detected by automated testing
– Application Services (weekly releases)
– Projects (web page redesigns)
Branch test scripts to mimic Development code
Calculate your ROI
45
Test Tools – Software Utilities
Test Utilities:
Ruby is a fully featured programming language.
R b i f ll f t d i l
– Gems like NET::SMTP to send mail or pop mail
• Verify emails sent from web app; like change password, click link.
– NET::SSH tail utility pulls server logs to desktop for viewing
– Missed destinations utility (feedback to Dev)
– Run SEO tests on web pages (too tedious to do manually)
Run SEO tests on web pages (too tedious to do manually)
– Site Spider that traverses site starting at the home page
• Able to traverse the whole site with minimal scripting time.
46
23

26. Test Tools ‐ Security
Security:
OWASP.org – web security testing tools
OWASP b it t ti t l
Ethical Hacker Network
HP Dev Inspect (for programmers)
HP QA Inspect (for QA testing)
p (
HP Web Inspect (for Production))
Hosted services; McAfee for production security
testing.
47
Test Tools ‐ Performance
Social Networking:
Facebook
– Can’t “load test” in Facebook’s domain.
– Created simulateUser.php (randomized actions)
• Register new friends
• View canvas, tag other friends with characteristics, save profile
– Ran apachebench (ab) against simulateUser.php
– Facebook application refactored in targeted areas
application refactored in targeted areas
– Placed database in RAM (limited risk)
– 312 to 46K requests/transactions (150 X better
performance )
24

27. Test Tools ‐ Flash
Flash Testing Apps ‐ Commercial:
QTP with plugin (instrument the Flash code)
QTP with plugin (instrument the Flash code)
TestComplete
Ranorex
AutoCzar
TestPlant ‐ EggPlant (image based)
Flash Testing Apps – O
Fl h T ti A OpenSource:
S
T‐Plan Robot
ASUnit
49
Quality Nuggets
Before Deployment Day!
Run your regression test scripts in Prod –
Run your regression test scripts in Prod why?
– Deployment failed, troubleshooting focused on new
release, root cause was a pre‐existing condition in Prod.
– Discovering issues before deployment eliminates the
confusion and unnecessary troubleshooting from assuming
that a new deployment caused the problem.
– By running automated regression the evening before a
By running automated regression the evening before a
deployment, several issues have been found since, some
serious.
– Content Management System changes
50
25

28. QUESTIONS?
51
Biography – Walter Mamed
Walter Mamed is Director of Quality Assurance at JWT (Digital Technology)
in Irving, Texas. He has over 30 years experience in a variety of quality
assurance and software test engineering development positions, focusing
on software and hardware test automation.
Walt has been building test automation frameworks for GUI testing and
web based applications for over 15 years. His web testing experience
includes secure Email, On‐boarding, ecommerce and lead generation as
well as large‐scale automated regression test suites.
Walt is very active in the professional community as Director of the Board
and Secretary for the Dallas/Ft. Worth (HP) Mercury User Group
/
(DFWMUG.com) for the last 7 years. He is an ASQ Certified Software
Quality Engineer.
52
26

29. Acronyms
• API – Application Program Interface
• CBT – Cross Browser Testingg
• CMS – Content Management System
• CSS – Cascading Style Sheets
• CVS – Concurrent Versioning System
• ETL – Extraction, Transformation, and Loading
• FD – Functional Decomposition
• RBT – Risk Based Testing
• RCS – Revision Control System
• SVN – SubVersion
• SQL – Structured Query Statements
• W3C – World Wide Web Consortium
53
27