'BIG Testing' with Hans Buwalda

Today’s webinar is presented by Hans Buwalda and he will
discuss BIG Testing.
Welcome to the EuroSTAR July Webinar Series
www.eurostarconferences.com
This webinar is due to start at 4pm. Make sure you stick around at the end
for the Q&A session and continue the conversation with the speaker on
Twitter after the show!
@esconfs
#esconfs

BIG Testing
Hans Buwalda, CTO, LogiGear (USA)
An internationally recognized expert in testing, Hans Buwalda is the pioneer of
keyword-driven test automation, an approach now widely used throughout the
testing industry. Working in The Netherlands for CMG (now CFI) he helped
develop the successful TestFrameTMframework. He now is CTO of LogiGear
Corporation in California, responsible for the keyword-driven Action Based
TestingTM method, and the tools that support it: TestArchitectTM and TestArchitect
for Visual Studio®. Hans speaks frequently at international conferences and is
coauthor of Integrated Test Design and Automation: Using the TestFrame
Method.
@esconfs
#esconfs

Q&A
If you have a question, the control panel on the right has a Questions
box. All questions will be answered at the end of the presentation.
The GoToWebinar attendee interface is made up of two parts. The
Viewer Window shows the presenter’s screen. The Control Panel is
where attendees can interact and ask questions.
The webinar will be recorded and you will be notified
via email when it is available.
@esconfs
#esconfs

How can I get the
slides?
Slides and all materials will be posted on the EuroSTAR website as well as emailed to you.
@esconfs
#esconfs

Join the conversation on Twitter
#esconfs
@esconfs
@hansbuwalda
Continue the conversation with the speaker on Twitter after the show!
@esconfs
#esconfs

BIG Testing
Hans Buwalda
LogiGear
Tutorial excerpts
Keywords, Test
Design, Automation, Virt
ualization, Globalization

© 2013 LogiGear Corporation. All Rights Reserved
About LogiGear
 Software testing company, around since 1994
 Testing and test automation expertise, services and tooling
consultancy, training
test development and automation services
"test integrated" development services
 Aims to be thought leader, in particular for large and complex test projects
 Products:
TestArchitect™, TestArchitect for Visual Studio™
integrating test development with test management and automation
based on modularized keyword-driven testing
www.logigear.com
www.testarchitect.com

What is "BIG"
 Big efforts in development, automation, execution and/or follow up
 It takes a long time and/or large capacity to run tests (lot of tests, lot of versions, lot of
configurations, ...)
 Scalability, short term and long term
 Complexity, functional, technical
 Number and diversity of players and stakeholders
pigs, chicken, elephants, ankle biters, ...
 Various definitions of "big" possible... and relevant...
"10 machines" or "10 acres"
"1000 tests" or "1000 weeks of testing"
 Big today means: big for you
"non trivial", you need to think about it
"Windows 8 has undergone more than
1,240,000,000 hours of testing"
Steven Sinofsky, Microsoft, 2012

Keywords, essential for scalability
 Distinguish tasks for test development and for automation
 The test developer creates tests using "actions". Each action consists of a keyword ("action
word") and arguments
 Automation focuses on automating the actions, each action is automated only once
however…. actions by themselves are just a format, not a method….

Action Based Testing
Test Development Plan
Test Cases
Test
Objectives
Test Module 1
Test Cases
Test
Objectives
Test Module 2
Test Cases
Test
Objectives
Test Module N
Actions
. . .
ACTION AUTOMATION
Break down
Automate

Test Design
 Effective test breakdown (into test modules)
make sure test modules (test clusters) have a clear focus
keep different kinds and levels of tests separate
 Right level of actions
as “high level” if possible, hiding as many details as much as possible
...but not if the details are relevant for the test
It is my believe that successful automation is not a technical challenge. It is most
of all a test design challenge.

What's the trick...

What's the trick...
 Have or acquire facilities to store and organize you content
 Select your stuff
 Decide where to put what
assign and label the shelves
 Put it there
 If the organization is not sufficient anymore, add to it or change it

Example of a business test module
 Test a business functionality, doesn't show details that are not relevant to that
 Navigation details are avoided, unless they're meant to be tested
TEST MODULE Car Rental Payments
user
start system john
TEST CASE TC 01 Rent some cars
first name last name car
rent car John Doe Ford Escape
rent car John Doe Chevvy Volt
last name amount
check payment Doe 140.4
FINAL
close application

Example of an interaction test module
 In this "low level" test details are not hidden, since they are the target of the test
 The right level of abstraction depends on the scope of the test, and is an outcome of your test
design process
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 "New Order" button
first name control
click main new order
window
check window exists new order
FINAL
close application

Eye on the ball, Scope
 Always know the scope of the test module
 The scope should be unambiguous
 The scope determines many things:
what the test objectives are
which test cases to expect
what level of actions to use
what the checks are about and which events should generate a warning or
error (if a “lower” functionality is wrong)

What I have seen not work
 "Over-Checking": having checks that don't fit the scope of the test
 Forcing data driven: making all tests data driven (variables, data files) without
clear reason
 Combinatorial explosions: test all ... for all ... in all ...
 All actions high level (or all actions low level)
 Many tests for forms and dialogs, little tests for business processes
 Abundance of irrelevant comments, and lack of relevant comments

Choosing actions
 Scope of the test determines the actions
 As high level as appropriate, but not too high
 As little arguments as possible
use default values for arguments not relevant to the scope of the test
 Clear names (usually verb + noun usually works well)
to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify
client" (or vice versa)
tests are not C++ code: avoid "technical habits", like mixed case and underlines
 Manage the Actions
 Document the Actions
 Actions should be by-product of the test design

Example of using actions
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test
key
key navigate F7
key navigate 3
page tab
locate page tab Scan Criteria
w indow
wait for controls loaded search
text
check breadcrumb general functions > search
w indow control value
select search scan direction Backward
w indow control value
enter value search business date match # bus date
source control
click search go
w indow
wait for controls loaded search results
w indow control variable
get search results sequence number >> seq num

variable
get sequence number >> seq num
Example of using actions
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test

Low-level, high-level, mid-level actions
 Low-level: detailed interaction with the UI (or API)
generic, do not show any functional or business logic
examples: "click", "expand tree node", "select menu"
 High-level: represent a business function specific to the scope of the test
hide the interaction
examples: "enter customer", "rent car", "check balance"
 Mid-level: auxiliary actions that represent common sequences of low level
actions
usually to wrap a form or dialog
greatly enhance maintainability
example: "enter address fields"
enter customer
enter address fields
enter select set . . .. . .

Test Development and Automation in sprints
Test Module
Definition
(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint ProductsProduct Backlog
Test re-use
Automation re-use
product
owner
team
prod owner &
team
User stories
Documentation
Domain understanding
Acceptance Criteria
PO Questions
Situations
Relations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules

Some Tips to Get Stable Automation
 Make the system under test automation-friendly
 Use "active" timing
always wait for something, never just wait a fixed amount of time ("passive
timing")
 Test your automation
 Use automation to identify differences between versions of the
system under test
 Keep an eye on the test design

 Look for properties a human user can't see, but a test tool can
 This approach is a must-do for speedier and more stable automation
interface mapping is often bottleneck, and source of maintenance problems
with predefined identifying property values an interface map can be created without "spy" tools
not sensitive to changes in the system under test
not sensitive to languages and localizations
 Examples:
"id" attribute for HTML elements
"name" field for Java controls
"AccessibleName" property in .Net controls (see below)
Automation-friendly design: hidden properties

Environments, configurations
 Many factors can influence details of automation
language, localization
hardware
version of the system under test
system components, like OS or browser
 Test design can reflect these
certain test modules are more general
others are specific, for example for a language
 But for tests that do not care about the differences, the automation just needs
to "deal" with them
shield them from the tests

Capture variations of the system under test in the actions and interface definitions, rather than in
the tests (unless relevant there).
Can be a feature in a test playback tool, or something you do with a global variable or setting.
Variation Variation Variation
"Variations"
"Master Switch"
Actions, Interface Definitions
. . .

Possible set up of variations
linked variation
keyworded variation
Specify for example in a dialog when you start an execution:

Virtualization
 Virtual machines rather than physical machines
allow "guest" systems to operate on a "host" system
host can be Windows, Linux, etc, but also a specialized "hypervisor"
the hypervisor can be "hosted" or "bare metal"
 Main providers:
VMWare: ESX and ESXi
Microsoft: Hyper-V (readily available in Windows 8)
Oracle/Sun: Virtual Box
Citrix: Xen (open source)
 Hardware support gets common now
processor, chipset, i/o
for example "slat" (second level address translation, helps map addresses to memory)
 For most testing purposes you need virtual clients, not virtual servers
most offerings in the market currently target virtual servers, particularly data centers
 Virtual clients will become more mainstream with the coming of VM's as part of regular operating systems
Windows 8: Hyper-V
Linux: KVM

Virtualization, a testers dream...
 In particular for functional testing
 Much easier to define and create needed configurations
you basically just need storage
managing this is your next challenge
 One stored configuration can be re-used over and over again
 The VM can always start "fresh", in particular with
fresh base data (either server or client)
specified state, for example to repeat a particular problematic automation situation
 Can take "snap shots" of situations, for analysis of problems
 Can use automation itself to select and start/stop suitable VM's
for example using actions for this
or letting an overnight or continuous build take care of this

Virtualization, bad dream?
 Performance, response times, capacities
 Virtual machine latency can add timing problems
see next slide
can be derailing in big test runs
 Management of images
images can be (very) large, and difficult to store and move around
• there can be many, with numbers growing combinatorial style
• configuration in the VM can have an impact, like fixed/growing virtual disks
distinguish between managed configurations and sandboxes
define ownership, organize it
IT may be the one giving out (running) VM's, restricting your flexibility
 Managing running tests in virtual machines can take additional efforts on top of managing the
VM's themselves
with the luxury of having VM's the number of executing machines can increase rapidly
one approach: let longer running tests report their progress to a central monitoring service (various
tools have features for this)

Virtual machines, capacity
 Key to pricing is number of VM's that can run in parallel on a physical machine
 An automated test execution will typically keep a VM more busy than human
use
 Factors in determining VM/PM ratio:
memory, for guest OS, AUT, test tooling
storage devices (physical devices, not disk images)
processors, processor cores
specific hardware support (becoming more common)
• processor, chipset, I/O
We started regression with 140 VMs.
Very slow performance of
Citrix VM clients.

Globalization
 Three Challenges:
other countries, other cultures
geographic distances
time differences
 Seven "Patterns":
"Solution"
"Push Back"
"Time Pressure"
"Surprises"
"Ownership"
"Mythical Man Month"
"Cooperation"
other country
surprise

Three Challenges
 Other country
tricky, but can be managed
study and prepare (culture, governance, etc)
have a clear company culture to complement regional
(and professional) culture
 Distance
cumbersome
have good (great) communication features, at all levels
 Time difference
this is the biggest problem, comes back every day
as much as possible have local management and responsibility
company culture

Seven patterns
 "Mythical Man Month"
more (inexpensive) resources does not mean faster timelines
 "Solution"
team "solves" problems rather than asking
 "Push Back"
on-shore side dictates work and practices in detail
tell teams to push back if necessary
 "Time Pressure"
deadlines are quite often imposed without need
 "Surprises"
bad news is communicated too late
 "Ownership"
establish clear ownership: shared responsibility is no responsibility
 "Cooperation"
distance creates confusion and conflict (human nature)
tip: work with briefs and agendas for meetings

Industrial Organization
 Large scale testing can move from a "design" to a "production" focus
mostly applies to test execution, but also seen for test development
this not black and white, both paradigms can occur in the same projects
 A production organization is different a development organization
different professional culture
emphasis more on delivery and scale, "thinking big"
discipline rather than creativity, "get stuff done"
activities are like planning, control, logistics, information
 Responsibilities
Keeping the tests running
Allocating resources
Respond to hick-ups
Analyze and address automation issues

Summary
 Not all "big project" challenges are the same
 Think before you do. Best results come from combining effective
concepts, tricks and tools
 Consider tests and automation as products that need planning and design
 Virtualization can be a solution, but also a challenge
 Globalization can help scalability, but needs knowledge and attention
 Big testing projects typically have a "production" focus, in addition to test
development and automation

Questions?
@esconfs
#esconfs

EuroSTAR Webinar Archive
Access almost 70 on-demand software testing webinars by the world's leading
testing experts. Topics include….
Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test
Documentation, Cloud Testing, Session-based Testing, Weekend Testers
Visit the archive!
http://www.eurostarconferences.com/community/member/webinar-archive
@esconfs
#esconfs

The EuroSTAR Community.
Are you a member?
Did you know we have an online resource area with almost …
600 software testing presentations, 70 topical webinars, over 40 videos
and podcasts and 25 expert eBooks
Do you know that membership is completely free and you can access these premium
testing resources anytime?
Join today!
http://www.eurostarconferences.com/community/
member
@esconfs
#esconfs

Are you coming to Gothenburg?
Join us in the Swedish Exhibition &
Congress
Centre, Gothenburg, Sweden from
the 04-07 November for the 21st
annual EuroSTAR Conference.
Visit www.eurostarconferences.com
for more information on what you
can expect to see at Europe’s largest
software testing event!
@esconfs
#esconfs
Book before Sept

Have you seen the latest competition?
You and 3 of your team could be joining us at the 2013 EuroSTAR Conference!
Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet
goals or solve problems.
Deadline for entries is Wednesday, August 14th!
More information here: http://www.eurostarconferences.com/content/teamstar-2013@esconfs
#esconfs

Don’t miss these great savings, book your ticket today!
Plus Make even bigger savings when you register for the Group discount
where every 5th attendee goes free!
For more information on the Conference Programme and pricing options
go to www.eurostarconferences.com@esconfs
#esconfs

'BIG Testing' with Hans Buwalda

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a 'BIG Testing' with Hans Buwalda

Semelhante a 'BIG Testing' with Hans Buwalda (20)

Mais de TEST Huddle

Mais de TEST Huddle (20)

Último

Último (20)

'BIG Testing' with Hans Buwalda

Notas do Editor