1. © Utopia Solutions
Building a Strategy for Test Automation Success
Lee Barnes, CTO
Utopia Solutions

2. © Utopia Solutions
Agenda
•What is the end goal?
•Why do we need a strategy?
•What do we need to consider?
•What should a strategy look like?
Test Automation Strategy Overview
Interactive Case Study

3. © Utopia Solutions
SECTION 1
What is Our End Goal?

4. © Utopia Solutions4
Test Automation Success Criteria
• Issue detection and recovery
• Accurate verification
• Unattended execution
Reliable
• Minimum sensitivity to application and test
case changes
• Test case definition separate from
automation code
Maintainable
• Test coverage expanded efficiently
• Automated test cases created by non-
automation resources
Scalable

5. © Utopia Solutions5
Test Automation Approach
Test Automation Framework
Automated Test Cases Test Results
Test Definition
Test Execution
Non-technical / Test Analysts & Subject Matter Experts
Technical / Automation Engineers

6. © Utopia Solutions
SECTION 2
Why Do We Need a Strategy?

7. © Utopia Solutions8
Why Test Automation Fails
Organizational
• Lack of support
• Unrealistic expectations
• Disjointed org structure
• Missing skill sets
Process
• Immature testing process
• Segregated processes
• Lack of test environment / data
control
Technical
• Poor tool support for AUT
• Under or over design automation
approach

8. © Utopia Solutions9
Are We Ready?

9. © Utopia Solutions
SECTION 3
What Do We Need to Know First?

10. © Utopia Solutions11
Test Automation Readiness Assessment
Org
• Structure
• Support
• Goals and
Expectations
• Resources and Skills
Process
• Test Maturity
• Test Assets
• Process Integration
• Test Environment /
Data
Tech
• AUT Technology
• AUT / Tool
Integration
Assessment
Topics

11. © Utopia Solutions
Where Do We Get the Information?
Assessment
Input
Stakeholders
Existing
Assets
AUTs

12. © Utopia Solutions
Ideal Organizational Attributes
Structure
• Centralized
testing
function
• Testing is a
peer of
development
• Automation
roles are
defined
Goals and
Expectations
• Automation is
an investment
• 100%
automation is
not likely
• Automation
requires
specialized
skills
• Automation
will free up
resources to
work on high-
value tasks
Support
• Executive
management
has visibility to
automation
• Automation
has a
dedicated
budget
Resources and
Skills
• Resources are
dedicated to
automation
• Resources are
skilled in
automation as
a process AND
technology

13. © Utopia Solutions
Ideal Process Attributes
Test Process
• Test objectives
are clear and
measurable
• Test cases are
defined and
documented
• Expected
results are
predictable
SDLC Integration
• AUT in a
frequent
release state
with visibility to
future roadmap
• Visibility to
system changes
as early as
possible in
release cycle
Test Environment
• Test
environment
dedicated to
automation
• Test data
controlled

14. © Utopia Solutions
Ideal Technology Attributes
AUT Technology
• Multiple AUTs built on
common technology
and/or development
framework
• Consistent use of UI
objects
• Consistent navigation
• Consistent presentation
of user feedback
(warnings, errors, etc.)
Tool / AUT Integration
• All UI objects supported
by tool
• Any alternative interfaces
(web services, API, DB,
etc.) supported by tool
• All target end user
environments supported
by tool

15. © Utopia Solutions
SECTION 4
Common Assessment Findings

16. © Utopia Solutions17
Organizational
Distributed
Testing /
Automation
Efforts
App
App
App
App
• Uneven automation
resource demand
• Inconsistent approach
• Duplicated efforts
• Restricted skill growth
and knowledge sharing
Leads to…

17. © Utopia Solutions18
Organizational
AUT A
AUT B AUT C
AUT D
Centralized
Test Automation
Team
Common Resources,
Processes &
Technology
• Flatten demand curve
• Accelerate skills
development
• Leverage reusable
knowledge and technology
Benefits
Centralize Automation
Activities

18. © Utopia Solutions19
Process
Analyze Design Build
Test Deploy
No ability to identify impact of app changes on
automated test suite
Unexpected /
Unpredictable
Maintenance

19. © Utopia Solutions20
Process
Analyze Design Build Test Deploy
Capture potential impacts as early as possible to enable
proactive maintenance
Capture App Changes
Analyze
Impact
Maintain Execute
Automation Activities
Application SDLC

20. © Utopia Solutions21
Environment
Shared Test Environment
Uncontrolled Test Data
Integration
Functional
Performance
Integration
Test
• Schedule conflicts
• Unreliable execution
• Unnecessary “hardening”
of the test suite
Leads to…
Automation

21. © Utopia Solutions22
Environment
Shared Test Environment
Controlled Test Data
Integration
Functional
Performance
Integration
Test
Dedicated Environment for
Automation
Test Data Controls
Dedicated
Environment
Controlled
Test Data
Automation
• Restore to known state
• Partition data related
to specific test
activities
• Create required data
each execution

22. © Utopia Solutions
SECTION 5
What Does a Strategy Look Like?

23. © Utopia Solutions24
Mobile Test Strategy Topics
Test
Automation
Strategy
Process &
Organization
Environment
TechnicalResources
Scope &
Roadmap

24. © Utopia Solutions25
Automation Strategy – Process and Organization
• Political and financial support
• Realistic expectations
• Centralized automation function
Key Organizational Components
• Test objectives and coverage goals
• Integration with SDLC → proactive
maintenance
Key Process Components

25. © Utopia Solutions26
Automation Strategy – Environment
• Dedicated environment for execution
• Controlled test data
• Controlled code migration
Key Environment Components

26. © Utopia Solutions27
Automation Strategy – Resources
• Roles
• Skills
• Time
Key People Components
• AUT test environment(s)
• Test suite development
• Test suite execution
Key Hardware Components
• Test tool
• Supporting utilities
Key Software Components

27. © Utopia Solutions28
Automation Strategy – Scope
• Test data setup
• AUT / environment preparation
• Complex / tedious results verification
Test Support Tasks
• Applications built on common technology
• Applications with frequent releases & friendly UI’s
Application Selection
• Frequently tested
• High test case to UI ratio
• Predictable results
• Easy to automate
Test Case Selection

28. © Utopia Solutions29
Automation Strategy – Roadmap
• Begin with required organizational and process changes
• Overlap with base framework and minimal coverage
• Enhance and add test coverage over small iterations
Create Implementation Timeline
Phase nPhase 1
Continuous Process & Organizational Improvements
Base Framework
Initial Coverage
AUT 1
Framework Enhancements
Expand Coverage
Initial Coverage
Phase 2
AUT 2
Expand Coverage
Expand Coverage
Initial Coverage
Phase 3
AUT 3

29. © Utopia Solutions
SECTION 6
Case Study

30. © Utopia Solutions31
Key Assessment Findings
Organization
• All enterprise testing activities
currently performed centrally
• Management believes automation
is a tool implementation
Process
• Manual tests are well structured,
but not maintained consistently
• Rely on SME’s to execute tests
Environment
• Manual regression test performed
in a isolated environment
• Test data refreshed in all
environments once per quarter
Technology
• All AUTs have browser based UI
• Developed on disparate
technologies
• Test tool supports all AUT
components
Resources
• Test organization consists of non-
technical SME’s
• System performance (mobile)
• Device performance

31. © Utopia Solutions32
Key Summary Points
• DO NOT automate without a clear vision and plan
• Understand your readiness from these perspectives
• Organization
• Process & Environment
• Technical
• Resources
• Implement in small phases

32. © Utopia Solutions33
Questions…
…. and Answers!
Direct future questions to:
Lee Barnes
Founder and CTO
Utopia Solutions, Inc.
Email: lee.barnes@utopiasolutions.com
Twitter: twitter.com/USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: www.RaiseYourMQ.com