A free webinar produced by the Technology Association of Oregon's Quality Assurance Forum Group. Presented by Trent Peterson, CEO and Co-Founder of Portland-based AppThwack, providing fast automated mobile device testing services for Android, iPhone, iPad, and mobile web apps. The recording to the webinar can be accessed here: http://youtu.be/r9Vm6ssSEU8

1. Testing for Android & iOS
When, Where, and How to
Successfully Use Test
Automation
Trent Peterson | @tdpeterson | trent@appthwack.com

2. Who?

3. AGENDA
What is testing?
What’s the problem?
Why should I care?
When: Choosing our battles
Where: Best Practices
How: Tools & Frameworks
Q&A

4. INTRO TO TESTING

5. Types of tests
•
•
•
•
•
•
•
•
Functional
Non-functional
Integration
Unit
Regression
Smoke
Usability
Performance
• Stress
• Acceptance
• Load
• Sanity
• End-to-End
• System
• Security
And on and on…

6. Testing distilled for this discussion
Functional Testing
Non-functional Testing

7. THE PROBLEM

8. Building things is hard.

9. WHY SHOULD I CARE?

10. So what’s the big deal?
1,000,000+ apps in
1,000,000+ apps in
2 crashes and 84% will uninstall
- Compuware as reported in TechCrunch

11. We’ve been here before.

12. CHOOSING OUR BATTLES

13. First, what’s our goal?
Save time!
BE REALISTIC
Save money!
• Increase regularity of testing
Shrink QA!
• Free up resources for other testing efforts
• Enable previously impossible tests
AUTOMATE
EVERYTHING
!

14. Deciding what to automate
How often will I test this?
Is this something a human is good at?
How much effort will automation require?

15. Some examples
AUTOMATE
MANUAL
• Granular functionality
(unit tests)
• Repetitive tasks
(update
paths, navigation, etc.)
• Performance
• UX
• Responsiveness and
“feel”
• New functionality

16. Our options (Machine)
JVM
Emulators
Real Devices
• FAST
• Unit tests
only*
• Mocks
• Cheap
• Catch layout
issues
• Simulate
calls/SMS
• SLOW
• Illusion of 1:1
with real
devices
• Exactly what end
customers use
• Performance data
• Fast
• Expensive:
Purchase, maintai
n, etc.

17. Our options (Human)
QA
Crowd
End Users
• Pros
• Real
feedback
• Know the
product
• Expensive
• Real feedback
• Might know
product
• Expensive
(usually)
• Slow
• Don’t do this.

18. BEST PRACTICES

19. No barriers between Dev & QA

20. Automation is a software project

21. Automation is a tool, not a solution

22. Automate with precision

23. It’s all about data

24. Know your matrix

25. Abstraction is important
…but don’t obsess

26. TOOLS and FRAMEWORKS

27. Tool: Recorders
GOOD
BAD
• Fast
• Non-developers can
create tests
• Fragile
• Difficult to maintain
• Non-developers can
create tests
BEST PRACTICE
• Treat as a fancy spy

28. Tool: UI Exerciser Monkey
$ adb shell monkey
–p my.sweet.app –v 1000

29. Framework: Android Testing
SUMMARY
• JUnit/Java
• Instrumentation provides hooks into Android
SDK to control Android-specific functionality
• Very basic functionality supported
• The foundation for most Android testing. Learn
it, use it, love it.

30. Framework: Robolectric
SUMMARY
• JUnit
• Runs in JVM
• Mocks Android functionality with “shadow”
objects
• Super fast
• Not everything is mocked (also, mocks)
• Great for staying sane while developing

31. Framework: UI Automator
SUMMARY
• JUnit/Java
• Extends testing framework with UI selectors
and manipulators
• Handle/control system dialogs, etc
• 4.1+ only
• Excellent choice if you have the luxury of
supporting 4.1+.

32. Framework: Espresso
SUMMARY
• JUnit/Java
• Extends testing framework with UI selectors
and manipulators
• Removes boilerplate code (sleeps, etc)
• Very new and unstable
• If you’re feeling adventurous give it a try

33. Framework: Robotium
SUMMARY
• JUnit extension
• Runs on emulators and real devices
• Oriented towards black-box testing and
ensuring real-world outcomes
• Limited to the app under test
• Great choice for developers going beyond
basic unit testing and testing UI.

34. Framework: Cucumber-based
SUMMARY
• Human-readable tests
• Runs on emulators and real devices
• Cross-platform
• Complex scenarios require development
• Behavior-driven design. Be careful!
• Great choice for simple flows and teams with
limited developer resources.

35. Framework: Appium
SUMMARY
• Supports many language adapters
• Runs on emulators and real devices
• Cross-platform
• New, and not all WebDriver concepts map
intuitively to native apps
• Great choice for teams happy with Selenium
and expanding to native.

36. Framework: UI Automation
SUMMARY
•
•
•
•
JavaScript
Installs with Xcode
Functions with other instruments
Has a recorder (Careful)
• Dev/maintenance of JS
• Good choice to get basic tests up and running
quickly.

37. Framework: XCTest
SUMMARY
• Objective C
• Installs with Xcode
• Poor documentation
• Obvious choice for unit testing.

38. Framework: KIF
SUMMARY
• Objective C
• Enables functional test automation capabilities
by providing UI helpers
• Relies on undocumented APIs
• Great choice for devs automating functional
tests.

39. Tool: Spoon
Spoon
Instrumentation Tests

40. Tool: Bots
Continuous Integration
Execute tests, etc.

41. Frameworks: Commercial
GOOD
BAD
• Support
• Added features and
integrations
• Longevity?
• Proprietary
• Often heavy and
difficult to maintain
• $$$
ADVICE
• Evaluate very carefully and acknowledge lock-in.

42. When choosing, ask…
What are we testing?
What are our automation goals?
Who will write the automated tests?

43. SHAMELESS PLUG

44. AppThwack
•
•
•
•
100s of non-rooted devices
Results in minutes
Built-in compatibility tests
Supports all popular
automation frameworks
• Simple API and integrations
• Focused, comprehensive
reports

45. Q&A
Trent Peterson | @tdpeterson | trent@appthwack.com