How does an organization effectively decide when to use browser emulation (in case of hybrid apps), mobile simulators/emulators and real devices? In this talk, Kwo Ding will review his mobile test pyramid approach and discuss best practices about when to use what. He will also talk about how to best structure and execute these tests.

1. Kwo Ding
Test Automation Consultant
SauceCon
7 June 2017
Mobile Test PyramidUI integration testing for mobile

2. Kwo Ding
Freelance Test Automation
Consultant
About Kwo
Over 10 years experience in software testing
Currently active in the financial industry
Specialized in implementation of test (automation) strategies
Experienced and certified Scrum Master, Product Owner and Lean Green Belt
Specialized in automation of web, mobile and API testing
Based in the Netherlands
Who Am I?
kwo@dingit.net
https://www.linkedin.com/in/kwoding
https://www.facebook.com/kwoding
@dingkwo

3. Mobile Testing Challenges

4. Browsers Simulators/Emulators Real devices
What are their strengths and how to apply?

5. Test Pyramid
Unit
Service
UI
Mike Cohn, Succeeding with Agile (2009)

6. Test Guidelines
Test as isolated as possible
Automate as much as possible
Zero tolerance on flakiness
Fast feedback (test as early as possible)
Regression = broken build
No risk = no test
NFRs are just as important as funcAonals

7. Mobile Test PyramidTest Guidelines
Test as isolated as possible
Automate as much as possible
Zero tolerance on flakiness
Fast feedback (test as early as possible)
Regression = broken build
No risk = no test
NFRs are just as important as funcAonals
Real devices
Simulators/Emulators
Browsers

8. Browsers
Mobile testing on desktop browsers

9. Pros and ConsUsage of desktop browser testing for mobile
Matter of milliseconds to launch a browser, also
headless possible
Fast execution
Easily set up 10+ browser instances per machine
Scalable
Ability to use browsers on different operating
systems
Cross platform
Mobile simulation in desktop browsers is still using
the desktop browser
Mobile simulation uses desktop browser version
No native keyboards, incoming calls etc.
No native integration
Incredibly fast, but it’s just not a device
Just not a device…

10. Focus AreasDesktop Browser Testing
Isolated browser tests performing full
functional validations
Resizing browsers and toggling user agents
Functional system testing
Responsive design
Use equivalent desktop browsers
No extensive visual checks because the
rendering is different than actual devices
Cross-browser
Overall visual layout

11. Demo Browser Testing

14. So mobile testing on desktop browsers is great, right?

15. Mobile Simulation in Browser vs Device

16. Mobile Simulation in Browser vs Device

17. Mobile Simulation in Browser vs Device

18. Mobile Simulation in Browser vs Device

19. Simulators/Emulators
Closer to the actual mobile experience

20. Pros and ConsUsage of mobile simulators/emulators
Simulators/emulators are easy to set up, just
download, install and run
Easy to set up and free
Virtualization means scalable and also running in
parallel on one machine
Scalable
Ability to test native APIs such as incoming calls and
GPS injection
Native API integration
Simulators are fast, because they only have to
simulate the software part. Emulators based on the
Intel architecture are fast.
Simulators or Intel-based emulators are fast
Manufacturer’s skins are available, but the device
behavior is still based on stock
Vanilla versions only
CPU/memory usage of machine in case of simulators.
Emulators try to simulate the hardware
No real resource usage
Connectivity with NFC, Bluetooth, network
connections
No real interoperability
Emulators based on the ARM architecture are slow,
which is actually the main architecture for Android
devices
Slow ARM-based emulators
Contrast/brightness inaccurate in light/dark
environment
Inaccurate color display in light/dark
Easy to debug simulator/emulators, already hooked
up to the machine to access logs
Debugging possibilities

21. Focus AreasSimulator/Emulator Testing
Click paths throughout the application
GPS injection, file attachments, incoming
calls etc.
Functional end-user flows
Native API integration
Emulators are limited to vanilla versions
Visuals (vanilla only)
Touch interactions such as swipe and tap
comes closer to the user experience of a
device than browser emulation
Touch interactions

22. Demo Simulator/Emulator Testing

24. Real Devices
The real thing…

25. Pros and ConsUsage of real devices
Ability to test native APIs not only with injections for
automation, but also actual NFC touch for example
Native APIs in real conditions
Some real devices are just faster than emulators due
to the simulation of hardware, especially compared to
the ARM-based emulators
Can be faster than emulators
Real devices come with a price, usually you pay per
device/cradle
Costs
A new device is usually not available on-the-fly, even
with cloud solutions
New device means procurement
Actual network conditions, battery/CPU/memory
usage, actual manufacturer’s sauce on top of the OS
Just the real thing…
iOS apps need to be signed with Development
Distribution Certificate and Provisioning Profile for
automation
Development iOS build required for automation

26. Focus AreasReal Device Testing
Validating usability such as actual click
areas, touch actions and voice over
CPU/memory usage, battery, network
strengths
Interruption (incoming calls, push
notifications), resource fighting (camera,
GPS), NFC, Bluetooth
Usability
Performance
Native API integration
Focus on devices which are not available as
simulators/emulators
Real OS from manufacturers
With specific manufacturer’s OS it comes
with built-in browsers which are different
than the standard and well-known
browsers such as Chrome
Visuals
Manufacturer’s sauce
Specific built-in browsers

27. Demo Real Device Testing

31. Best practicesMobile Testing
Use native gestures for usability
testing such as touch and tap
Scroll elements into view
before clicking
Use native keyboards instead of
injecting text or “sendKeys”
Webview ≠ mobile
browser rendering
Take technical click areas into
account (“fat finger click”)
Launch a “clean”
device before test
Generic Automation

32. Real devices: real-life conditions
Performance testing, native API integration such as
incoming calls, GPS, background apps, NFC etc.
Real devices: visual and usability testing
Visual validations and usability flows
Simulators/Emulators: functional end-user flows and visuals
Fast feedback on end-user flows with touch
interactions and visuals
Browsers: system testing and responsive design
In case of hybrid and web apps: Responsive design
and system testing
Mobile Test Pyramid
Real devices
Simulators/Emulators
Browsers

33. Contact
Kwo Ding
Freelance Test Automation
Consultant
kwo@dingit.net
https://www.linkedin.com/in/kwoding
https://www.facebook.com/kwoding
@dingkwo