We all know that Appium can do (almost) everything you tell it to do, but in the end we always ask it to do the same thing: to automate our happy and/or our error flow. The result can give us the confirmation that the user can still buy a product, or verify if his bank account still has a positive balance.
But what if we need to know more than that: what if we also need to verify the layout of our app? The answer would be easy, as there are plenty of open-source or paid solutions that can help you with that. However, almost all of these solutions focus on web/hybrid apps, not on native iOS and or Android apps.
Wim faced this challenge when he was automating a React Native app for a customer. There was no image comparison tool that could do what he wanted it to do, so he rolled up his sleeves and started building one. In this talk, Wim will walk down the path he took and he will explain:
- Why he used ResembleJS as a core visual comparison solution
- The things he learned about Appium in the process (pro tips!)
- The differences between iOS and Android and how he managed to solve this in one cross-platform solution
- The pros and cons of the solution
5. REQUIREMENTS
• Native App support
• Cross platform
• Integrate WebdriverIO
• Paid tools === trial first
• Support blockouts
• Support element screenshots/bounding box
FOR A VISUAL REGRESSION / TESTING SOLUTION
7. RESEARCH
VISUAL REGRESSION / TESTING FOR NATIVE APPS
iOSSnapshotTestCase
Nakal
sightr.io
screenshot-tests-for-android
8. RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-
for-android
iOSSnapshot
TestCase
Nakal sightr.io VisWiz Applitools
Native apps support Yes Yes Yes Yes Yes Yes
WebdriverIO support
- JS
- No extra env
dependencies
No No No No Yes
Not for
native apps
Trial? Open source Open source
Open
source
Beta Yes Yes
Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive
Blockouts No No Yes and no No no Yes
Element screenshots No No No No No No
Bounding box No No No No No Yes
9. RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-
for-android
iOSSnapshot
TestCase
Nakal sightr.io VisWiz Applitools
Native apps support Yes Yes Yes Yes Yes Yes
WebdriverIO support
- JS
- No extra env
dependencies
No No No No Yes
Not for
native apps
Trial? Open source Open source
Open
source
Beta Yes Yes
Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive
Blockouts No No Yes and no No no Yes
Element screenshots No No No No No No
Bounding box No No No No No Yes
10. RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-
for-android
iOSSnapshot
TestCase
Nakal sightr.io VisWiz Applitools
Native apps support Yes Yes Yes Yes Yes Yes
WebdriverIO support
- JS
- No extra env
dependencies
No No No No Yes
Not for
native apps
Trial? Open source Open source
Open
source
Beta Yes Yes
Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive
Blockouts No No Yes and no No no Yes
Element screenshots No No No No No No
Bounding box No No No No No Yes
11. RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-
for-android
iOSSnapshot
TestCase
Nakal sightr.io VisWiz Applitools
Native apps support Yes Yes Yes Yes Yes Yes
WebdriverIO support
- JS
- No extra env
dependencies
No No No No Yes
Not for
native apps
Trial? Open source Open source
Open
source
Beta Yes Yes
Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive
Blockouts No No Yes and no No no Yes
Element screenshots No No No No No No
Bounding box No No No No No Yes
12. RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-
for-android
iOSSnapshot
TestCase
Nakal sightr.io VisWiz Applitools
Native apps support Yes Yes Yes Yes Yes Yes
WebdriverIO support
- JS
- No extra env
dependencies
No No No No Yes
Not for
native apps
Trial? Open source Open source
Open
source
Beta Yes Yes
Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive
Blockouts No No Yes and no No no Yes
Element screenshots No No No No No No
Bounding box No No No No No Yes
?
14. LOGIC
WHO SHOULD DO WHAT?
Requirements Module Comparison engine
Actively maintained - Yes
Environment dependencies No No
Blockout(s) Yes, determine and provide Yes
Element screenshot Yes -
Create baseline Yes -
Screenshots per device Yes -
Support compare options
(for example antialiasing)
Yes Yes
15. LOGIC
WHO SHOULD DO WHAT?
Requirements Module Comparison engine
Actively maintained - Yes
Environment dependencies No No
Blockout(s) Yes, determine and provide Yes
Element screenshot Yes -
Create baseline Yes -
Screenshots per device Yes -
Support compare options
(for example antialiasing)
Yes Yes
28. COMPARESCREEN
WHERE COULD IT GO WRONG, WHERE COULD IT CAUSE FLAKINESS?
Navigation bar Home bar indicatorStatus bar
29. STATUS BAR
• Android and iOS differ in height
• iOS OS versions / devices differ
• Android OS versions / devices differ
• Cross platform / device solution