Workshop on "Getting started with Visual Testing using Applitools - @TPC (Test Practitioners Club, NCR), Feb2020"

1. Getting started with Visual
Testing using Applitools
@BagmarAnand
Anand Bagmar
Software Quality Evangelist, Essence Of Testing

2. About Me
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand

3. Game time!
@BagmarAnand

4. How do you test ….
@BagmarAnand

5. @BagmarAnand

6. http://littlecoffeefox.com/wp-content/uploads/2017/05/DSCN1738.jpg
@BagmarAnand

7. @BagmarAnand

8. @BagmarAnand
http://www.rfwireless-world.com/images/telecom-billing-system-architecture.jpg

9. What does this mean?
@BagmarAnand

10. What is missing here?
@BagmarAnand

11. How do you ensure what
was working before, is
working now as well!
@BagmarAnand

12. Spot the difference!
@BagmarAnand

17. It happens in Software too!
@BagmarAnand
https://cdn.pixabay.com/photo/2013/07/12/15/58/smiley-150663_1280.png
Surprise

20. It happens to Twitter
@BagmarAnand

21. It happens to UPS
@BagmarAnand

22. It Happens to Financial Times
@BagmarAnand

23. It Happens to Amazon
@BagmarAnand

24. UX / Visual Testing
The Missing Piece
@BagmarAnand

25. Test Automation Pyramid
Unit (xUnit / JavaScript)
Component
Integration
View
Web Service
UI
Manual / Exploratory
@BagmarAnand

26. Visual Testing
How do you do it?
@BagmarAnand

27. • Mostly done manually done
• Functional Automation can help
@BagmarAnand
Visual Testing

28. • Business / Revenue loss
• Brand / Credibility loss
• You loose your users!
@BagmarAnand
If Visual Testing is NOT done …..

29. • Mostly done manually done
• tedious
• very error prone
• impossible to scale and repeat – especially in days of CD!
• Functional Automation may help – a little
@BagmarAnand
Challenges of Visual Testing

30. Solution
Visual Test Automation
@BagmarAnand

31. • Create baselines by taking screenshots of expected UI
• Compare screenshots
• Whole page
• Snippets
• Update baselines as product evolves / changes
@BagmarAnand
How does it work?

32. Have you used any
Visual Testing tools?
@BagmarAnand

33. Challenges of Automating
Visual Testing
@BagmarAnand

34. • False positives / negatives
• Baseline maintenance
• Browsers
• Devices
• Resolutions / Viewports
• Result analysis & maintenance
@BagmarAnand
Challenges of Automated Visual Validation

35. AI in Visual Test Automation
@BagmarAnand

36. Visual Test Automation
needs a Strategy!
@BagmarAnand

37. A Simple
One Page,
Single Browser
Example
39

38. First, Let’s Provide Initial Coverage
18 Lines of
Code
21 Locators
and Labels
60 Minutes
40
Navigatio
n
Button
Click
Validatio
n

39. NEW
41
Real Bug Escapes
Text Changes
Caught
New Feature
Missed
Visual Bug
Escapes
Broken Locators
Next, We Test a New Build
2 Late
Stage Bugs
2 Changes
Caught
2 False
Positives
Baseline New Candidate
New Feature
Missed

40. Test Results Maintenance Action Time
Functional Bug Write Another Validation 15 mins
Visual Bug Try Writing New Validations, Then Give Up 15 mins
Placeholder Text Differences Update Existing Validations 10 mins
New Terms & Conditions
Feature
Write Another Validation 5 mins
Changed Locators Update Locators 15 mins
Total Time Spent 60 mins
Finally, We Have to Maintain our Tests
NEW

41. Plus Those 21 Locators and Labels That Might
Change Without any Notice!
Build and Maintenance Summary
43
Build & Maintain:
120 Minutes
Bug Detection:
40%
False Positive Rate:
40%

42. Login Page to Test
Remember Our
Simple Example?
Let’s Try Again.
44

43. 45
Let’s Provide Initial Coverage Using Visual AI
2 Minutes to Create
Vs. 60 Minutes
1 Locator
Vs. 21 Locators
and Labels

44. 46
Visual AI Handles the New Build
NEW
Functional Bug
Found
Text Changes
Found
Visual Bug Found
New Feature
Found
Original New Build
No Broken
Locators
All Bugs Found
Text Changes and
New Feature
Detected
0 Broken Locators

45. Test Results Visual AI Maintenance Action Legacy Visual AI
Functional Bug None; Found Automatically 15 mins 0 mins
Visual Bug None; Found Automatically 15 mins 0 mins
Placeholder Text Differences “Thumbs Up” for Each Difference 10 mins 40 secs
New Terms & Conditions Feature “Thumbs Up” for Difference 5 mins 20 secs
Changed Locators Locators No Longer Used 15 mins 0 mins
Total Time Spent 60 mins 1 min
Visual AI Streamlines Test Maintenance
NEW

46. Legacy Test Automation
Visual AI Delivers Results
48
Visual AI
40x Faster
21x More Stable
2.5x Increase in
Bug Detection
Eliminate False
Positives!
Build & Maintain:
3 Minutes
Locators:
1
Build & Maintain:
120 Minutes
Locators and Labels:
21
Bug Detection:
100%
Bug Detection:
40%
False Positive Rate:
0%
False Positive Rate:
40%

47. 49
Legacy Test
Stack
Plus Applitools Visual
AI
Test
Frameworks
Selenium, Appium,
Cypress, Manual
Testing, Visual
DOM/Diff Tools, Etc.
80% Less Code
40x Faster
21x More Stable
Testing &
DevOps
Infrastructure
Jira, GitHub, BitBucket,
Slack Jenkins, Travis,
Circle, Azure, Sauce,
Perfecto, Browserstack
30x Faster
50% $$ Savings
Early Stage
Bug
Detection
Catch 2.5x More Bugs
No False Positives!
Fix Bugs In Minutes (Not
Hours)
Incredibly Accurate, Efficient Test Automation

48. Demo
@BagmarAnand

49. Test Pyramid … includes UX & NFRs
Unit (xUnit / JavaScript)
Component
Integration
View
Web Service
UI
P
e
r
f
o
r
m
a
n
c
e
S
e
c
u
r
i
t
y
Accessibility
UX
@BagmarAnand
Manual / Exploratory

50. • The Practical Test Pyramid -
https://martinfowler.com/articles/practical-test-pyramid.html
• Testing Pyramids & Ice-Cream Cones -
https://watirmelon.blog/testing-pyramids/
• Applitools – www.applitools.com
@BagmarAnand
References

51. Thank you!
Anand Bagmar
EssenceOfTesting.com
@BagmarAnand