O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Visual Testing Of Web Apps in a Headless Environment

228 visualizações

Publicada em

slides for the VodQA talk on how to create a repeatable robust setup for visual testing of web applications using Docker, Xvfb inside a CI/CD setup.

Publicada em: Software
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Visual Testing Of Web Apps in a Headless Environment

  1. 1. VISUAL TESTING OF WEB APPS IN A HEADLESS ENVIRONMENT Manish Katoch, Software Developer 1
  2. 2. THE PREMISES Team Alpha wants to build a perfect web application. “ We want to it be robust, bug free, stellar! The best of the best. We will provide an unparalleled user experience!” 2
  3. 3. THE STRATEGY 3 all is good until.. “How do we ensure we don’t break the UI of our product?” “We have GUI tests, it will ensure all our functionality is flawless!” “But what about aesthetics, the look and feel?” “Err…” UI Tests API/Integration Tests Unit Tests
  4. 4. VISUAL TESTS SAVES THE DAY 4 1 2 Go to screen Take a snapshot While there are more screens Save images 1 2 Go to screen Take a snapshot While there are more screens Compare images Identical Differences First Pass Successive Passes
  5. 5. THE SETUP 5 Code committed Build Created Unit/Integration tests Git CI-CD system Functional Tests Visual Tests No display Agents (virtualized or physical) (Build and Unit tests) Test agents with displays (Functional and visual tests)
  6. 6. BUT… 6 “The tests are flaky they thought, the tools insufficient! the identical images won’t match. ”
  7. 7. WHY MY TESTS ARE FLAKY? Pixel Differences ■ pixel per inch (PPI). ■ PPI is defined as number of pixel that a display can project. ● PPI is calculated as 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙 𝑜𝑛 𝑎 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑠𝑖𝑧𝑒 𝑜𝑓 𝑑𝑖𝑠𝑝𝑙𝑎𝑦 𝑖𝑛 𝑖𝑛𝑐ℎ 7
  8. 8. WHY MY TESTS ARE FLAKY? - II 8 For 1024x768 resolution 98.4 PPI 85.33 PPI 75.29 PPI The floating point arithmetic is hardware dependent. It can differ for same manufacturer too.
  9. 9. WHY MY TESTS ARE FLAKY? - III What if we could make PPI a constant whole number? Good idea. Problem: PPI is determined by hardware display. Can we program hardware displays? 9
  10. 10. WHY MY TESTS ARE FLAKY? - IV 10 Graphics Processing Unit • Processes data • Determines what to display • Puts on its RAM called Frame Buffer. Frame Buffer • Part of modern GPUs. • Is essentially a bitmap of image to be displayed. • Evolved to also denote colors via lookup tables. • Creates video signals using DACs Display • Displays the video signal recieved
  11. 11. WHY MY TESTS ARE FLAKY? - V 11 The virtual frame buffer: Xvfb Graphics Processing Unit • Processes data • Determines what to display • Puts on its RAM called Frame Buffer. Virtual Frame Buffer • Lives in RAM. • Implements entire framebuffer protocol • End point to GPU Display
  12. 12. THE IMPROVED SETUP FOR VISUAL TESTS 12 Start Xvfb with 1024x768 resolution and 100 PPI Launch application (Browser) Execute your visual tests Report “Fixed PPI and resolution guarantees that the image comparison will be truthful“
  13. 13. A good test suite must be repeatable and should give the same result every single time 13
  14. 14. TAKING IT A STEP FURTHER  Snowflake agents.  repeatability vs flakiness. “Containerize the test suite!” 14
  15. 15. CONTAINERS: DOCKER Docker: OS level virtualization ( no virtual machines )  Think of it as layers of different files, unified together into a single file system view  This OS like abstraction gives you ability to run your tests in a sandbox environment. 15 Image courtesy devopscube.com
  16. 16. USING DOCKER  Create your Docker image with all the dependency packages added to the image  Share this image on a location accessible to your CI/CD system.  Run this image, mount your test files into a folder and execute tests from there. 16 Image courtesy devopscube.com
  17. 17. THE FINAL SETUP 17 Fetch Docker image if not already present Checkout code/ Download Build Start Docker image Run Visual TestsReport To CI Stop Docker image
  18. 18. WHAT DID TEAM ALPHA ACHIEVE? A consistent and headless environment to run builds and all tests. They moved to totally virtualized environment No more flaky visual tests, very high repeatability Full reusability of infrastructure Savings on specialized test tools Early feedbacks even on UI specifications Rave reviews of their product! 18
  19. 19. WHO IS TEAM ALPHA? 19 Image courtesy devopscube.com Team Alpha is not a team of developers or a team of QAs.
  20. 20. THANK YOU