Test automation is extremely crucial in adoption of an agile delivery. However, it can take one for a ride, if the approach is not correct. In this sensational, heart throbbing, experience report, we'll share our story of how we turned around an inefficient, expensive automation style to lean, efficient style. In addition to sharing a real-world example, we'll also share some of the key challenges we faced and how we solved them. If you are convinced about the Testing Pyramid, but are struggling to invert it, then this session is for you. Business Impact: * Earlier Defect Detection - Higher test coverage at Unit/Intermediate layers lead to earlier defect detection. Reduced number of issues found on higher test environments/Production. Reduced cost of defect fixing. * Reduced maintenance cost - UI tests are fragile and costlier to maintain Vs backend tests. No of changes in services layer are comparatively less. Reduced test execution time - Backend tests are much faster. Almost 7-10 times faster than UI Tests - improved build certification time. * Test feedbacks are naturally distributed across layers of application. Test feedbacks are more pin pointed/ granular. Details: http://confengine.com/agile-pune-2014/proposal/471/inverting-test-pyramid-a-first-hand-experience-report Conference: http://pune.agileindia.org/

2. Inverting Test PyramidExperience ReportSachin Natu @natusachinNaresh Jain @nashjain

3. 25YEARS
6000+CLIENTS
Nr 1M ROOMS
94COUNTRIES
95% +
ago we started helping clients manage their revenue
later we’re still finding the revenue opportunity in every situation
around the world are priced every single day using IDeaS
on 6 continents have hotels that now run on IDeaS Revenue Management Systems
More than 95 percent of all our clients are still working with us today.
About IDeaS

4. Our Worldwide Clients : Hospitality + Travel + Distribution

5. Agenda
●Where were we?
●Problem
●How we tried to solve?
●How we optimized?

6. Status of regression time
Page

7. No Automation !

8. Problems
●Thousands of regression tests
●Cross cutting defects found were more.
●Repeat Execution

9. Moving to shorter release cycle -a dream?

10. Regression period is incompressible

11. Manual and repetitive tasks are error prone.

12. Issue leakages
Page
Releases

13. Let’s automate regression
Page

14. UI Test
Page

15. After Automation Introduced

16. Production feedback
Page

17. Duration…
2 Yrs
Team of 2 full time Automation Testers

18. State of the Product
ManualChecking 45%
End-to End UI Tests 40%
Integration Tests
10%
Unit Tests
5%

19. Always Catching Up

20. No pinpointed feedback

21. Silos

22. Tests are slow

23. Costly to maintain

24. Is there way out?
Page

25. Analysis of Bugs Report

26. Why are we testing from far?

27. ManualChecking
End-to End Selenium Tests 70%
Integration Tests
20%
Unit Tests
10%
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests 4%
UI 1%
Typical test strategies lead to an inverted testing pyramid
This is the need of the hour….
Courtesy: Naresh Jain @AgileFAQs.com
The Test Pyramid

28. Our transition story…
•We decided to "Test things right , at right place on right time" which would result in "right test pyramid“
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
E to E 4%
UI 1%

29. Business logic test

31. Current Status

32. Benefits

33. Automation is part of development
•TDD
•BDD
•UI Tests only worry for UI

34. Improved collaboration
Developer
QA

35. UI Tests –Grouping Functionality
80 Scenarios -300 + Min!

36. Correct automation improved time.
80 Scenarios < Min!
Workflow -4 Min
UI Tests -4 Min

37. Regression timelines
Page
Months

38. Feels like Agile now...
Page

39. Challenges
•Legacy code -“Unit test unfriendly”
•Mapping acceptance tests with various slices of pyramid
•Building team competencies
•Pairing/ collaboration is backbone

40. Key learning
•Use UI automation only where really necessary
•It’s team responsibility
•Testability is important criteria.
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests 4%
UI 1%

41. Naresh Jain @nashjain
Sachin Natu @natusachin
Page

42. Some Statistics...