End-to-End test environments, a dead End road
â˘Transferir como PPTX, PDFâ˘
2 gostaramâ˘510 visualizaçþes
With the rise of Distributed Architecture, independent DevOps teams and automated CI/CD the End-to-End test environments needs to be reconsidered. They become flaky, shaky, untrustworthy and hard to maintain. Why are End-to-End test environments a dead End road and what are the alternatives.
Recomendados
Recomendados
Mais conteĂşdo relacionado
Mais procurados
Mais procurados (11)
Semelhante a End-to-End test environments, a dead End road
Semelhante a End-to-End test environments, a dead End road (20)
Ăltimo
Ăltimo (20)
End-to-End test environments, a dead End road
- 1. A dead End road 2/8/2019 Roy Braam @rbraam
- 19. Agenda ⢠Definition ⢠Problem ⢠@ the Rabobank ⢠Why doing it ⢠Alternatives ⢠Questions
- 20. The End-to-End test environment My definition
- 21. âA fully integrated environment with all systems that are part of some flow or functionality, with the single purpose of testingâ
- 22. Aliases ⢠Chain Environment ⢠User Acceptance Environment ⢠Production Acceptance Environment ⢠Broad-stack Environment ⢠NGA Environment ⢠Acceptance Environment ⢠Pre Production Environment ⢠Enterprise-wide Integration Test Environment ⢠Full-stack Environment ⢠Production-like Environment
- 23. Full blown setup of multiple dependent systems that communicate to each other to achieve something
- 24. The dead end road What is the problem?
- 25. Problems with E2E ⢠Rise of DevOps,CI/CD and Microservices ⢠Test data hell
- 26. Test Data hell ⢠Asking for test-data ⢠When consuming data, test- data needs to be prepared for every run ⢠Not 'your' test-data, others may change it ⢠Test-data needs to be correlated ⢠Bye-bye independent DevOps Transaction Timeline User Accounts User Transactions
- 27. Problems with E2E ⢠Rise of DevOps,CI/CD and Microservices ⢠Test data hell ⢠Not so âproduction likeâ
- 28. Not so âproduction likeâ ⢠Simultaneously testing new version ⢠Multiple updates before deploying in production ⢠Not deploying in the End- to-End environment Version: P Version: P Version: P Version: PVersion: P Version: P Version: P+1 Version: P+1 Version: P+1 Version: P-1 Version: P+1 Version: P-2Version: P-3Version: P-4 Version: P+2 E2E-environment
- 29. Problems with E2E ⢠Rise of DevOps,CI/CD and Microservices ⢠Test data hell ⢠Not so âproduction likeâ ⢠Test scope ⢠False positives ⢠Slow Feedback time ⢠Reduce time-to-market ⢠Costs
- 30. @ the Rabobank We love End-2-End environments
- 31. On-premises
- 32. On-premises
- 33. On-premises
- 34. On-premises
- 36. Dev: Hey, can you help me out. Flow x is not working on the environment 3, can you have a look if the platform is the cause Dev: Yes.... Hey, now it's working. O, wait. You are looking at the chain environment 3. Me: Sure. I don't see any problem with the platform. Can you show me the broken flow on my device? Support conversation to a team
- 37. Me: Yes, that is what you said, environment 3... Dev: No, I meant acceptance environment 3 Me: But that's the same, it's environment 3. Dev: No, it is an acceptance environment 3 Me: But that's the same platform environment Dev: But not to us, in acceptance environment 3 we connect to services in environment 5 Me: What !!?!?
- 40. Migrating to the public Cloud End-to-End Environment
- 41. Why? What are the reasons teams rely on End-to-End environments
- 42. Test data
- 43. âTests with semantic correct data can only be run in a End-2-End test environment.â
- 44. Improve by ⢠Pull that data out of the e2e environment ⢠Tests in lower level of the Pyramid ⢠Build for failure
- 45. Trust
- 46. âI want to see it work before I go to productionâ
- 47. Improve by ⢠Move tests down the pyramid ⢠Proof no new issues are found ⢠For every incident, create an automated test case ⢠Invest in tests to gain trust ⢠Proof it will not break in production
- 48. Immaturity
- 49. âSystem X had a new release that broke our system, even though the contract is the same.â
- 50. Improve by ⢠Stricter and clearer contracts ⢠Consumer Driven Contract testing ⢠Automate an 1-on-1 integration system if needed ⢠Resilience
- 51. Compensation
- 52. âWhen they release a new version we first want to test our system, they often break things.â
- 53. ⢠Put the responsibility where it belongs ⢠Resilience ⢠Run your automated expectations against their test environement Improve by
- 54. Requirement
- 55. âOur stakeholders demand that we first show it in an End-to-End environment, with that we guarantee our qualityâ
- 56. ⢠Are stakeholders telling us how to guarantee quality? ⢠Gain their trust ⢠Always try to improve your process Improve by
- 57. Alternatives Ways to minimize usage of E2E environments
- 58. CDC Tests
- 59. CDC Tests ⢠Consumer Driven Contract Tests ⢠Testing pattern ⢠2006 (by Martin Fowler??) ⢠Consumer offers expectations to producer
- 60. CDC Tests ⢠Expectations provided as tests by consumer ⢠Producer: Trust to not break consumers ⢠Consumer: Guarantee to not break contracts ⢠API design ⢠API usage ProducerConsumer Repo
- 61. CDC tools ⢠Spring Cloud Contract (SCC) ⢠Pact ⢠Postman/Newman
- 62. Reveal Unused Interfaces Well-Fittedness Costs Stability Feedback time - - - - - - + - - - + + + + + + + Maintenance Effort +- + Complexity Test Data Setup Testing Data Semantics +- - + + + E2E Tests CDC TestsMock Tests Isolation +- + + https://reflectoring.io/7-reasons-for-consumer-driven-contracts/
- 63. Minimize Risks
- 64. ⢠Separate deployment from release ⢠Canary releases ⢠Blue/Green deployments ⢠Feature toggle ⢠Test groups ⢠Monitor ⢠Focus on mean time to recover
- 67. Sources ⢠Practical test pyramid: https://martinfowler.com/articles/practical- test-pyramid.html ⢠7 reasons for CDC: https://reflectoring.io/7-reasons-for-consumer-driven-contracts/ ⢠Thoughtworks tech radar https://www.thoughtworks.com/radar/techniques/enterprise-wide- integration-test-environments
Notas do Editor
- Audioble in the back? LET ME INTRODUCE TO YOU
- NO
- - Solution Architect for teams responsible for the core backend services in the online platform - 3+ years ago I got involved in this discussion - Lots of struggle within the organisation about this - Became one of my favorite but most frustrating topic - Talking about clean architecture >> clean testing strategies
- [5 MINUTES]
- - Real deployments with test data - Involves both back-end as frontend - Excludes Production system that you take out of service for testing - Not the e2e angular developers no
- - Martin Fowler - NGA > Almost Accaptence environment - Who recognizes names?
- [8 minutes]
- The speed of change makes it hard to maintain
- DevOps Teams - Normal flow Everybody is doing its utterly best to make it not so production like...
- Test scope: Who is responsible Central team is anti patern in devops world False positives that makes you puzzle Not runnable local, so environment needed to seup and maintain
- [15 MINUTES] 400 teams, 250 devops teams
- @ Rabobank we love E2E environments We have it for more then 10 years The size: Orange box alone has around 120 teams. We know it is not working and always broken zbut still we love them love them so much that we not have 1
- In their displeasure with the setup of those 4 environments they created smaller environments
- I have no words for this....... It makes me cry
- Show you how cool this is Me trying to do my best to offers some support Based on a True story, it realy happend.
- I have no words for this....... It makes me cry -
- - not all Teams rely on these environments - I'm in one of those teams, we don't run tests in those environements - But we have to deploy it there
- Moving to the public cloud Online platform, no End-2-End env. No systems that connect to End-2-End envs
- [20 MINUTES] We can group all heard reasons in 5 groups
- I should say 'SEE it work' I want to click through it
- It will be hard and you will fail. But in the long term it will help
- Something in the boundaries of the contract broke it
- The hardest one
- Put the expectations in a separated pipeline and run it every day
- [30 MINUTES]
- Worth a separate talk
- ex: consumer consumes the producer API
- Newman is a command-line collection runner for Postman Handy when you have non coorperative teams that don't have GIT
- Some characteristics of test types *** Isolation: Isolation is good: Easy to execute, 10times, moving down the pir. Complexity: Setup, maintenance etc. Testdata setup: How to setup your testdata Semantics: vs Syntax. Business logics Feedback time: Time to get feedback from tests Fittedness: Consumer driven API Unknown Consumers: Unkown group of consumers Cost Maintenance effort
- Not saying skip integration tests, you should Also not saying it is easy. But what i'm saying is that an fullblown E2E environment is not the correct answer to your problems
- [40 minutes]