O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

In the beginning there was a model: Using requirements models to drive rigorous test automation

85 visualizações

Publicada em

A live webinar presented by Curiosity Software on October 21st 2020, featuring Jim Hazen.

Watch the on demand webinar recording - https://us02web.zoom.us/rec/play/6AvhjXKlWxyv-bpXIbfNMBP545etsnKqVDg6EBspto0ttfc8mnx5Njnxf2-jWIl9FknKjxMlIJo86ioV.u1bewtXzzwMAcs5i?startTime=1603295712000

Visual modelling is nothing new in software development, from the diagrams today drawn in sprint meetings, to defined methodologies like systems modelling, visual task analysis and UCML. Applied to perennial challenges in testing, modelling can accelerate test creation, reduce test maintenance time, and help define upfront the complex data needed in testing. In their fullest application, models bring stakeholders across the delivery lifecycle into close collaboration: System designers can formulate requirements in visual models, with engineers rapidly developing and testing in parallel from the accurate specifications.

Jim Hazen is a 25-year veteran of automation in testing, who in recent years has developed a keen interest in modelling for testing and development. Jim will discuss the practical value of different modelling approaches live on this webinar, illustrated through real-world experiences and stories. Posing the question of whether any one approach can realise the full potential of modelling, Jim will hand over to Curiosity Director of Technology James Walker. James will provide a practical demonstration of cutting-edge techniques in model-based test and requirements design, discussing how they are helping organisations design, develop and build quality software in short iterations.

Publicada em: Software
  • Seja o primeiro a comentar

In the beginning there was a model: Using requirements models to drive rigorous test automation

  1. 1. © Curiosity Software Ireland Ltd. 2020 In the beginning there was a model Using requirements models to drive rigorous test automation Moderator: Tom Pryce Communication Manager, Curiosity Thomas.Pryce@Curiosity.Software @TomTestsToo Jim Hazen Performance Test Engineer Switchfly James Walker Director of Technology, Curiosity. James.Walker@Curiosity.Software @CuriositySoft
  2. 2. © Curiosity Software Ireland Ltd. 2020 In the beginning there was a model 1. Where is the value of models across design, dev and test? 2. How can we realise these benefits in practice? 3. Demo: Model-based design and test automation 4. Q&A and discussion
  3. 3. © Curiosity Software Ireland Ltd. 2020 Modeling Methods for Automation in Testing Jim Hazen
  4. 4. © Curiosity Software Ireland Ltd. 2020 What is a Model? • Merriam-Webster definitions1 • an example for imitation or emulation • a description or analogy used to help visualize something that cannot be directly observed • a computer simulation based on a system • Basically, a physical representation of a thing or system for emulation and visualization.
  5. 5. © Curiosity Software Ireland Ltd. 2020 A model
  6. 6. © Curiosity Software Ireland Ltd. 2020 Another model
  7. 7. © Curiosity Software Ireland Ltd. 2020 Systems Modeling • Systems modeling is the use of models to conceptualize and construct systems in business and IT development. • A common type of systems modeling is function modeling, with specific techniques such as Functional Flow Diagram’s (FFD). • Another type of systems modeling is architectural modeling which uses the systems architecture to conceptually model the structure and behavior of a system.
  8. 8. © Curiosity Software Ireland Ltd. 2020 A Systems Model
  9. 9. © Curiosity Software Ireland Ltd. 2020 Modeling Techniques • Traditional Modeling Techniques • Unified Modeling Language (UML), Function Flow Diagram, Data Flow Diagram, State Transition diagram, Architectural diagram • After initial specification, not fully collaborative • Modern Modeling Techniques • Story Boarding/Mapping3, Mind Map4, User Community Modeling Language (UCML)5, Visual Task Analysis6, Behavior Based Testing (BBT) Map7 • Can be part of initial specification, collaborative
  10. 10. © Curiosity Software Ireland Ltd. 2020 Modeling Tests & Scenarios • Models are a visual representation of a system • Allow for determining different paths in system • Allow for determining the levels of functionality • Allow for defining the data or different states of data • Allow for identifying the other systems to interact with • Can be used to design tests • Can be used to determine test scenarios • Aid in the development of automation code (types of functionality needed in framework)
  11. 11. © Curiosity Software Ireland Ltd. 2020 Types of Models - Traditional
  12. 12. © Curiosity Software Ireland Ltd. 2020 Modeling – Functional Flow Diagram (FFD)
  13. 13. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the functionality present • Provides information about logic flow of functionality • Provides visual representation of paths through functionality • Provides information about “users” of functionality • Helps with design of automation code functionality needed in framework • Helps with automation code modularity design
  14. 14. © Curiosity Software Ireland Ltd. 2020 Modeling – Data Flow Diagram (DFD)
  15. 15. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the data used/processed by the system under test • Provides information about transition states of data as it moves through the system • Provides visual representation of data relationships as it moves through the system • Provides information about functionality used to process data at various points in system • Helps with design of Test Data
  16. 16. © Curiosity Software Ireland Ltd. 2020 Modeling – State Transition Diagram
  17. 17. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Finite State Machine (FSM) • Provides information about states and transition as “user” moves through the system, and the actions that cause them • Provides visual representation of paths through the system, both primary and alternative ones • Helps with design of Test Scenarios
  18. 18. © Curiosity Software Ireland Ltd. 2020 Thoughts On Traditional Methods Pro’s Con’s Standardized methods and formats Can be rigid in process High level graphical representation Decomposing into more detail is problematic Each method is supported by many tools Cost of tools can be prohibitive Diagrams give information of process Methods are specific to design and not testing, have to interpret into tests Different methods & diagrams give different pieces of information
  19. 19. © Curiosity Software Ireland Ltd. 2020 Types of Models - Modern
  20. 20. © Curiosity Software Ireland Ltd. 2020 Always Used Seldom Used Criticality (UsageLevel) Usage Sequence Log On Customer Management Order Processing Accounting Inventory Management Add New Customer New Order Accounts Receivable Inventory Report Edit Customer Modify Order Add Inventory Delete Customer Delete Order Accounts Payable Modeling – Story Mapping
  21. 21. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the functionality present • Provides information about usage levels of functionality • Provides visual representation of paths through functionality • Helps with design of automation code functionality needed in framework • Helps with automation code modularity design • Helps with prioritization of automation code development
  22. 22. © Curiosity Software Ireland Ltd. 2020 Mail Application Compose Can user send mail Can user send mail with blank subject Can user send mail with blank body Can user send mail to Cc, Bcc list recipients Add Attachment Single Attachment Multiple Attachments Different File Formats File greater than 25 MB Insert HTML Link Insert File Modeling – Mind Map
  23. 23. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the functionality • Provides information about usage of functionality • Provides visual representation of paths of functionality • Provides visual representation of data • More atomic/granular view of functionality to test • Helps with design of test scenarios and combinations • Helps with prioritization of automation code development
  24. 24. © Curiosity Software Ireland Ltd. 2020 Modeling - UCML
  25. 25. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the functionality • Provides information about usage of functionality • Provides information about the usage level of functionality • Provides visual representation of paths of usage • Helps with design of test scenarios and combinations, including End-to-End • Helps with determining workload model for Performance Testing • Helps with defining Non-Functional Requirements
  26. 26. © Curiosity Software Ireland Ltd. 2020 UI JS API APP DB Successful Login Flow Load Page Potential Check/Test Seam / Layer Action Build Login Page Interact with Login Send API Request Process API Request Check Credentials / Create token Send API Response Flag as Logged In Render Home page Home page Modeling – Visual Task Analysis
  27. 27. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about Technology Stack • Provides a visual representation of topology of functionality layers/seams • Helps with determining what to interact with and which tool may be utilized • Helps to decide the types of data needed to drive the tests • Start testing pieces of the flow in isolation
  28. 28. © Curiosity Software Ireland Ltd. 2020 Modeling – Behavior Based Testing
  29. 29. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Provides information about the functionality • Provides information about usage of functionality • Provides information about the data needed for test • Helps with design of test scenario • Helps with defining generic data needed for test • Helps with defining Gherkin’s “Given, When, Then” structure for associated input data, action, and end state condition or validation
  30. 30. © Curiosity Software Ireland Ltd. 2020 Modeling –Business Process Model and Notation (BPMN)
  31. 31. © Curiosity Software Ireland Ltd. 2020 Application to Automation • Information about the functionality • Information about usage of functionality • Information about the data needed for test • Helps w/ design & generation of test scenarios & data • Can be used to generate code for automation in testing • Helps with design of test scenario • Helps with defining generic data needed for test
  32. 32. © Curiosity Software Ireland Ltd. 2020 Thoughts On Modern Methods Pro’s Con’s Multiple methods and formats Picking the “right” one(s) can be problematic High level graphical representation, and can have specific details Can get to be too detailed, lose the forest for the trees. Lots of tools Picking the “right” one can be problematic Diagrams give information of process Different methods & diagrams give different pieces of information Some methods & diagrams are geared towards testing
  33. 33. © Curiosity Software Ireland Ltd. 2020 References 1. “Model”, Miriam-Webster Dictionary, www.merriam-webster.com/dictionary/model 2. “The New Model for Testing”, Paul Gerrard, Agile Testing Days, https://youtu.be/1Ra1192OpqY, 2017. 3. "Story Mapping," Agile Alliance, www.agilealliance.org/glossary/storymap/, 2018. 4. “What is a Mind Map? - Definition & Examples”, Study.com, study.com/academy/lesson/what-is-a-mind-map- definition-examples.html, 2018. 5. “User Community Modeling Language (UCMLTM)," Scott Barber, www.perftestplus.com/articles/ucml.pdf, 2003. 6. “Pyramids Are Ancient - Let’s Talk Automation Strategy”, Richard Bradshaw, SauceCon 2019 presentation, https://youtu.be/83aHKC5C14Q, 2019. 7. “Tired of Your Software Development Testing Automation Failing Over Time?”, Tim Harrison, SQA2, http://sqasquared.com/blog/2019/07/23/tired-software-development-testing-automation-failing-time/, 2019.
  34. 34. © Curiosity Software Ireland Ltd. 2020 How can we realise these benefits in practice? An approach for realising as many model-based benefits as possible
  35. 35. © Curiosity Software Ireland Ltd. 2020 Problems worth solving Design 64% of bugs emerge in the requirements analysis and design phase.1 Requirements defects can account for as much as 82% of total defect remediation efforts.2 A bug found in testing costs 15 times more to fix than one found during design.3 Dev Developers spend 50.1% their time fixing bugs.4 50% of developers encounter the most delays during testing.5 QA/Testing Testing still consumes 23% of the average IT budget.6 Fixing a bug found post- release costs double fixing one found in QA.3 Only 19% of UATs today are automated.6 44% of QA time is spent searching, managing and generating data.7 [1] P Mohan, A Udaya Shankar, K JayaSriDevi (2012), “Quality Flaws: Issues and Challenges in Software Development”, in Computer Engineering and Intelligent Systems (3:12:40-48). [2] James Martin, cited from Bender RBT (2009), Requirements Based Testing Process Overview. [3] IBM, cited in M Dawson et al (2010), “Integrating Software Assurance into the Software Development Life Cycle (SDLC)”, in Journal of Information Systems Technology & Planning, (3:6: 49-43). [4] T Britton et al (2013), "Reversible Debugging Software", report created for the Judge Business School, University of Cambridge. [5] GitLab (2019), 2019 Global Developer Report. [6] Capgemini, Sogeti, Micro Focus (2019), World Quality Report 2019-20. [7] Capgemini, Sogeti, Broadcom (2020), Continuous Testing Report 2020.
  36. 36. © Curiosity Software Ireland Ltd. 2020 Silo Wall Information Hop Delay No feedback or traceability Dev Automation EngineersQA/TestBAs Effort: I write up lots of repetitive user stories, requests and tickets. Quality:Most bugs originate in incomplete, ambiguous requirements. Effort: I recode after misunderstood designs and uncaught bugs. Quality:Bugs are more damaging the later they are discovered. Effort: I slowly create and maintain 1000s of low-coverage tests. Quality:Manually derived tests hit a fraction of system logic. Effort: I maintain a growing mountain of repetitive test scripts. Quality: Data clashes, false positives, false negatives, low coverage. Formulate complex requirements and change requests in disparate, unconnected formats. Convert the incomplete designs into code, trying to fix the impact of changes on complex systems. Unsystematically create high volume, low coverage test cases, mismatched with system designs. Choose between creating new scripts for new logic or maintaining existing code. Flat RQMTs = Silos = effort and errors Mini-waterfalls – manual effort and mistranslation mounts with each “information hop”
  37. 37. © Curiosity Software Ireland Ltd. 2020 Flowchart modelling Recorded tests activity Test Cases BPMN diagrams BDD Scenarios Business Requirements Application Scans Reactive Model of the system Find / Make / Allocate Test Data Run Results Tests New Iteration  Perfect sets of tests  Fast Maintenance  Reusable Components  Move faster  Deliver with confidence Test AutomationTest Automation
  38. 38. © Curiosity Software Ireland Ltd. 2020 Modelling solves automation challenges 1. Modelling automates repetitive test case and data design. 2. Modelling minimises maintenance time – updating models updates all artefacts generated from them. 3. Test optimisation finds bugs earlier and at less cost to fix. 4. Re-usability allows engineers to focus on feeding in new code which is re-used across cross-functional teams. 5. Modelling minimises test data frustrations, creating unique combinations for every test.
  39. 39. © Curiosity Software Ireland Ltd. 2020 Models are a collaborate piece Most bugs stem from requirements. The best way to test these bugs is to catch them as they arise. • Collaboratively modelling provides continuous feedback – each tester, dev, BA etc. spots things the others didn’t. • Modelling eradicates ambiguity as we pick only one option. • Visual models make it easy to spot incompleteness upfront. We then execute tests directly from the Living Documentation, reducing manual effort and driving quality. Behaviour-Driven Scenarios ChatOps and Oral Agreements “Flat” diagrams Requirements Documents Email Requests Spoken Agreements Requirements are often spread over unconnected formats. Modelling consolidates them, spotting incompleteness & ambiguity. This “living documentation” avoids technical debt and keeps teams aligned. ⨯
  40. 40. © Curiosity Software Ireland Ltd. 2020 Models pool knowledge and skills The models then automate the creation test assets and specifications, driving accurate development and in-sprint testing Skills sharing & cross-team collaboration: People who understand the user requirements People who understand testing People who understand automation People who understand data People who understand integration People who understand CI/CD A single source of truth for all assets Test scripts Test Cases Written Requirements & Diagrams Gherkin specifications Test Data Service Definitions & Virtual Services Cross-team alignment & real-time communication • No technical debt • In-sprint automation • Minimal dev rework • Avoid duplicate effect and cumbersome maintenance
  41. 41. © Curiosity Software Ireland Ltd. 2020 Modelling is an accelerator, not replacement, for good test automation With modelling, automation becomes a largely automated comparison of the modelled system to the system built in code. That means accurate regression tests executed in-sprint. Feeding in new code, it becomes re-usable in future sprints and by non- coders. You can focus on testing what’s new and interesting, testing fast- changing systems.
  42. 42. Demo © Curiosity Software Ireland Ltd. 2020
  43. 43. © Curiosity Software Ireland Ltd 2020 Questions and Discussion
  44. 44. © Curiosity Software Ireland Ltd. 2020 Want to learn more from Jim? Check out Before the Code. 15 chapters of expert guidance on what you should do before you write a single line of automation code. Get your (discounted) copy at https://bit.ly/3lKqBiI
  45. 45. © Curiosity Software Ireland Ltd. 2020 The broken promise of test automation Daniel Howard Senior Researcher Bloor Research November 10th 2020 2pm GMT / 9am EST Huw Price Managing Director Curiosity Software SIGN UP TODAY James Walker Director of Technology Curiosity Software Why are we still hand-cranking tests? © Curiosity Software Ireland Ltd. 2020
  46. 46. © Curiosity Software Ireland Ltd. 2020 Carry on the conversation? • We’d love to chat further. Book a web meeting at calendly.com/curiosity-james • Want to try MBT? Visit Testmodeller.io for a free trial. • Follow @CuriositySoft • Email us on James.Walker@Curiosity.Software • Connect with Jim and James on LinkedIn
  47. 47. © Curiosity Software Ireland Ltd 2020 Thank you James.Walker@Curiosity.Software

×