Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (19)
Destaque
Destaque (18)
Semelhante a STARCANADA 2013 Keynote: Lightning Strikes the Keynotes
Semelhante a STARCANADA 2013 Keynote: Lightning Strikes the Keynotes (20)
Mais de TechWell
Mais de TechWell (20)
Último
Último (20)
STARCANADA 2013 Keynote: Lightning Strikes the Keynotes
- 1. Lightning Strikes The Keynotes Moderated by Lee Copeland lee@sqe.com
- 2. Featuring Jon Bach Michael Bolton Janet Gregory Keith Klain Paul Holland Gerard Meszaros Fiona Charles Griffin Jones Nate Oster
- 7. Error Message Via Rail, between Montreal and Toronto, 2007
- 8. But I can’t contact my… oh, never mind. Error Message Via Rail, between Montreal and Toronto, 2007
- 10. If you can’t do math, it’s a nickel extra. Hyatt Regency, 2008
- 11. Why you shouldn’t let an unsupervised algorithm choose your sponsored links (1). Vimeo’s Web Page Spring 2010
- 12. Why you shouldn’t let an unsupervised algorithm choose your sponsored links (2). Vimeo’s Web Page Spring 2010
- 13. Why you shouldn’t let an unsupervised algorithm choose your sponsored links (3). Vimeo’s Web Page Spring 2010
- 14. Google Chrome
- 16. OK, fine. Don’t Know Who This Was
- 17. Adobe Acrobat
- 18. Adobe Acrobat
- 20. Intuit Quicken
- 21. Intuit Quicken
- 22. Intuit Quicken
- 23. Here’s another err. Intuit Quicken
- 24. ibm.com
- 25. Windows
- 26. I think I’ll shoot my own trouble for now. Windows
- 27. Windows
- 29. iTunes
- 30. Testing is more than checking.
- 31. Welcome to the new world. “The World of Heineken” Heineken Experience, Amsterdam, 2006
- 32. Paul Holland
- 33. Functional Specification Blinders - My Experiences with Creativity Paul Holland Test Consultant and Teacher at Testing Thoughts
- 34. Test Idea Creation • A “Typical” method of developing test cases or test ideas: 1. 2. 3. 4. 5. 6. Start by reading functional specification Write down test ideas (or create TCs) Try to think of other ways to test product If you think of any, write those down too When product is available, execute test ideas While testing try to think of more test ideas April, 2013 ©2013 Testing Thoughts 34
- 35. What I Do on my Teams • Over the past five years as a test manager at Alcatel-Lucent, I would reverse two steps: 1. 2. 3. 4. 5. 6. Think of all ways you can to test product Write down test ideas Only then you read the functional specification Write any new ideas down When product is available, execute test ideas While testing try to think of more test ideas April, 2013 ©2013 Testing Thoughts 35
- 36. How To Model Without FS • Oh no! I can’t read the functional specification. How can I possibly think of test ideas? 1. If the product is available, then play with it 2. If not, then talk to designers, architects, marketing, customers, project managers, etc. 3. Find out why the new feature/product is being developed and how it will be used 4. Play with competitive/similar products/features 5. Learn as much as you can at a high level April, 2013 ©2013 Testing Thoughts 36
- 37. Justification • For years I had asked my team to think of their own test ideas before reading the functional specification • I now have (statistically invalid) proof that reading the specification first can stifle individual creativity and limit the testing ideas that are generated April, 2013 ©2013 Testing Thoughts 37
- 38. Experience Report • During a modeling exercise of the Rapid Software Testing course November, 2012: – Students were asked to identify all the dimensions of an object that may be relevant to testing it April, 2013 ©2013 Testing Thoughts 38
- 39. Experience Report • Students typically start with intrinsic dimensions: – Height, width, material, volume, etc. • They then progress to relative dimensions: – Durability, dishwasher safe, contents, place of manufacture, where it is now, where it has been, etc. April, 2013 ©2013 Testing Thoughts 39
- 40. Experience Report • Some students mention “conformance with standards” • There is an ISO specification for a specific version of this object – A few have found this ISO specification exists – One student of mine found the spec and read it April, 2013 ©2013 Testing Thoughts 40
- 41. Experience Report • He found the spec BEFORE he had tried to think of his own dimensions • After he read the spec, he could not think of ANY other dimensions • In other exercises, this student showed good levels of creativity • In this exercise, his creativity had been completely stifled by putting on the “specification blinders” April, 2013 ©2013 Testing Thoughts 41
- 42. Summary • Try to think of your own test ideas BEFORE reading the functional specification, previously existing test cases, or any other rigid documentation that may compartmentalize your thoughts • Do not stifle your creativity. The functional specification will be waiting for you when you have done some of your own thinking April, 2013 ©2013 Testing Thoughts 42
- 43. April, 2013 ©2013 Testing Thoughts 43
- 44. Griffin Jones
- 45. WREST Workshop on Regulated Software Testing Software subject to review by an internal or external regulatory body
- 46. Purpose and Format • Share and generate ideas and techniques • Provide a forum for people interested in the topic • Participation is free and open to all • LAWST-style rules of engagement
- 47. Workshop Structure • Facilitated • Series of experiential presentations and group discussions • Atmosphere is collaborative, supportive and constructive • Focus on the practical and useful
- 48. More Information • Next WREST: May 3rd 2013 Hosted by SQE at STAREAST • Contact: Karen N. Johnson, John McConda, Griffin Jones • Website: wrestworkshop.com
- 49. Gerard Meszaros
- 50. Avoiding Unmaintainable Automated Tests Gerard Meszaros StarCanada2013@gerardm.com
- 51. Premise: Automated Tests are a Self-Verifying Behavior Specification Because: They Tell us What the System Should Continue to Do After any changes or bug fixes we apply
- 52. The Challenge: How to Avoid Creating Tests/Checks/Specs that Need Frequent Maintenance ?
- 53. My Belief: Most Automated Tests Have Too Much Detail Which is What Makes Them Require Frequent Maintenance
- 54. Behavior Specification at Right Level High Detail Low Need to specify at the right detail and scope Broad Scope Narrow
- 55. Behavior Specification at Right Level High Detail Low Specify broad scope at minimum detail Specify most detailed req’ts at narrowest scope Incomplete Spec MultiUse Case Workflows Transactions (Use Cases) Too much detail Unmaintainable Broad Scope Rules & Algorithms Narrow
- 56. Example: Configurable Bank TX Notifications Configure Notification Rules Suspend Notification Account Holder Resume Notification Process Transaction Transaction Settlement
- 57. Example: Testing Notifications - 1 Given: User and Accounts When: Notification Rule Configured Then: Then: Notification Rule Notification Rule Configured and Configured and Active Active
- 58. Example: Testing Notifications - 2 Use Case: Process Transactions When: The Transactions to be processed Then: Expected Notifications Medium Detail; Large Scope Too much detail given the scope
- 59. Issues With This Test • Difficult to understand which TX should notify – because cause (rules) and effect (notification) are far apart • Only verifies one simple combination of rules – – We will require many more tests to test all the other combinations Lots of repetition of workflow & data across test cases • Tests will take a long time to run – Due to need to configure first, then run transaction processing
- 60. Changing Level of Abstraction/Detail Too detailed for verifying workflow and Too broadly scoped for checking the rules Need to Reduce Detail or Reduce Scope Incomplete MultiSpec Use Case Workflows Transactions (Use Cases) Re De duce ta il a L High ot Detail Low • • • e u c bi t d R e th a Bo Too Rules & Re much detail dupceea Lot! Algorithms Sc o Unmaintainable Broad Scope Narrow
- 61. Example: Given: User & The overall workflow should be defined at a very high level Thresholds Specifying Notification Workflow When: Transactions Are Processed Broad Scope; Minimum Detail; Irrelevant Details Omitted! Then: Expected Notifications
- 62. Changing Level of Abstraction/Detail Incomplete MultiSpec Use Case Workflows Transactions (Use Cases) Re De duce ta il a L High ot Detail Low • Need the detail to specify notification rules • Therefore, need to reduce scope e u c bi t d R e th a Bo Too Rules & R much detailedupceea Lot! Algorithms Sc o Unmaintainable Broad Scope Narrow
- 63. Example: Business Rule Specs When we ask NotificationRequired? with this transaction: Threshold per Charge Type Given these Process Transaction Configuration rules High Detail; Narrow Scope Easy to Understand Then: The answer should be
- 64. Business Rules Tests Encourage: • Encapsulation of Rules in “Rules Component” – which can Expose a Test API System Under Test Configuration Configure User Interface Threshold Notification Rule Test Transaction Interface Notification Notification RulesRules Fake Notification Process Transaction Notification Required? Do Notification. • Accept Rules via “Data Injection” – To Simplify Test Automation Notification Log • Making Automation a Dev’t Responsibility – To incent design-for-testability
- 65. In Summary: Manage Scope vs. Detail – to Avoid Unmaintainable Tests/Specs Provide the Tests/Specs to Dev’t Before System Built – to allow Dev’t to automate the test – to ensure Design-for-Testability Thank You! Gerard Meszaros StarCanada2013@gerardm.com http://www.xunitpatterns.com http://ScopeVsDetail.gerardm.com Jolt Productivity Award winner - Technical Books http://testingguidance.co deplex.com/
- 66. Our Thanks To Jon Bach Michael Bolton Janet Gregory Keith Klain Paul Holland Gerard Meszaros Fiona Charles Griffin Jones Nate Oster
Notas do Editor
- Good afternoon. My name is Gerard Meszaros and I’m speaking on how to avoid unmaintainable automated tests.
- The challenge, as I see it, is to avoid creating tests or specs that need frequent maintenance.
- My belief is that most automated tests have too much detail and that is what makes them unmaintainable.
- So, How Can we use this to avoid unmaintainable tests? We need to specify each part of the functionality at the right level of detail and scope. Tests with very broad scope (such as the end-to-end process) should be specified with a minimum of detail for each step. Tests that specify a great deal of detail should be kept very narrow in scope. Such as a specific business rule or algorithm. Specifying broad scope in high detail results in tests that are very long and too verbose to understand. This results in a lot of duplicated and unmaintainable detail. Specifying with too little detail for narrow scope results in too many specs that say very little. But this is rarely the problem. The goal is to make our examples and tests each to understand and easy to write. This requires using the right language in each spec.
- Tests with very broad scope (such as the end-to-end process) should be specified with a minimum of detail for each step. Tests that specify a great deal of detail should be kept very narrow in scope. Such as a specific business rule or algorithm. Specifying broad scope in high detail results in tests that are very long and too verbose to understand. This results in a lot of duplicated and unmaintainable detail. Specifying with too little detail for narrow scope results in too many specs that say very little. But this is rarely the problem. The goal is to make our examples and tests each to understand and easy to write. This requires using the right language in each spec.
- Suppose we are specifying how our bank will notify us of transactions on our accounts. The bank provides us with a user interface to configure the notification rules it will use while processing transactions.
- Here’s a first crack at writing a test that specifies how our application should work. First, a user signs in and configures a notification rule on one account. All transactions of any type over $10,000 on their chequing account should result in an e-mail to them. They can verify that the rule was accepted by reviewing the list of accounts and the active notifications. All of this effort just to set up the Given for the test conditions. Next we need to create some transactions against these accounts.
- So we process a bunch of transactions in various amounts above and below the threshold for various accounts and transaction types. These constitute the When’s of the test conditions. The final table lists the expected notifications; the Then’s of the test conditions.
- It’s getting a bit difficult to follow along because the cause & effect are not close to each other. And this test only verifies one particular combination of notification rule. We’ll need to create other test cases for each of various combinations of rules we could have. That will result in a lot of repetition across the test cases. And these test cases will take a long time to run if we hook them up to the user interface of our application.
- These tests are too detailed for verifying the overall workflow and too broadly scoped for checking the notification rules. We need to either reduce the level of detail in the workflow test or Reduce the scope to just the business rules if we want to keep the details. So let’s look at how we can reduce the detail.
- The overall workflow should be defined at a very high level. Only the essential details should be specified here because we’ll need to touch on many of the transactions and providing too much detail would make these specs too hard to understand. So we start by identifying our user and the thresholds in effect. The transactions we process become the when’s of our test. And the list of notifications we expect specify the Then’s. This test is still broad in scope (several use cases) but with only the bare minimum details required to explain the functionality.
- Now that we understand the overall workflow, we can move on to specifying the details of the notification rules. This requires a lot of detail so we must reduce the scope to just the NotificationRequired component.
- First we must specify the notification thresholds we want to use (on the left side of the slide.) Then we want to invoke the NotificationRequired component once for each row in the table on the right. The first 3 columns specify the inputs to the notification decision and the last column is the expected result. This format makes it easy to read the test conditions directly from these tests. (Given these thresholds, when we ask IsNotificationRequired? With Account 100372 with Travel charge for 999.99, the answer should be No.)
- A very desirable side-effect of specifying our requirements this way is that it makes our testability requirements explicit: These tests encourage us to encapsulate the notification rules in a Rules Component which can be accessed directly by the test via an API. This allows our tests to focus on what they want the NotificationReq’d component to do and not how to interact with it via the Process Transaction components. It also encourages us to structure the code so that it is easy to “inject” a set of rules directly into the component thereby avoiding use of the rules configuration logic. This simplifies test automation significantly. We can further incent dev’t to structure the code this way by making the automation of these tests/specs part of their job.
- So in Summary, my key points are: It’s important to: * manage the scope and level of detail of tests to avoid unmaintainable tests. * Provide the tests to the developers before they build the system to allow them to automate the test which then ensures the system is designed for testability.