2. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
3.
4. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
5. How do we do it?
Deriving Scope from goals
• Who sets the scope of the work?
o Traditionally the product/business owner
6. How do we do it?
Deriving Scope from goals
• Who sets the scope of the work?
o Traditionally the product/business owner
• Is there a better way?
o Let the team develop scope from goals
7. Why do we need to derive
scope from goals like this?
• Sometimes the real goal isn’t what you thought, or
there is a better solution
8. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
9. Specifying Collaboratively
Why do we want to do this?
• Leverage expertise
“Specification by Example won’t work if we write documents in insolation, even if
we implement all the other patterns described in this book.” – Gojko Adzic
10. Specifying Collaboratively
Why do we want to do this?
• Leverage expertise
• Promote communication and involvement
“Specification by Example won’t work if we write documents in insolation, even if
we implement all the other patterns described in this book.” – Gojko Adzic
11. Specifying Collaboratively
Why do we want to do this?
• Leverage expertise
• Promote communication and involvement
• Shared understanding
“Specification by Example won’t work if we write documents in insolation, even if
we implement all the other patterns described in this book.” – Gojko Adzic
12. Specifying Collaboratively
Why do we want to do this?
• Leverage expertise
• Promote communication and involvement
• Shared understanding
• Breaking down walled gardens
“Specification by Example won’t work if we write documents in insolation, even if
we implement all the other patterns described in this book.” – Gojko Adzic
13. Feature: Free Delivery
Scenario Outline:
Free delivery is offered to VIP customers once they
purchase a certain number of books. Free delivery
is not offered to regular customers.
Given that the number of books to get free delivery is 5
And I am a VIP customer
When I checkout with 5 books in my shopping cart
Then the delivery methods listed should be Free and Standard
14. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
17. Feature: Free Delivery
Scenario Outline:
Free delivery is offered to VIP customers once they purchase a
certain number of books. Free delivery is not offered to regular
customers.
Given that the number of books to get free delivery is 5
And I am a <customer> customer
When I checkout with <contents> in my shopping cart
Then the delivery methods listed should be <delivery>
Scenarios:
| customer | contents | delivery |
| VIP | 5 books | Free, Standard |
| VIP | 4 books | Standard |
| Regular | 10 books | Standard |
19. Feature: Free Delivery
Scenario Outline:
Free delivery is offered to VIP customers once they purchase a certain
number of books. Free delivery is not offered to regular customers or
customers buying anything other than books
Given that the number of books to get free delivery is 5
And I am a <customer> customer
When I checkout with <contents> in my shopping cart
Then the delivery methods listed should be <delivery>
Scenarios:
| customer | contents | delivery |
| VIP | 5 books | Free, Standard |
| VIP | 4 books | Standard |
| Regular | 10 books | Standard |
| VIP | 5 Washing Machines | Standard |
| VIP | 5 books, 1 Washing Machine | Standard |
| VIP | 6 books | Free, Standard |
| Regular | 100 books | Standard |
| Regular | 15 Washing Machines | Standard |
| VIP | 4 books, 1 Washing Machine | Standard |
20. Illustrating Using
Examples
• Avoid ambiguities
• Examples should cover complete scope
o - Experiment with data
o - Come up with alternate ways to test it
• Easy to understand
21. Illustrating Using
Examples
• Avoid ambiguities
• Examples should cover complete scope
o - Experiment with data
o - Come up with alternate ways to test it
• Easy to understand
o - avoid specifying every combination possible
22. Feature: VIP customers get free delivery
Free delivery is offered to VIP customers once they purchase a certain
number of books. Free delivery is not offered to regular customers or VIP
customers buying anything other than books
Scenario Outline: Free Delivery
Given that the number of books to get free delivery is five
And I am a <Customer> customer
When I checkout with <Contents> in my shopping cart
Then the delivery methods listed should be <Delivery>
Scenarios: VIP customers
| Customer | Contents | Delivery |
| VIP | 5 books | Free, Standard |
| VIP | 4 books | Standard |
| VIP | 5 Washing Machines | Standard |
| VIP | 5 Books, 1 Washing Machine | Standard |
| Regular | 10 books | Standard |
23. Illustrating Using
Examples
• Avoid ambiguities
• Examples should cover complete scope
o - Experiment with data
o - Come up with alternate ways to test it
• Easy to understand
o - avoid specifying every combination possible
o - implied concepts
24. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
27. Refining the Specification
• Polish the specification for maximum clarity
o - lack of ambiguity
o - maintainability
28. Refining the Specification
• Polish the specification for maximum clarity
o - lack of ambiguity
o - maintainability
• What to focus on
o - examples should be precise and testable
29. Refining the Specification
• Polish the specification for maximum clarity
o - - lack of ambiguity
o - maintainability
• What to focus on
o - examples should be precise and testable
o - business functionality, not software design
30. Refining the Specification
• Polish the specification for maximum clarity
o - lack of ambiguity
o - maintainability
• What to focus on
o - examples should be precise and testable
o - business functionality, not software design
o - use descriptive titles and explanations
31. Feature: VIP customers get free delivery
Free delivery is offered to VIP customers once they purchase a certain
number of books. Free delivery is not offered to regular customers or VIP
customers buying anything other than books
Scenario Outline: Free Delivery
Given that the number of books to get free delivery is five
And I am a <Customer> customer
When I checkout with <Contents> in my shopping cart
Then the delivery methods listed should be <Delivery>
Scenarios: VIP customers
| Customer | Contents | Delivery |
| VIP | 5 books | Free, Standard |
| VIP | 4 books | Standard |
| VIP | 5 Washing Machines | Standard |
| VIP | 5 Books, 1 Washing Machine | Standard |
Scenarios: Regular customers
| Customer | Contents | Delivery |
| Regular | 10 books | Standard |
32. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
37. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
41. Validate Frequently
• Promotes reliability and stability
• Provides rapid feedback
• Allows the generation of test metrics
• Isolate external systems for reliability and speed
42. Validate Frequently
• Promotes reliability and stability
• Provides rapid feedback
• Allows the generation of test metrics
• Isolate external systems for reliability and speed
• Separate tests into units for targeted validation
43. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
46. Evolving a
Documentation System
• Tests must remain easy to understand
• Avoid complexity and redundancy in your tests
• Use a consistent domain language
47. Evolving a
Documentation System
• Tests must remain easy to understand
• Avoid complexity and redundancy in your tests
• Use a consistent domain language
• Make sure documentation is easily accessible
48. Specification By Example
• Derive scope from goals
• Specify collaboratively
• Illustrate using examples
• Refine the specification
• Automate validation
• Validate frequently
• Evolve a documentation system
Alex Freidah
Patch/AOL Data Warehouse Operations
Alex.Freidah@patch.com
50. Image Credits
• http://funpicc.blogspot.com/2010/12/im-huge-metal-fan-me-too.html
• http://specificationbyexample.com/
• http://johnmartel.blogspot.com/2010/07/how-web-design-team-disconnect-during.html
• http://en.wikipedia.org/wiki/File:F-16_June_2008.jpg
• http://www.elisabethhubert.com/wp-content/uploads/2012/02/Experiment.gif
• http://corkboardconnections.blogspot.com/2012/02/that-was-easy.html
• http://www.busytrade.com/buying-leads/1843437/Buying-Loose-Raw-And-Uncut-Diamonds.html
• http://www.ecrater.com/p/7389126/50-carats-natural-uncut-rough-raw
• http://www.onedesk.com/2013/01/integrating-customer-feedback-into-product-
development/feedback-2/
• http://www.digitallinguistics.com/
• http://uxmag.com/articles/the-dirtiest-word-in-ux-complexity
• All images were found using google image searches – these are links to where each image was
linked from by google.
Notas do Editor
My name is Alex Freidah. I do application monitoring for Patch within the AOL Data Warehouse. I’d like to talk about Specification by Example and it’s approach to delivering software.
Specification by example is a set of collaborative process patterns for the team to follow beginning at the initial setting of goals and goes all the way through the lifecycle of the software that strives to reduce feedback loops, reduce rework, increase product quality and build a more flexible and versatile team through collaboration, universal understanding across all the different parts of the team, and solid documentation.
Traditionally it is expected that the product owner or business user is responsible for setting scope and the software development process doesn’t begin until after that point. The product owner or business user is expected to essentially design a solution and this is where the problem begins and how the wrong product ends up getting built. By having customers supply a list of user stories or use cases, the customers are essentially being asked to design the solution. That is a waste of resources and knowledge that could have benefited the project.
A new approach is to have the product-owner/busines-user set business goals and then to collaborate to set the correct scope that will accomplish those goals. This means that from the outset the expertise of everybody is taken into consideration, and universal understanding is achieved that will allow for a cheaper/faster/easier to deliver productThis means that the business side can concentrate on the intent of the desired feature and what value they expect to get from that. This builds universal understanding of not just WHAT, but WHY which will help the team suggest the solution that is cheaper/faster/more maintainable than one come up with by just a business user/customer.
The F-16 is a good example of this. The initial requirement for the aircraft contract said that the plane must be capable of speeds of Mach 2-2.5. When the Air Force was asked why that was a requirement (instead of just settling for it), the lead-designer was told that the plane had ‘to be able to escape from combat’. The design of the F-16 never went past Mach 2, and yet it won the bid because it implemented other unique and new features that allowed the plane to escape combat based on superior agility and access to combat information. This made it superior to other designs while also cheaper to build and became a great commercial success. This is a perfect example of getting at the real goals instead of forcing the design to be dictated by the wrong party – the overall goal was met more effectively by letting the domain experts apply their knowledge to the high-level goal instead of being forced into one specific implementation
Tech experts know infrastructure and how to apply technologies. Testers know where to look for potential issues, business owners/users have key examples. The important information is spread across groups – All of their knowledge is going to come into play at some point in the process so it is foolish to intentionally keep groups separated that can each bring something important to the table if they were allowed to share and collaborate.
Effective and efficient communication is far more engaging than long boring documents which may never get read. We want to collaborate on shared documents which leads to universal understanding, clearer documentation, and no more “throwing it over the wall”. The different groups should be working together and not against each other. Working this way means that each group has a far better understanding of what the other groups are doing, what their work involves, what problems they face, etc. Developers will be more engaged in testing, business users will understand the tests better, testers are more involved in analysis, etc.
As previously mentioned, one of the primary benefits of the collaboration is shared understanding. When you actually get together and talk out ideas it is far less likely that there is going to be a disconnect between groups and their view of what something means. Not only that, but we are tapping into the collective brain of the group which can lead to flushing out intricate details that would not have even been thought of by any one person or group in isolation. One of the most important outcomes of this is the ‘cross-functional team’ that is far more versatile than the traditional team model as different groups can help to alleviate bottlenecks and keep things moving
we are breaking down walled-gardens. We are trying to get away from a model where different groups ‘own’ different aspects of the information. It should all be shared from inception and throughout the lifecycle of the software
A good example will help us avoid ambiguities – there must be no room for misunderstanding.
There should be enough examples to describe the entire scope of a feature. Expected behavior in primary business cases and simple examples is a good place to start, but they do not represent the full scope of what needs to be implemented. We will likely have to extend the initial set of examples to provide a full picture of functionality.
Take the initial set of examples that you feel is fairly complete, look at the data and try to come up with valid combinations of inputs that could violate rules and break the system – by revealing what you may have missed, you have made the specification more complete and stronger. One example would be that if your example includes numerical values, try to use large and small numbers around boundary conditions, try 0 or negative numbers as they are often sources of broken functionality. This might generate what seems like too many examples with unimportant differences, but we will clean that up in the next step – refining the specification
A common mistake is to illustrate examples using complex and/or convoluted examples. One of the biggest advantages of Specification by Example is that the precision of it makes it easier to spot functional gaps and inconsistencies. To do this, an understanding of the entire set of examples for a given feature is required. If the example is not easy to understand then it becomes impossible to evaluate it’s completeness and consistency.
A common mistake when implementing Specification by Example is to insist on covering every possible combination or arguments. Granted, there IS reason to come up with as many cases as possible that cover a certain data scenario, but going through examples that illustrate existing cases is pointless. Examples should move the discussion forward and improve understanding, so having multiple examples that illustrate the same exact thing is redundant and wasteful. However, much care and discussion should be used before discarding any examples that are intended to illustrate edge cases – for obvious reasons. If a new example is suggested that others view as redundant then either that person doesn’t have a full understanding of the feature, or they have truly discovered an edge case that the others missed – either way it warrants discussion before adding or discarding the example
As with software itself, if the examples start seeming too complex and have too many examples, it is likely time to refactor and express the examples at a higher level of abstraction. Identify concepts that are hidden in the examples and separate them out and make them explicit examples of their own. Separating the concepts out will help facilitate a more meaningful discussion to flush out confusion and inconsistencies. This leads to an easier to understand specification which in turn leads to better software design since the specification is going to dictate the design.
He compares an unrefined specification to a raw diamond. It is valuable, but the value once refined (polished and cut) is far more. To get the most out of our examples, we need to refine them, polish them to show the key points clearly, and to create specifications that the team can use both now and in the future. A common point of failure for teams implementing Specification by Example is not taking the time to process and evolve these examples. Discussion about the specification often leads to experimentation and new insights that lead us to restructure the examples to look at them from a higher level of abstraction. This experimentation is not always successful and some of it might be thrown out, but by doing this, the chances of a specification being too unrefined to work are much lower. If nothing else you have only furthered your understanding of the domain of the example.
A proper specification with examples should unambiguously define the required functionality from a business perspective without trying to define HOW the system is supposed to implement that. This gives developers the freedom to find the best possible solution to meet those requirements.
Remember that these specifications should be able to serve as long-term documentation of the system. This means that it must be written such that others can pick it up months or even years later and easily understand what it does, why it does it, and what it is trying to describe. It should be focused, self-explanatory, and in domain language. This leads to a far more easily maintainable specification over the lifetime of the system
A specification is an objective measure of success. It must be able to unambiguously tell us whether the feature has been properly implemented or not. It must have verifiable information that can be checked against the system.
This idea has been a running theme throughout the book. Specifications should not be spelling out HOW a feature is implemented. Developers are trained in how to use the right techniques and tools to achieve a goal and should be allowed to apply this expertise, not have it spelled out to them. Almost more important than that is the idea of flexibility again. If your spec is not tied to a specific implementation, then if in the future the implementation has to change for whatever reason, your test is still valid. This is the idea of striving for loose coupling between your tests and the implementation of the actual feature.
It is crucial to have a descriptive title to the specification that summarizes the intent of the feature. To further clarify, you should have a brief description in a couple of sentences that explains the goal of the specification and it’s context. Try writing just the example first and then explain it to somebody else – this explanation is what you want to capture. Another idea is to show somebody else your specification and ask them to explain what they think it is without you telling them anything – this will help flush out unclear specs.
The argument against automation of validation is that it increases the up-front cost and time of development. Now we have source code to maintain as well as specification documents, as well as the code that validates those specifications. However, the argument for it is that an initial slight jump in cost is still far less than the cost of having to constantly validate these specifications manually for the lifetime of the project. Without hooking the specifications into an automation tool, we are missing out on one of the most important aspects of Specification by Example. With automation, you can check more cases, more regularly, and faster than previously possible. That is what we are going for – an extremely low-cost of doing functional regression testing that is fully integrated into the development and deployment process, making it impossible to check in (let alone deploy) code that breaks the requirements.
By planning on automation up front, the team will design the software with automation in mind. What you don’t want is to go into it thinking you don’t need automation or you can do it later, only to find out that the software is not properly designed to be tested easily, making it extremely difficult to add in the tests down the road. You might be able to get through your stories quicker without automation but the long term costs are far higher and you are missing the point. An example from the book that is quite apt is ‘A bus can go a lot faster if it doesn’t have to stop to pick up passengers, but it isn’t really doing it’s job then.’
Your automation code is NOT less important than your application code. It should be taken just as seriously, be handled by responsible workers – since your tests are going to dictate your production code design it is equally important and deserves the same level of attention and thought as any other code in the project, and be checked into version control right alongside your production code.
A big problem when dealing with data-driven apps is the maintenance of transactional data. Each test is going to manipulate that data, so you are now chasing a moving target and it becomes quite cumbersome. You usually can’t pre-load all the required transaction data for this reason. A suggestion from the book that multiple teams had implemented was to only pre-populate reference data that is not subject to change, and to have each test handle the creating and cleaning of transactional data. *** go into live ruby example here ***
Frequent validation make sure that the project doesn’t get off track and that we are maintaining the correct production environment
By frequently validating we become aware of problems immediately and can mitigate their damage. Otherwise a problem might permeate through the project, affecting other areas and making the eventual cost of repair much higher due to a larger amount of misguided code.
we can not only get rapid feedback on a new feature, but we will have the ability to track test history and gain metrics on the overall stability of the project. This is especially important when working with legacy systems where it might not be possible to make all of the changes up-front and decisions will have to be made on what parts to rework first. Too many major changes at once will destabilize the system and is not advisable. However, being able to track what parts of the system are still unstable is a valuable metric.
For a system that participates in a larger workflow with lots of external systems being involved, it is a good idea to selectively isolate these external services. This can mean using fake data sources to simulate an external system – this will make the tests run faster and not break if that external resource goes down for whatever reason.
Especially on legacy systems and projects that have been worked on for months to years, the test pack might get fairly large and start taking a long time to run. This defeats the purpose of rapid feedback. A good compromise here is to break the tests into different ‘units’ so that we can run the quick tests more frequently and run the long running tests at the most convenient moment…obviously this still needs to be done regularly, but you don’t want to have to wait an hour or more to confirm every change you commit because that will lead to less commits, with each commit being much larger – we don’t want that, the whole point is frequent commits that can be verified rapidly, larger commits are more likely to break things and then it will be harder to debug and determine the true source of the problem because you have introduced a lot of new functionality all at once. If there are long running tests that need to be run overnight, they shouldn’t be relegated to night runs until they have shown good stability and are unlikely to fail. Another approach is to break the tests into chunks based on the goals of the current iteration.
One of the most important points we have talked about is that these tests should be easy to understand, not just for technical people, but for everybody involved in the project. They should be self-explanatory, clear, and make use of the domain language.
As your system grows, watch out for specifications that get too long. This reduces clarity and is a sign that something isn’t quite right. It is likely that the concept isn’t explained at the appropriate level of abstraction. It might need to be broken into several separate specifications that are more clear. There might be too much contextual information that needs to be refocused on the important attributes that illustrate the goal of the test. On the flip side, you might find that concepts that seemed unique at the beginning have ended up becoming two sides of the same coin. If you have multiple tests that are describing the same feature then they might need to be merged together.
Specs described in a technical language are not a useful communication tool. If your test is starting to look like a technical acceptance test then you need to rethink it. It might be an appropriate concept for a unit test, but it has no place in your functional acceptance tests that should be accessible to business users, customers, and technical staff. Language should be consistent across tests. Code may come and go, but these tests are your documentation for the system and they should have some level of continuity and consistency. A kind of specification language will begin to evolve and a reusable pattern for specifications will show itself. Collaboration on the language used is just as important as collaboration on the scope of the goals themselves. They are all pieces of the same end goal.
The number of tests will grow as the project progresses and can become quite large. Users have to be able to find a description of a given function easily and quickly. It is a good idea to group tests into hierarchies of related functionality. This means that not only is it quicker to find the useful docs for a feature without having to filter through hundreds or thousands of unrelated documents, but it also allows quick and easy execution of just the relevant tests when necessary.
Add examples of tools for different languages and links for referenceLink to online resources for spec by example including book (safari link perhaps?)
