When I joined PaddyPower in October 2012 I was asked to improve quality without affecting throughput. I studied the teams for a couple of months and I came up with this model based on Gojko Adzic’s Specification By Example and a white paper on ATDD from Elisabeth Hendrickson.
One year after, the bugs are a distant memory and cycle time has been almost halved, so I thought sharing the approach might be useful to somebody out there.
Here we go!
Acceptance Test Driven Development
Acceptance Test Driven Development is about people, communication, collaboration and delivering business value.
ATDD is a software development methodology based on enhanced communication among its actors.
ATDD uses high communications processes and tools to help developers write tests in a business language understandable by all actors. Such tests help developers focus on what to code. The tests can be automated and represent the blueprint of the application being developed. The tests live with the code never getting out of date. Any deviation between tests and application is communicated to the team in the form of a failing test. ATDD sits very well with many agile software development approaches including Scrum and Kanban and with XP engineering practices.
The Actors
Activities Artifacts and Goal
Acceptance Test Driven Development can be described using 4 activities (Discuss, Distill, Develop, Demo) 4 artifacts (User Story, Examples, Tests, Working Software) and 1 goal (Business Value). Each activity takes as input an existing artifact and produces as output an evolved version of such artifact until the goal is reached as explained in the Business Value Evolution train picture below:
The starting artifact is a User story, during the Discuss activity such artifact is transformed in Examples, in the Distill activity we transform the Examples into Tests, Tests are transformed in to Working Software during the Develop activity and Working software demonstrates Business Value at the Demo activity.
For this process to be successful it is fundamental that none of the activities is performed in isolation (by a single actor) . As many actors as possible should participate to any of the activities, specific actors’ requirement for specific activities are described later in this document.
One Source Of Truth
The first 3 artifacts describe requirements, the fourth is the representation in software of such requirements and the last is the final business goal. The Tests will represent the one point of truth about what it is turned into software and business value. Once the Demo is completed the User story and the Examples can be disposed and the only source of truth will be in the Tests. This is due to the fact that in the transformation the team will have learned a lot more about the deliverable than what it is described in the user story or the Examples. The tests will be updated during all activities to quickly adapt to changes and new discoveries, it is not necessary to retrofit such changes into User Stories or Examples. Tests stay forever and describe the application.
It is imperative that the Tests are at all time visible to all the actors. This means for example that having them buried into source control is not an option because business people don’t use source control.
Actors on the Train
The picture below expands on the Business Value Evolution Train by showing which Actors participate in which activities.
Discuss
Required Artifact: User Story – We need a business requirement to start from. This doesn’t necessarily need to be in the format of a user story, and it can be expressed in any format. What is needed is a business value to be delivered.
Required Actors: At least 3 members of the Development team should participate, ideally a mix of testers developers and business analysts. Either one between Product Owner or Business Analyst need to participate to this activity.
Format: Meeting with access to a whiteboard
How it works: The Business Analyst has previously developed the user story through his conversations with the Product Owner, he will be able to explain to the other actors the user story’s business value. He will also be able to explain the conditions of satisfaction. Shared understanding of goals will guarantee the real goal is attained and not a consequence of somebody’s assumption
The conditions of satisfactions will be translated into examples, for example if the user story is about giving free delivery for customers that buy 5 books or more the initial examples might be:
5 books free delivery
4 books paid delivery
Questions will be asked and other examples might come out, for example
5 books outside Ireland paid delivery (if the product owner decides to give free delivery only for home users)
4 books and 2 CDs paid delivery
5 books and 1 washing machine free book delivery and paid w/m delivery
and so on…
By the end of the meeting the examples very likely will describe more scenarios than we thought when reading the user story for the first time and by trying to create concrete examples, few questions over the applicability of rules will be raised. If all questions have not been answered and the help of the Product Owner is required, the Business Analyst will get the answers from the Product Owner and have a quick catch up with the other actors to define the last examples. Having the product owner at the Discuss activity can make it more effective as most of the questions will be answered during the activity.
If the questions have unlocked some large uncovered areas and cannot be addressed by the Product Owner, more analysis will be required before we can proceed with the next activity.
Outcome1: Examples – Notice as the examples cover all the aspects of the user story plus those aspects that were not covered in the user story. If we add a 2 liner with the business value to the examples we have an improved version of the user story. The original user story is now out of date and can be expended.
Outcome2: The team have a common understanding of the business value of the user story
Outcome3: The discuss activity might highlight that the user story is too big to be delivered, in this case the activity will produce a list of user stories and the examples for the first one that is taken into development.
Distill
Required Artifact: Examples
Required Actors: Two members of the development team need to participate. At least one Developer needs to participate, ideally the developer that will be designing the code for this item. The second member should possibly be a tester or a Business Analyst, if they are not available, a second developer would do.
Format: Pair programming
How it works: Now that we have the examples written down, we can transform them into tests in a format that works with our test automation framework. There are a variety of test automation frameworks that support defining the tests in advance of the implementation including Jbehave and Cucumber. Tests will be written using the Given When Then format. Tests will cover all the examples that were identified as result of the Discuss activity. Extra tests could be added based on the improved understanding of the business goal.
Outcome1: Tests – The Tests cover all the aspects of the examples plus those aspects that were not covered in examples that were uncovered while writing the tests. The tests will also contain the 2 liner with the business value contained in the examples. Again we have an improved version of the examples.
Outcome2: The tests will be written in English so that every actor is able to understand and give feedback. The examples are now out of date and can be expended. The Tests represent the blueprint (documentation) for what we will eventually deliver. The tests will be highly visible and easily accessible at any time.
Develop
Required Artifact: Tests
Required Actors: Two Developers need to participate.
Format: Pair programming or Single developer writing code + Code Review
How it works: When implementing the code, the developers are following a test-first approach, they execute the tests and watch them fail. They will write the minimum amount of code required to get the acceptance tests Green. Once the acceptance tests are green he will manually verify that everything hangs together and will call another Developer or a Tester to perform Exploratory Testing. Once exploratory testing is completed and any defects fixed the user story is done and working software is ready to be delivered. While coding the developer might identify scenarios that were not identified earlier and add tests for them. Such tests need to be added to the previous set and shared with the rest of the actors. If the new scenarios identified represent a large amount of work a decision might be made that pushes the new uncovered scenarios to a subsequent user story or we could decide to deliver them.
Outcome: Working software + more comprehensive tests
Demo
Required Artifact: Working Software
Required Actors: The Product Owner and 2 members of the development team. Possibly the developer that wrote the software and the other actor that performed the exploratory testing.
Format: Meeting with large monitor
How it works: Before organising a Demo the development team needs to be sure the user story adheres to the definition of done. One very good practice is to create a demo script in which the demo facilitator writes down the steps to follow in order to demonstrate the user story business value to the product owner.
The demo should be an occasion for the development team to be proud of what was delivered.
The product owner will be able to use the Tests to validate all the required functionality has been delivered. At the end of a successful Demo, the product owner will accept the original User Story through the business value demonstrated by running the tests.
Outcome1: Business value
Benefits:
1) Shared understanding
2) Front load issue resolution => reducing defects detected at exploratory testing
3) Tests, for their nature, are specific and unambiguous. Using tests as requirements, developers will create their code based on specific and unambiguous requirements.
4) Developers can avail of the specificity of the requirements and avoid over-engineering
5) The tests are not written in a technical language and can be reviewed/discussed/improved by anybody with some business domain knowledge. Everybody can participate in designing the application by collaborating in defining the tests.
6) The tests are the one source of truth for the application’s behaviours. The acceptance tests live with the code and are executed in CI, they will always be up to date unless the application diverges. The tests are the application documentation. With new functionality come new tests or old tests get adapted to fit the new behaviours.
7) Increased communication builds trust and alliances between the actors
8) Increases transparency; the tests content and results are shared, easily accessible and have high visibility among all actors.
Pleasant Side Effects:
1) Acceptance tests are automated and as such they grow into a comprehensive regression suite
2) Can help Impact analysis. If we want to assess the impact of a change on our application we can simulate that change and verify what pre-existing behaviours it affects by reading the acceptance test results.
3) In the not so uncommon scenario of a re-write, if the original application was written using ATDD it is now possible to recreate the old functionality that needs to be transferred by reusing the Acceptance Tests
Acceptance Tests add no value and can be counter productive when…
1) When we write Acceptance Tests for anything but new deliverables
2) When we write Acceptance Tests to create a regression suite of an existing application
3) When we confuse Acceptance Tests with the tool for writing Acceptance Tests
4) When we write Acceptance Tests that do not give fast feedback
5) When we write Acceptance Tests that are brittle and difficult to maintain
6) When we write Acceptance Tests that contain variables and parameters
7) When we write Acceptance Tests that contain unnecessary detail
8) When we write Acceptance Tests that do not focus on what the change is but on how to exercise such change
9) When we write Acceptance Tests that test the application using the UI when the change is not in the UI
10) When we write Acceptance Tests in isolation and we hide them in source control
11) When we write Acceptance Tests as workflows/scripts
Finally, if you want to learn how to write better acceptance tests have a look at https://mysoftwarequality.wordpress.com/2012/12/14/how-to-transform-bad-acceptance-tests-into-awesome-ones/
mysoftwarequality 
What a great article! Really good format and I really like the pictures/diagrams. I have a couple of questions:
1. The process you’ve identified outlines only 4 artifacts(User Story, Examples, Tests, Working Software) How do you deal with defects found after the story has been closed such as Production defects? Do you reopen the old story, create a new story or is there a 5th artifact(defects/unplanned work)
2. During the “Discuss” and “Distill” stages do you ever find that the “Story” is too big? What happens if the story has lots and lots of examples? Is another outcome of the “Discuss” and “Distill” stages ‘Refined/broken down User Stories’?
3. You mention that “The Business Analyst has previously developed the user story through his conversations with the Product Owner, he will be able to explain to the other actors the user story’s business value.”
As none of the other actors have been involved at this stage(Only BA/PO), how do we ensure the quality at this stage? Such as to ensure the BA has “sized” the story correctly and captured the Product Owners desires.
4. You mention “The product owner will be able to use the Tests to validate all the required functionality has been delivered. At the end of a successful Demo, the product owner will accept the original User Story through the business value demonstrated by running the tests.”
Does the Product Owner run the “Automated” tests or just the manual tests? Or in reality do they not run the tests at all and just go on a “feature tour” of the main flows? Do we need to demo every story to the Product Owner?
5. Where do Performance and Security considerations sit within this process? In addition to “Functional” Acceptance Tests do we define and write “Peformance” Acceptance Tests and also “Security” Accpetance tests? If so, how do we write these tests, does Cucumber or Jbehave support Perormance or Security testing?
Thanks a million for your feedback Toby! Let me try to answer your questions
1. A defect coming from production would firstly be assessed by our product owners and if they consider it worth fixing we create a card and move it through the lanes with higher priority. The same activities we perform for a normal card are performed for a defect. For defects that we find at exploratory testing or UAT have a look at this for more details http://bit.ly/1eAJ8XR
2. You are 100% correct, one of the outcomes of the Discuss activities is finding out whether the story is small enough to be delivered. We try to keep the stories as small as possible to maintain a similar cycle time (for predictability). If the story is too big and too many examples come out of the activity, we use the same activity to break down the story into smaller ones and take the first one into Distill.
3. In general that’s what happens, the BA comes up with a rough story after a conversation with the PO. If there are problems with the size or other attributes of the story, we normally find out at the Discuss stage. Our BA’s have become very good at sizing the stories now and rarely we have to change them, but the discuss stage serves also this purpose.
4. The PO’s don’t run the automated tests, our CI does and the reports are published. The product owner has access to the acceptance tests (Thucydides report) and can do both: validate whether the tests are correct and execute them as part of their UAT. Recently we have optimised the process as we had noticed that some of the testing was duplicated from exploratory to UAT and now our product owners get an improved demo of each user story that allow them to accept the story without having to perform UAT.
5. We treat performance separately and in a different way. The application we are currently working on is a trading platform and performance is extremely important. If the prices pushed by our traders don’t reach the customers quickly, then we lose money. On a production like environment we verify we comply with the slas and when we are happy we are, we create a baseline that we replicate in our build and test pipeline. One of the stages of the build pipeline will run performance tests against this baseline at every check in, so if any change has affected performance, we will know immediately. As per security, I am not an expert, actually I probably should say I know very little about it and the job is done by our security department.
Really appreciate your quick response! Just following on, is your “Performance” Acceptance Criteria documented or written anywhere? Sounds like your “Functional” Acceptance Criteria is written in the form of GIVEN WHEN THEN statements. Do you do the same for Performance e.g.
GIVEN 100 users access the home page
WHEN the page loads it is <2 seconds
AND CPU Usage is <60%
Also do you satisfy your performance requirements before closing the story i.e Is Performance Criteria part of your Done Definition?
Hi Toby, no we don’t document the performance tests like the acceptance tests, thinking about it it could make sense.
Yes performance is a prerequisite for each card to be done.
I think of performance, security, etc. as constraints that the team needs to think about for each story / feature.
Yes Janet, that’s what we do. Normally at discuss we would ask ourselves if a new performance test needs to be created. By having the performance tests run against every check in we reduce the risk of introducing “that code change” that slows everything down.
I definitely need to learn more about security, where would be your starting point for my research?
Very nice article, Augusto.
You mention “conditions of satisfaction” and “rules” in the Discuss phase and say that the “conditions of satisfaction are translated into examples.” I see the examples *illustrating* the rules and use the Discuss phase to deliberately seek examples that discover rules that aren’t yet articulated.
Once the story is delivered you throw away the stories and examples. That’s fine, but the rules are important. They give context to the tests and often turn out to be business-wide. I tend to document them in my Cucumber feature files.
Thanks for your feedback Seb!
I completely agree with you on how the examples illustrate the rules, I should have expressed that concept more clearly, thanks for pointing it out, some rework needed!
What I am not sure I understand is why you would hang on to the examples when you have the acceptance tests that express them in business language.
Examples for me are something like:
4 books – paid delivery
5 books – free delivery
and the tests would be something like:
Given a customer
When he orders 4 books
Then he will pay for delivery
and
Given a customer
When he orders 4 books
Then he will get free delivery
What I am trying to say is that when I have the tests I don’t need the examples any longer.
Thanks again for your feedback!
Rules can be complex. Examples are concrete instances of the rule application. Without the examples the rules can be vague and ambiguous. Without the rules the examples can be insufficient to grasp the overarching concept.
Consider rules for “strong” passwords. The rules might be simply expressed:
“Must be at least 8 characters long”
“Must contain at least one upper case character”
“Must contain at least one non-alpha character”
etc.
Examples might be:
“widget” – invalid
“Paran0ias” – valid
“123Red321” – valid
It’s not easy to deduce the rules from the tests/scenarios/examples. The rules can be captured in scenario titles, but I like collecting them at the top of the feature file where we can see what the scope of the entire feature is.
I get it now Seb, thank you! Yes, in that case I would document the rule in the jbehave/cucumber file.
[…] When something works, share it! – Augusto Evangelisti – https://mysoftwarequality.wordpress.com/2013/11/12/when-something-works-share-it/ […]
[…] Read the full article… […]
Great article – really nicely written…! I really like the way you have explained using pictures…
Thanks for the kind words Amandeep, I hope you find it useful.
[…] [2] [6] [9] When Something Works Share it […]
[…] [2] [6] [9] When Something Works Share it […]
Hi Very interesting article about ATDD. I am just curious to see if you have some sort of a excel template that we can use when we do the Discuss and Distill sessions?