Executable specifications – what would you do?

Can you help?

Executable specifications are used in BDD/Spec By Example as a mean to describe an application behaviour in business language. They are executable because they are linked to tests that exercise the behaviour of the application itself.

Let’s imagine we want to develop the solution to the FizzBuzz problem, Which one of the following solutions would you adopt and why? If you have a different better solution than the two I list, please add it!

1. Specific examples:

Given we are printing numbers using FizzBuzz
When the number is <number>
Then we will print <answer>

|      1      |     1    |
|      3      | Fizz    |
|      5      | Buzz  |
|      6      |  Fizz   |
|      10    | Buzz   |
|      15    |FizzBuzz|

2. Generic Rules

Given we are printing numbers using FizzBuzz
When the number is a multiple of 3
Then we will print Fizz

Given we are printing numbers using FizzBuzz
When the number is a multiple of 5
Then we will print Buzz

Given we are printing numbers using FizzBuzz
When the number is a multiple of 3 and 5
Then we will print FizzBuzz

Given we are printing numbers using FizzBuzz
When the number is a not a multiple of either 3 or 5
Then we will print the number


When something works, share it!

When I joined PaddyPower in October 2012 I was asked to improve quality without affecting throughput.  I studied the teams for a couple of months and I came up with this model based on Gojko Adzic’s Specification By Example and a white paper on ATDD from Elisabeth Hendrickson.

One year after, the bugs are a distant memory and cycle time has been almost halved, so I thought sharing the approach might be useful to somebody out there.

Here we go!

Acceptance Test Driven Development

Acceptance Test Driven Development is about people, communication, collaboration and delivering business value.

ATDD is a software development methodology based on enhanced communication among its actors.

ATDD uses high communications processes and tools to help developers write tests in a business language understandable by all actors. Such tests help developers focus on what to code. The tests can be automated and represent the blueprint of the application being developed. The tests live with the code never getting out of date. Any deviation between tests and application is communicated to the team in the form of a failing test. ATDD sits very well with many agile software development approaches including Scrum and Kanban and with XP engineering practices.

The Actors

ATDD Actors
ATDD Actors

Activities Artifacts and Goal

Acceptance Test Driven Development can be described using 4 activities (Discuss, Distill, Develop, Demo) 4 artifacts (User Story, Examples, Tests, Working Software) and 1 goal (Business Value). Each activity takes as input an existing artifact and produces as output an evolved version of such artifact until the goal is reached as explained in the Business Value Evolution train picture below:

Business value Evolution Train
Business value Evolution Train

The starting artifact is a User story, during the Discuss activity such artifact is transformed in Examples, in the Distill activity we transform the Examples into Tests, Tests are transformed in to Working Software during the Develop activity and Working software demonstrates Business Value at the Demo activity.

For this process to be successful it is fundamental that none of the activities is performed in isolation (by a single actor) . As many actors as possible should participate to any of the activities, specific actors’ requirement for specific activities are described later in this document.

One Source Of Truth

The first 3 artifacts describe requirements, the fourth is the representation in software of such requirements and the last is the final business goal. The Tests will represent the one point of truth about what it is turned into software and business value. Once the Demo is completed the User story and the Examples can be disposed and the only source of truth will be in the Tests. This is due to the fact that in the transformation the team will have learned a lot more about the deliverable than what it is described in the user story or the Examples. The tests will be updated during all activities to quickly adapt to changes and new discoveries, it is not necessary to retrofit such changes into User Stories or Examples. Tests stay forever and describe the application.

It is imperative that the Tests are at all time visible to all the actors. This means for example that having them buried into source control is not an option because business people don’t use source control.

Actors on the Train

The picture below expands on the Business Value Evolution Train by showing which Actors participate in which activities.

How it Works
How it Works


Required Artifact: User Story – We need a business requirement to start from. This doesn’t necessarily need to be in the format of a user story, and it can be expressed in any format. What is needed is a business value to be delivered.

Required Actors: At least 3 members of the Development team should participate, ideally a  mix of testers developers and business analysts. Either one between Product Owner or Business Analyst need to participate to this activity.

Format: Meeting with access to a whiteboard

How it works: The Business Analyst has previously developed the user story through his conversations with the Product Owner, he will be able to explain to the other actors the user story’s business value. He will also be able to explain the conditions of satisfaction. Shared understanding of goals will guarantee the real goal is attained and not a consequence of somebody’s assumption

The conditions of satisfactions will be translated into examples, for example if the user story is about giving free delivery for customers that buy 5 books or more the initial examples might be:

5 books free delivery
4 books paid delivery

Questions will be asked and other examples might come out, for example

5 books outside Ireland paid delivery (if the product owner decides to give free delivery only for home users)
4 books and 2 CDs paid delivery
5 books and 1 washing machine free book delivery and paid w/m delivery

and so on…

By the end of the meeting the examples very likely will describe more scenarios than we thought when reading the user story for the first time and by trying to create concrete examples, few questions over the applicability of rules will be raised. If all questions have not been answered and the help of the Product Owner is required, the Business Analyst will get the answers from the Product Owner and have a quick catch up with the other actors to define the last examples. Having the product owner at the Discuss activity can make it more effective as most of the questions will be answered during the activity.

If the questions have unlocked some large uncovered areas and cannot be addressed by the Product Owner, more analysis will be required before we can proceed with the next activity.

Outcome1: Examples – Notice as the examples cover all the aspects of the user story plus those aspects that were not covered in the user story. If we add a 2 liner with the business value to the examples we have an improved version of the user story. The original user story is now out of date and can be expended.

Outcome2: The team have a common understanding of the business value of the user story

Outcome3: The discuss activity might highlight that the user story is too big to be delivered, in this case the activity will produce a list of user stories and the examples for the first one that is taken into development.


Required Artifact: Examples

Required Actors: Two members of the development team need to participate. At least one Developer needs to participate, ideally the developer that will be designing the code for this item. The second member should possibly be a tester or a Business Analyst, if they are not available, a second developer would do.

Format: Pair programming

How it works: Now that we have the examples written down, we can transform them into tests in a format that works with our test automation framework. There are a variety of test automation frameworks that support defining the tests in advance of the implementation including Jbehave and Cucumber. Tests will be written using the Given When Then format. Tests will cover all the examples that were identified as result of the Discuss activity. Extra tests could be added based on the improved understanding of the business goal.

Outcome1: Tests – The Tests cover all the aspects of the examples plus those aspects that were not covered in examples that were uncovered while writing the tests. The tests will also contain the 2 liner with the business value contained in the examples. Again we have an improved version of the examples.

Outcome2: The tests will be written in English so that every actor is able to understand and give feedback. The examples are now out of date and can be expended. The Tests represent the blueprint (documentation) for what we will eventually deliver. The tests will be highly visible and easily accessible at any time.


Required Artifact: Tests

Required Actors: Two Developers need to participate.

Format: Pair programming or Single developer writing code + Code Review

How it works: When implementing the code, the developers are following a test-first approach, they execute the tests and watch them fail. They will write the minimum amount of code required to get the acceptance tests Green. Once the acceptance tests are green he will manually verify that everything hangs together and will call another Developer or a Tester to perform Exploratory Testing. Once exploratory testing is completed and any defects fixed the user story is done and working software is ready to be delivered. While coding the developer might identify scenarios that were not identified earlier and add tests for them. Such tests need to be added to the previous set and shared with the rest of the actors. If the new scenarios identified represent a large amount of work a decision might be made that pushes the new uncovered scenarios to a subsequent user story or we could decide to deliver them.

Outcome: Working software + more comprehensive tests


Required Artifact: Working Software

Required Actors: The Product Owner and 2 members of the development team. Possibly the developer that wrote the software and the other actor that performed the exploratory testing.

Format: Meeting with large monitor

How it works: Before organising a Demo the development team needs to be sure the user story adheres to the definition of done. One very good practice is to create a demo script in which the demo facilitator writes down the steps to follow in order to demonstrate the user story business value to the product owner.
The demo should be an occasion for the development team to be proud of what was delivered.

The product owner will be able to use the Tests to validate all the required functionality has been delivered. At the end of a successful Demo, the product owner will accept the original User Story through the business value demonstrated by running the tests.

Outcome1: Business value


1) Shared understanding
2) Front load issue resolution => reducing defects detected at exploratory testing
3) Tests, for their nature, are specific and unambiguous. Using tests as requirements, developers will create their code based on specific and unambiguous requirements.
4) Developers can avail of the specificity of the requirements and avoid over-engineering
5) The tests are not written in a technical language and can be reviewed/discussed/improved by anybody with some business domain knowledge. Everybody can participate in designing the application by collaborating in defining the tests.
6) The tests are the one source of truth for the application’s behaviours. The acceptance tests live with the code and are executed in CI, they will always be up to date unless the application diverges. The tests are the application documentation. With new functionality come new tests or old tests get adapted to fit the new behaviours.
7) Increased communication builds trust and alliances between the actors
8) Increases transparency; the tests content and results are shared, easily accessible and have high visibility among all actors.

Pleasant Side Effects:

1) Acceptance tests are automated and as such they grow into a comprehensive regression suite
2) Can help Impact analysis. If we want to assess the impact of a change on our application we can simulate that change and verify what pre-existing behaviours it affects by reading the acceptance test results.
3) In the not so uncommon scenario of a re-write, if the original application was written using ATDD it is now possible to recreate the old functionality that needs to be transferred by reusing the Acceptance Tests

Acceptance Tests add no value and can be counter productive when…

1) When we write Acceptance Tests for anything but new deliverables
2) When we write Acceptance Tests to create a regression suite of an existing application
3) When we confuse Acceptance Tests with the tool for writing Acceptance Tests
4) When we write Acceptance Tests that do not give fast feedback
5) When we write Acceptance Tests that are brittle and difficult to maintain
6) When we write Acceptance Tests  that contain variables and parameters
7) When we write Acceptance Tests  that contain unnecessary detail
8) When we write Acceptance Tests  that do not focus on what the change is but on how to exercise such change
9) When we write Acceptance Tests  that test the application using the UI when the change is not in the UI
10) When we write Acceptance Tests in isolation and we hide them in source control
11) When we write Acceptance Tests as workflows/scripts

Finally, if you want to learn how to write better acceptance tests have a look at https://mysoftwarequality.wordpress.com/2012/12/14/how-to-transform-bad-acceptance-tests-into-awesome-ones/

The real value of BDD

Talk talk talk talk
Talk talk talk talk

I have been working with agile teams for a few years now and I have used a flavour or another of BDD almost since the very start of my journey. Like everybody I used it badly at the start, misunderstood it and made a mess of it for a good while. Unlike others I have no problems admitting it. Mistake after mistake, fuck up after fuck up I learned a lot of things and today I believe I have developed a decent approach that I will try to improve. Through these years of many failures and small successes one thing clearly emerged for me and it is I believe the true value of BDD.

To me BDD is about communication, collaboration and delivering business value.

The most important part of it? The discussions you have with your team just before you write the tests.

The inherited automation, the derived documentation, the shared knowledge, the reduction in gold plating are side effects and it’s nice to have them, but the real value is in the conversations.  The conversations where the team explores new grounds, the conversation where if you want to add value, it doesn’t matter how senior or technical or business savvy you are, what it counts is how much you care about your customer.

Things that make me happy

What did you learn today?
What did you learn today?

…doing a workshop on ATDD to new hires that are enthusiastic and participate actively making the experience fulfilling.

Myself and my colleague Mary Walshe gave our now traditional ATDD workshop to PaddyPower’s latest new hires and the atmosphere was great, everybody wanted to participate and learn, it just felt great.

Even more satisfying is seeing their answers to the “what did you learn today?” question on post its.  They really got it!

I am a lucky man, I enjoy my job.


Be lean, ask “why?”


Your best piece of code is the one you won’t have to write.

Yes I am serious, and no, I am not intoxicated, let me tell you a story.

It was a sunny summer day in Dublin and “the three amigos” (Tester, Developer and Product Owner) were discussing the next user story to deliver.  It all started as usual with the Product Owner explaining what we were meant to do and as the story was quite small and easy we were soon starting to write our examples that would become acceptance tests and eventually code.

Everything was going smoothly, we had already agreed that we would add a field to our payload with the new information, until out of the blue, the tester said:  “why? I mean, why do we need to do this?”

The product Owner said: “Because we need to send extra information to customer A!”

The tester insisted: “But why do we need to send it to customer A?”

The product Owner said: “Because we always send it to system X so that it can create product Y”

Then the tester said: “But customer A doesn’t use system X and doesn’t sell product Y, so why do we have to send it?”

The product Owner said: “You might be right, but Mr. SuperBoss said he wants to send it!”

The tester said: “Well, we need to ask Mr. SuperBoss WHY he wants to send it”

The product Owner stood up, went straight to Mr. SuperBoss and asked him.

The fact is, there was no reason, we were wrongly assuming that customer A would need it.

Would you agree with me that the code we didn’t write is super clean, extremely cheap to develop, test, deploy, maintain and sunset?

It only ever costed a conversation, and a few whys, I bet you can’t do it much cheaper!

If you don’t see the business value of what you are delivering, ask for it, you might get a nice surprise and find out you don’t need it at all.

How to transform bad Acceptance tests into Awesome ones

So you want to learn how to write good acceptance tests? There’s only one way, let’s write some.

This is a practical example that is designed to help beginners write clear and easily maintainable acceptance tests.

Our System

We are BOG, “Bank Of Gus” and we have a Loan Approval Processing System that takes in input some data regarding the applying customer and his loan requirements, as output it returns either Accept (the customer will be given the loan) or Reject (the customer will not be given the loan).

The marketing manager wants to start selling a new Holiday Loan and produces the following user story:

As a Customer
I want to borrow money from the bank
So that I can go on Holiday and enjoy myself

Acceptance Criteria:
In order to get the Holiday Loan approved
1) The Customer must be 18 or older
2) The Customer’s salary must be > €20,000.00
3) The Customer Time in employment must be >= 6 months
4) The Loan amount < (Customer salary)/5

The Loan Application Form (UI) exists already, the Loan Application Form calls a REST service that is what we are now updating to allow for this new product. The UI is also ready and able to display our outcome, a big green Approved or red Rejected string based on what our service returns.

The Loan Application Form looks something like this:

I am eager to start writing acceptance tests, and I start writing down the first one without thinking much, (please don’t get bored by the first test, I promise it gets better, this is the worst one you’ll see)

I’m going to use a rich celebrity for my first test, let’s try to make things interesting.


Auch… 16 steps for ONLY ONE TEST, by the time I do all the necessary scenarios with boundary analysis I am going to have a document the size of the iTunes licence agreement and this is only the start!

HINT #1: Focus on “what” you are testing and not on “how”

First of all, do I really need to say that I go to a page and that I fill each field and that I push a button? That’s the “how” I use the app, it is not necessarily “what” I do with it. The “what” is “a customer is applying for a loan”.

Could I generalize and get to a concept of a customer applying for a loan? YES
Do I really need to fill the web form to exercise the code I am writing/testing? NO
Can I abstract it, use a test double and call directly my code? YES

Focus on what you are testing, you are not testing the UI, you are testing the loan approval logic! You don’t need to exercise it through the UI. You would exercise it through the UI only if you were testing the UI.

Ok let’s use a test double. I create a mock with the data as per example above and will use it for testing, but, it’s not making writing the test any easier Sad

I could do something like


Besides the fact that I abstracted the how (the customer entering strings and clicking buttons) with the what (the customer applying for a loan) I still have a very messy test with loads of detail and quite difficult to read and maintain.

It looks slightly better but not good enough, I couldn’t even fit all the data on one line and I took the lazy option of adding ellipses, but in the real world ellipses don’t work, they can’t be automated, imagine repeating this for all the scenarios I need to cover, it’s a disaster, what am I going to do?

HINT #2: Eliminate irrelevant Detail

Do I really need to know the name of the customer to decide if I want to approve his loan? NO
Do I need to know his sex? NO
Shall I continue asking rethorical questions? NO

The only important variables for designing the logic of my application are the ones described in the acceptance criteria, look back: Age, Salary, Time in employment, Loan amount

OK this looks promising, let me try to write the original test using only those.


This looks definitely better, it exposes only the parameters that have an impact on the loan approval logic, it is more readable and while reading it I have some idea of how the system will work, that’s better isn’t it?

OK let’s write all the scenarios to comply with the acceptance criteria using boundary analysis, equivalence partitioning and other test techniques.


Auch again… I haven’t even started looking at the cases where the loan will be rejected and I have already 4 very similar tests that will bore to tears the Product Owner, so much that he won’t speak to me for a month, what can I do?

HINT #3: Consolidate similar tests with readable tables

I know of a very useful way of writing tests that are very similar without repeating myself over and over and make the readers fall asleep. It’s called scenario outline and I’m not going to explain in words what it does, I’m just going to show it to you because I know that looking at it you won’t require any explanation.


Wow, this looks much better! One test of 3 lines and examples that cover all the possible scenarios! Do you remember when you needed 16 lines of unnecessary detail to describe only the first line in the examples above? This is certainly a an improvement, more readable, more maintainable and all around 100 times better than the original one.

Also, look at it closely; it gives the business an amazing power of using this test in the future to make changes! Imagine that we end up in a credit crunch (again) and the banks want to tighten the way they lend money. So they decide to increase the minimum salary to 30,000 and the minimum time in employment to 12 months for example.

A quick copy and paste + small refactor and we get:


That’s quite powerful isn’t it?

Now if I was a tester and I wanted to be picky I would tell you that there are plenty of scenarios that have not been tested and a full decision table should be created to give good coverage.

Yes you guessed, I am a picky tester, let’s build the decision table for the Credit Crunch scenario.

HINT #4: Use decision tables and boundary analysis to get high coverage (DON’T DO THIS! It is an anti pattern and I am leaving it here as an example of something I learned to avoid along the way)

How do I build a decision table?
First you need to know what your variables are and what “interesting values” need to be considered.

What’s an “interesting” value? They are all the values a variable can take that might make the logic fail. Generally they are Boundary values.

Ok back to the Credit crunch requirements:

2) Customer salary must be > €30,000.00
3) Customer Time in employment must be >= 12 months

The salary variable, for example has 3 interesting values: 29,999.99, 30,000.00, 30.000.01
Respectively, left boundary, boundary and right boundary (some observers might say that 0 and -1 could be interesting values as well, I agree but for the purpose of this exercise we won’t consider them).

How about time in employment, the interesting values are: 11, 12, 13

OK I have 2 variables each with 3 “interesting” values (or dimensions)

I can immediately calculate the amount of tests I need to get 100% coverage with all possible combinations of “interesting” values.

NumberOfTests = dim(salary)*dim(time_in_employment) = 3*3=9

9 test cases will cover all possible paths using all combinations of “interesting” values.

Let’s build the decision table, and guess what? It can be expressed as a test!


1 test, 3 steps, 9 examples, 100% boundary analysis coverage, in English, readable, maintainable, clearly expressing the business value delivered, what do you want more?

One last thing; you might be in a situation where decision tables with many variables with large dimensions will require hundreds or even thousands of test cases. If these tests are run at the unit level I wouldn’t worry too much about the run time but if for instance you are testing some Javascript logic and are able to do so only through the UI this will require a long time to execute and not represent a good Return On Investment.

What can you do? There are many ways of reducing the amount of tests to run still maintaining relevant coverage. One technique is called pairwise testing, it is very straight forward and uses tools to quickly identify the highest risk tests that should be included and eliminate the ones with less risk associated. Pairwise testing is outside the scope of this document, If you are interested in knowing more about it, check this out! http://bit.ly/T8OXjZ

SCRUM, from failure to success

This is a description of what made our previously failing SCRUM team a success within our organization. The lessons learned through failure were as important as the final success.

Before the “Revolution”: Our organization had been going through a transition to SCRUM for around one year. In the process of transitioning we had delivered a couple of software projects. Such projects had been seen as a failure by our Product Owner (PO from now on) for not delivering business value, and by the development team for failing to deliver a quality product.

The goal: The software development department needed a big success to justify continuing the transition to SCRUM and we were all determined to deliver a great product to our customers to demonstrate how much we had learnt from our own previous failures. The PO was extremely sceptical about continuing with SCRUM and didn’t refrain from showing their feelings.

The Plan: One SCRUM team was created to work on “Project Revolution”. Our goal was to deliver in a short period of time quality software that would exceed customer’s expectations and drive them back to embracing agile development.

The Focus: For the very first time we religiously adhered to SCRUM practices, we focused our efforts on Software Quality practices and built a solid relationship between the development team and the Product-Owner.

How we did it: We engaged with the PO from day zero and we tried to infect him with our enthusiasm for software development and quality. Before the start of the projects we gave a demonstration of our Quality plans and new software development approach: Acceptance Test Driven Development.  The PO showed interest in our approach beyond our expectations; he was blown away by the power of the tests ubiquitous language and clearly understood its potential value. He was also reassured by the demonstration of the Software Quality infrastructure we had built to harness the development of our application and took large interest in how acceptance tests were run and results reported. The development team discovered that something that seemed initially only a technical matter was very valuable to our PO.

SCRUM framework was followed rigorously by everybody including the PO that in previous projects was somehow cut away from the core of the SCRUM team. This time we really started to discover SCRUM’s real benefits of fast feedback and continuous improvement. The ATDD approach gave great benefits by allowing front-loading the discussions over incomplete/ambiguous requirements at acceptance test creation (the very start of development). We discovered quickly how front-loading such discussions would on one hand slow down development but on the other hand would allow us to develop only once and only what was really required rather than get to the final product by continuously fixing defects. Having a large bed of acceptance and unit tests gave the development team the confidence of refactoring freely and we were able to see the positive in the fast feedback of our builds.

Transparency: A full transparency policy was adopted, we were all part of one team, there were no secrets among us.

Collaboration brings success: Slowly the PO started gaining confidence in our work and when at the demos he started saying things like “This is a fantastic job guys!” or even “You’ve done in a 2 weeks sprint what in the past we were used to getting in 3 months and at a level of quality that is not even comparable”. It’s easy to understand that when our PO started trusting us, we were able to go even one step further and propose our alternative solutions to him. While in the past such solutions were categorically refused and a command and control approach was used by the PO, we were now at a stage where full collaboration was the norm and feedback was working both ways.

Fun: The product was delivered in time with excellent level of quality, the products’ business value exceeded the PO original expectations and best of all we all had great fun in developing it.

Project Revolution was an amazing experience.


Industry: Credit Information

Project Scope: Credit Information Management System

Technology: Java (Spring), Tomcat, HTML, jQuery, SOAP, Oracle ESB

Tools: JUnit, Cucumber, Selenium, Crucible, Sonar, Jenkins, Maven