Executable specifications – what would you do?

Can you help?

Executable specifications are used in BDD/Spec By Example as a mean to describe an application behaviour in business language. They are executable because they are linked to tests that exercise the behaviour of the application itself.

Let’s imagine we want to develop the solution to the FizzBuzz problem, Which one of the following solutions would you adopt and why? If you have a different better solution than the two I list, please add it!

1. Specific examples:

Given we are printing numbers using FizzBuzz
When the number is <number>
Then we will print <answer>

|      1      |     1    |
|      3      | Fizz    |
|      5      | Buzz  |
|      6      |  Fizz   |
|      10    | Buzz   |
|      15    |FizzBuzz|

2. Generic Rules

Given we are printing numbers using FizzBuzz
When the number is a multiple of 3
Then we will print Fizz

Given we are printing numbers using FizzBuzz
When the number is a multiple of 5
Then we will print Buzz

Given we are printing numbers using FizzBuzz
When the number is a multiple of 3 and 5
Then we will print FizzBuzz

Given we are printing numbers using FizzBuzz
When the number is a not a multiple of either 3 or 5
Then we will print the number

23 thoughts on “Executable specifications – what would you do?

  1. In this case I would probably lean more towards the general rules. To me it provides more information about the rules.

    Could you get the best of both examples though by including the name of the specification/scenario?

    Scenario: A number divisible by 3 will print “Fizz”
    Given we are printing numbers using FizzBuzz
    When the number is 3
    Then we will print Fizz

    Scenario: A number divisible by 5 will print “Buzz”
    Given we are printing numbers using FizzBuzz
    When the number is 5
    Then we will print Buzz

  2. I would probably go with the specific examples, using the examples table. But I would add more to the examples table to cover some other variables around error handling…

    Scenario: Printing numbers using FizzBuzz
    Given I enter into the field
    Then should be displayed

    | input | output |
    | 1 | 1 |
    | 3 | fizz |
    | 5 | buzz |
    | 6 | fizz |
    | 10 | buzz |
    | 15 | fizzbuzz |
    | 100 | 100 |
    | 101 | Invalid |
    | -3 | Invalid |
    | 0 | Invalid |
    | a | Invalid |

    I think this is much quicker for people to read and is more efficient for people to write too.
    It’s the same scenario, but just varying the input data to return the 5 different various output strings.

    Its also worth noting that I would have conversations with the stakeholders regarding a suitable level of coverage regarding the valid input numbers too. Is checking 1 number to cover each output message enough? Is there variables around numbers that might affect the output (like single digits versus double digits)? etc…

    I might incorporate some randomisation into my examples too (as each of my examples seem stuck on the “firsts” pattern – e.g. first number to produce a number (“1”), first number to produce a Fizz (“3”), first number to produce a Buzz (“5”), first multiples for Fizz and Buzz independently (“6” and “10”), first number to produce a FizzBuzz (“15”), etc… I might want to mix this up a bit through randomisation to cover other patterns).

  3. Interesting question – I tend to go to the generic rule because a set of examples is not a specification in this case.
    However – I think you need to define the rules more strictly, as the current rules are buggy.
    1) when number is multiple of 3 and not multiple of 5
    2) when number is multiple of 5 and not multiple of 3
    3) when number is multiple of 15 (or 3 and 5, whichever suits you)

    I wonder though – is the value of “executable requirements” worth the fuss? it seems to me that it is only a way to gain a false sense of safety when a bug in the requirements causes a false “passed” execution.
    Another concern I have is that requirements are (if they are good) succint and easy for a human to understand. executable requirements are cumbersome – The requirements you gave are understandable by a human but buggy for a machine. were we to change the requirement so that numbers divisible by 7 will add “bang”, this would complicate the executable requirements greatly.

    • Thanks for your feedback Amit.
      Maybe executable specifications are overkill for the FizzBuzz game, but I found them extremely helpful for more complex scenarios.
      In my humble opinion the most important part is the conversation team members have before writing the executable specifications, but this is not the core of this thread so I won’t dwell on it.
      Adding bang for 7 would be quite easy using the first approach, don’t you think?

      • Thanks for the answer,
        One thing that you’ve said I actually find quite intriguing. You mentioned that executable requirements actually helped you in more complex projects, where my expectations were that the more complex the product is, the more difficult it will be to use them effectively.
        What led me thinking this way was that while written static requirements have obvious drawbacks, they have one major advantage – they are workable even if not carefully worded and complete.
        Using the FizzBuzz example, with static requirements I can get away with saying “every 3rd number is a fizz, every 5th is a buzz, when a number is both – it’s fizzbuzz”
        in executable requirements I’ll actually need to describe the complete logic – if it’s 3rd but not 5th, if it’s 5th but not 3rd and if it’s both. It seems to me both cumbersome and risky.
        (plus, in my experience, once anything is “code” some people immediately trust it)

        Could you describe a case in which executable requirements helped you?were all requirements made executable or just some of them?

        As for ease of change – I agree, the 1st approach is very easy to change this way (but not if we chage 5s from “buzz” to “woof”), but my concern is that examples in this case just don’t make the rule obvious, leaving the reader to guess the rule himself.

        • Thanks for your feedback Amit.
          When you say “every 3rd number is a fizz, every 5th is a buzz, when a number is both – it’s fizzbuzz” you state a set of rules, when you express the same as in the examples in my post you make the rules explicit. This IMHO can make it easier to put into code, in particular if you use one test at the time in pure TDD style.
          As I said earlier, the most important part in using BDD/Spec By Example is the conversation team members have before writing the executable specifications, that’s where most of the value of such approaches is realised. I express this concept in more detail here https://mysoftwarequality.wordpress.com/2013/11/09/the-real-value-of-bdd/
          Where did executable specifications help me? Difficult to say I have been developing test first with executable specifications for many years, if they didn’t help me I wouldn’t bother 🙂

    • Interesting response. I was looking at it from a slightly different perspective from you regarding “executable requirements”. I was looking at it from a communication perspective before developing.

      The way I was thinking was that we are supplying the requirement example with the initial GWT which is the example of the functionality – essentially: “Given we supply an input, we get an output”.
      The examples table allows us to simplify our english efficiently for the further communication around the variables of the feature (the variables being different input data and different output information). We can essentially simplify our verbal communication to “inputting ‘1’ should output a ‘1’ ” or “inputting ‘3’ should output ‘Fizz’ “.

      Simplifying our sentences (even just using less words in the sentence), usually equates to a reduction in the risks of confusion and misunderstanding. Starting to move into the realms of psychology here…

      Anyway, from being able to then pick this up and start developing, it would appear more effective as we have detailed the functionality, and listed examples of the variables of the inputs and expected outputs.

      But having the perspective of “executable” meaning machine checkable, then I agree… If we ONLY rely on this information (that /should be/ written up front at the beginning for design purposes) then we miss testing the things that we do not know and are not able to write with an explicit expectation at that point in time of writing the feature file. I guess that’s a risk regarding people’s mindset of /testing/ though.

  4. Hi Augosto!

    When I look at the 2nd option (general rules), I tend to think what is the first matching rule, it will be considered. If this is true, 3rd rule would never be applied. (I numbered them since you left an empty row between them.) Thus, I’d put 3rd rule as 1st. Then, if this rules would not match the condition, the script would continue to see if number is 3, 5, or neither.

    I’d adjust it a bit more still and move the “if neither is 3 or 5” as 2nd option. This is because 1st rule already checks if the number is a multiple of 3 and 5, I know I should just check if I am given 3 or 5. If I’d first check this, I’d get for example 15 printed as a number.

    When it comes down to using non-numbers (and maybe broken data), I think this is covered on General Rules even if it’s not explicitly specified. To be more sure of this, I could add a 5th check “if nothing matches, print a funny cat picture”.

    Hope that clarifies a bit my tweet!

    Best regards,

  5. I believe Gherkin, and BDD in general, are about communication so my answer is “whatever makes more sense for the stakeholders”. By looking at the way the requirement is expressed _”Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”_ I believe the second approach better matches with the requirement format, which should be more understandable by whomever defines them.

  6. Hi Gus,
    I think this is a really interesting question.
    Personally I like the Given When Then Approach to driving the behaviour.
    Two reasons:
    1. I think it is less ambiguous. In the first technique the tester could hard code the outcome only for the numbers given, and not spot the pattern. The pattern is called out in the second technique
    2. I think that the first technique is more prescriptive to how the test works. It is declaring the values, not the behavior. The second example reads as a manual to the team, and the developer is trusted to test that behavious as they see fit.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s