mysoftwarequality

Breaking people's balls since February 1970

How to transform bad Acceptance tests into Awesome ones

So you want to learn how to write good acceptance tests? There’s only one way, let’s write some.

This is a practical example that is designed to help beginners write clear and easily maintainable acceptance tests.

Our System

We are BOG, “Bank Of Gus” and we have a Loan Approval Processing System that takes in input some data regarding the applying customer and his loan requirements, as output it returns either Accept (the customer will be given the loan) or Reject (the customer will not be given the loan).

The marketing manager wants to start selling a new Holiday Loan and produces the following user story:

As a Customer
I want to borrow money from the bank
So that I can go on Holiday and enjoy myself

Acceptance Criteria:
In order to get the Holiday Loan approved
1) The Customer must be 18 or older
2) The Customer’s salary must be > €20,000.00
3) The Customer Time in employment must be >= 6 months
4) The Loan amount < (Customer salary)/5

The Loan Application Form (UI) exists already, the Loan Application Form calls a REST service that is what we are now updating to allow for this new product. The UI is also ready and able to display our outcome, a big green Approved or red Rejected string based on what our service returns.

The Loan Application Form looks something like this:

I am eager to start writing acceptance tests, and I start writing down the first one without thinking much, (please don’t get bored by the first test, I promise it gets better, this is the worst one you’ll see)

I’m going to use a rich celebrity for my first test, let’s try to make things interesting.

first_test

Auch… 16 steps for ONLY ONE TEST, by the time I do all the necessary scenarios with boundary analysis I am going to have a document the size of the iTunes licence agreement and this is only the start!

HINT #1: Focus on “what” you are testing and not on “how”

First of all, do I really need to say that I go to a page and that I fill each field and that I push a button? That’s the “how” I use the app, it is not necessarily “what” I do with it. The “what” is “a customer is applying for a loan”.

Could I generalize and get to a concept of a customer applying for a loan? YES
Do I really need to fill the web form to exercise the code I am writing/testing? NO
Can I abstract it, use a test double and call directly my code? YES

Focus on what you are testing, you are not testing the UI, you are testing the loan approval logic! You don’t need to exercise it through the UI. You would exercise it through the UI only if you were testing the UI.

Ok let’s use a test double. I create a mock with the data as per example above and will use it for testing, but, it’s not making writing the test any easier Sad

I could do something like

second_test

Besides the fact that I abstracted the how (the customer entering strings and clicking buttons) with the what (the customer applying for a loan) I still have a very messy test with loads of detail and quite difficult to read and maintain.

It looks slightly better but not good enough, I couldn’t even fit all the data on one line and I took the lazy option of adding ellipses, but in the real world ellipses don’t work, they can’t be automated, imagine repeating this for all the scenarios I need to cover, it’s a disaster, what am I going to do?

HINT #2: Eliminate irrelevant Detail

Do I really need to know the name of the customer to decide if I want to approve his loan? NO
Do I need to know his sex? NO
Shall I continue asking rethorical questions? NO

The only important variables for designing the logic of my application are the ones described in the acceptance criteria, look back: Age, Salary, Time in employment, Loan amount

OK this looks promising, let me try to write the original test using only those.

third_test

This looks definitely better, it exposes only the parameters that have an impact on the loan approval logic, it is more readable and while reading it I have some idea of how the system will work, that’s better isn’t it?

OK let’s write all the scenarios to comply with the acceptance criteria using boundary analysis, equivalence partitioning and other test techniques.

fourth_test

Auch again… I haven’t even started looking at the cases where the loan will be rejected and I have already 4 very similar tests that will bore to tears the Product Owner, so much that he won’t speak to me for a month, what can I do?

HINT #3: Consolidate similar tests with readable tables

I know of a very useful way of writing tests that are very similar without repeating myself over and over and make the readers fall asleep. It’s called scenario outline and I’m not going to explain in words what it does, I’m just going to show it to you because I know that looking at it you won’t require any explanation.

sixth_test

Wow, this looks much better! One test of 3 lines and examples that cover all the possible scenarios! Do you remember when you needed 16 lines of unnecessary detail to describe only the first line in the examples above? This is certainly a an improvement, more readable, more maintainable and all around 100 times better than the original one.

Also, look at it closely; it gives the business an amazing power of using this test in the future to make changes! Imagine that we end up in a credit crunch (again) and the banks want to tighten the way they lend money. So they decide to increase the minimum salary to 30,000 and the minimum time in employment to 12 months for example.

A quick copy and paste + small refactor and we get:

seventh_test

That’s quite powerful isn’t it?

Now if I was a tester and I wanted to be picky I would tell you that there are plenty of scenarios that have not been tested and a full decision table should be created to give good coverage.

Yes you guessed, I am a picky tester, let’s build the decision table for the Credit Crunch scenario.

HINT #4: Use decision tables and boundary analysis to get high coverage

How do I build a decision table?
First you need to know what your variables are and what “interesting values” need to be considered.

What’s an “interesting” value? They are all the values a variable can take that might make the logic fail. Generally they are Boundary values.

Ok back to the Credit crunch requirements:

2) Customer salary must be > €30,000.00
3) Customer Time in employment must be >= 12 months

The salary variable, for example has 3 interesting values: 29,999.99, 30,000.00, 30.000.01
Respectively, left boundary, boundary and right boundary (some observers might say that 0 and -1 could be interesting values as well, I agree but for the purpose of this exercise we won’t consider them).

How about time in employment, the interesting values are: 11, 12, 13

OK I have 2 variables each with 3 “interesting” values (or dimensions)

I can immediately calculate the amount of tests I need to get 100% coverage with all possible combinations of “interesting” values.

NumberOfTests = dim(salary)*dim(time_in_employment) = 3*3=9

9 test cases will cover all possible paths using all combinations of “interesting” values.

Let’s build the decision table, and guess what? It can be expressed as a test!

eight_test

1 test, 3 steps, 9 examples, 100% boundary analysis coverage, in English, readable, maintainable, clearly expressing the business value delivered, what do you want more?

One last thing; you might be in a situation where decision tables with many variables with large dimensions will require hundreds or even thousands of test cases. If these tests are run at the unit level I wouldn’t worry too much about the run time but if for instance you are testing some Javascript logic and are able to do so only through the UI this will require a long time to execute and not represent a good Return On Investment.

What can you do? There are many ways of reducing the amount of tests to run still maintaining relevant coverage. One technique is called pairwise testing, it is very straight forward and uses tools to quickly identify the highest risk tests that should be included and eliminate the ones with less risk associated. Pairwise testing is outside the scope of this document, If you are interested in knowing more about it, check this out! http://bit.ly/T8OXjZ

About these ads

16 responses to “How to transform bad Acceptance tests into Awesome ones

  1. K. M. Leung (@LeungKimMIng) December 16, 2012 at 4:20 pm

    Hi,

    nice example.

    My experience after practicing BDD and Specification by Example for 2 years, I tend to agree with Paul Gerrard that “by Example” is not enough. The better name should be “with Example”. His has a very sound example. Imagine just create test cases without mentioning (Y = X x X + 2X + 5). Yes, you can have good acceptance tests but no good “Specifications”. Hence, I prefer to add comments:

    Scenario Outline: Credit Crunch Loan Approval Process
    # Customer must have salary over 30,000 & time in employment over 11 years.
    Given …
    When …
    Then…
    Examples:
    | Salary | Time_in_employment | response |

    For the above formula
    # Y = X x X + 2X + 5
    | X | Y |
    | 0 | 5 |
    | 1 | 8 |

    I like to call this “Specification with measurements” and I have applied this concept not only to user stories, but Business Process and Business Goal (http://leungkm.blogspot.hk/2012/08/the-new-v-model.html).

    • mysoftwarequality December 16, 2012 at 5:43 pm

      Thanks for your comment K.M. I have attended Paul Gerrard’s talk at Agile Testing Days in Berlin last month and I quite like his approach. The only slight reservation I would have is the fact that only the examples stay up to date, all the rest is disconnected from the code base and unfortunately can become obsolete.

      • Michael Leung December 18, 2012 at 4:36 pm

        Totally agree but when you transit from hint #3 to hint #4, you cannot rely solely on the examples of hint #3. You recall the requirements:

        “Ok back to the Credit crunch requirements:
        2) Customer salary must be > €30,000.00
        3) Customer Time in employment must be >= 12 months”

        I simply put these requirements back to the Gherkin language as remarks.

        Of course there is the risk of out-sync. but as the examples can be trusted, I can verify the requirements using the examples.

  2. Justin Hunter (@Hexawise) December 17, 2012 at 1:50 pm

    Gus,

    This is my first visit to your blog but it won’t be my last. Great article.

    Well written. Good points. Great clear example. A link encouraging people to learn about pairwise testing…. What more can you ask for?

    – Justin

    • mysoftwarequality December 17, 2012 at 5:55 pm

      Thanks Justin, I am glad you liked it.
      I received quite a lot of valuable feedback on this article and I am working on implementing it to make the tests even more readable, make sure you drop by again in a few days :-)

  3. Andrey Botalov December 17, 2012 at 10:50 pm

    Do you plan to exercise all examples from 6th table through UI? If yes, it will be CO$TLY. If not, then what examples will be run through UI?

    • mysoftwarequality December 17, 2012 at 10:58 pm

      Hi Andrey, in this case we are building logic in a REST service, my choice will be not to test it through the UI (as specified in the article) but directly against the service itself. There is no logic in the UI to test, hence no need to test through the UI. None of the examples in this article should be run through the UI. So fast and cheap rather than CO$TLY and slow :-)

      • Andrey Botalov December 18, 2012 at 9:47 am

        But I think a UI test (maybe, only one) is still needed. We need to check that data returned by REST service is actually shown to the user

        • mysoftwarequality December 18, 2012 at 10:06 am

          Integration tests are indeed needed Andrey, I just would not formulate as an acceptance test. I write acceptance tests mainly for driving the production code, I consider the fact that they are also automated and will provide regression in the future as a pleasant side effect. But BDD to me is about software development and not about testing. Normally I would use other instruments/frameworks to write integration tests also because I use the Acceptance tests as documentation of the application and adding an integration test there might not make sense and simply re-state the same behaviour at a different level. If you have a way to avoid this I would be very interested!

      • Michael Leung December 26, 2012 at 5:00 pm

        My preference is to use MVC or MVVM in UI and drive the automated acceptance tests thru the controller or viewmodel. Using the controller, you can click a button and examine values in those binding valuables. UI are just dummy views without logic. This is the closest to UI test.

        As for real-life end-to-end business process testing, our users will preform them only once when enough stories for the business process are developed. They are not only testing the system but the people readiness and business processes readiness.

        Once accepted, they will trust the automated testing for regression tests in subsequent iterations and will not re-execute them again.

  4. Pingback: How to Write Good Acceptance Tests - Testing Excellence

  5. Pingback: When something works, share it! | mysoftwarequality

  6. Ntobeko June 2, 2014 at 2:42 pm

    Hi,

    Ditto for some good work on the examples above. What tool did you use to write your acceptance test?

    Regards
    Ntobeko

  7. Pingback: Sydney testers meetup #37 – Journey of Acceptance Test Driven Development #Sydtest | Happytesting

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 59 other followers

%d bloggers like this: