Breaking people's balls since February 1970
How to transform bad Acceptance tests into Awesome ones
December 14, 2012Posted by on
So you want to learn how to write good acceptance tests? There’s only one way, let’s write some.
This is a practical example that is designed to help beginners write clear and easily maintainable acceptance tests.
We are BOG, “Bank Of Gus” and we have a Loan Approval Processing System that takes in input some data regarding the applying customer and his loan requirements, as output it returns either Accept (the customer will be given the loan) or Reject (the customer will not be given the loan).
The marketing manager wants to start selling a new Holiday Loan and produces the following user story:
As a Customer
I want to borrow money from the bank
So that I can go on Holiday and enjoy myself
In order to get the Holiday Loan approved
1) The Customer must be 18 or older
2) The Customer’s salary must be > €20,000.00
3) The Customer Time in employment must be >= 6 months
4) The Loan amount < (Customer salary)/5
The Loan Application Form (UI) exists already, the Loan Application Form calls a REST service that is what we are now updating to allow for this new product. The UI is also ready and able to display our outcome, a big green Approved or red Rejected string based on what our service returns.
The Loan Application Form looks something like this:
I am eager to start writing acceptance tests, and I start writing down the first one without thinking much, (please don’t get bored by the first test, I promise it gets better, this is the worst one you’ll see)
I’m going to use a rich celebrity for my first test, let’s try to make things interesting.
Auch… 16 steps for ONLY ONE TEST, by the time I do all the necessary scenarios with boundary analysis I am going to have a document the size of the iTunes licence agreement and this is only the start!
HINT #1: Focus on “what” you are testing and not on “how”
First of all, do I really need to say that I go to a page and that I fill each field and that I push a button? That’s the “how” I use the app, it is not necessarily “what” I do with it. The “what” is “a customer is applying for a loan”.
Could I generalize and get to a concept of a customer applying for a loan? YES
Do I really need to fill the web form to exercise the code I am writing/testing? NO
Can I abstract it, use a test double and call directly my code? YES
Focus on what you are testing, you are not testing the UI, you are testing the loan approval logic! You don’t need to exercise it through the UI. You would exercise it through the UI only if you were testing the UI.
Ok let’s use a test double. I create a mock with the data as per example above and will use it for testing, but, it’s not making writing the test any easier
I could do something like
Besides the fact that I abstracted the how (the customer entering strings and clicking buttons) with the what (the customer applying for a loan) I still have a very messy test with loads of detail and quite difficult to read and maintain.
It looks slightly better but not good enough, I couldn’t even fit all the data on one line and I took the lazy option of adding ellipses, but in the real world ellipses don’t work, they can’t be automated, imagine repeating this for all the scenarios I need to cover, it’s a disaster, what am I going to do?
HINT #2: Eliminate irrelevant Detail
Do I really need to know the name of the customer to decide if I want to approve his loan? NO
Do I need to know his sex? NO
Shall I continue asking rethorical questions? NO
The only important variables for designing the logic of my application are the ones described in the acceptance criteria, look back: Age, Salary, Time in employment, Loan amount
OK this looks promising, let me try to write the original test using only those.
This looks definitely better, it exposes only the parameters that have an impact on the loan approval logic, it is more readable and while reading it I have some idea of how the system will work, that’s better isn’t it?
OK let’s write all the scenarios to comply with the acceptance criteria using boundary analysis, equivalence partitioning and other test techniques.
Auch again… I haven’t even started looking at the cases where the loan will be rejected and I have already 4 very similar tests that will bore to tears the Product Owner, so much that he won’t speak to me for a month, what can I do?
HINT #3: Consolidate similar tests with readable tables
I know of a very useful way of writing tests that are very similar without repeating myself over and over and make the readers fall asleep. It’s called scenario outline and I’m not going to explain in words what it does, I’m just going to show it to you because I know that looking at it you won’t require any explanation.
Wow, this looks much better! One test of 3 lines and examples that cover all the possible scenarios! Do you remember when you needed 16 lines of unnecessary detail to describe only the first line in the examples above? This is certainly a an improvement, more readable, more maintainable and all around 100 times better than the original one.
Also, look at it closely; it gives the business an amazing power of using this test in the future to make changes! Imagine that we end up in a credit crunch (again) and the banks want to tighten the way they lend money. So they decide to increase the minimum salary to 30,000 and the minimum time in employment to 12 months for example.
A quick copy and paste + small refactor and we get:
That’s quite powerful isn’t it?
Now if I was a tester and I wanted to be picky I would tell you that there are plenty of scenarios that have not been tested and a full decision table should be created to give good coverage.
Yes you guessed, I am a picky tester, let’s build the decision table for the Credit Crunch scenario.
HINT #4: Use decision tables and boundary analysis to get high coverage
How do I build a decision table?
First you need to know what your variables are and what “interesting values” need to be considered.
What’s an “interesting” value? They are all the values a variable can take that might make the logic fail. Generally they are Boundary values.
Ok back to the Credit crunch requirements:
2) Customer salary must be > €30,000.00
3) Customer Time in employment must be >= 12 months
The salary variable, for example has 3 interesting values: 29,999.99, 30,000.00, 30.000.01
Respectively, left boundary, boundary and right boundary (some observers might say that 0 and -1 could be interesting values as well, I agree but for the purpose of this exercise we won’t consider them).
How about time in employment, the interesting values are: 11, 12, 13
OK I have 2 variables each with 3 “interesting” values (or dimensions)
I can immediately calculate the amount of tests I need to get 100% coverage with all possible combinations of “interesting” values.
NumberOfTests = dim(salary)*dim(time_in_employment) = 3*3=9
9 test cases will cover all possible paths using all combinations of “interesting” values.
Let’s build the decision table, and guess what? It can be expressed as a test!
1 test, 3 steps, 9 examples, 100% boundary analysis coverage, in English, readable, maintainable, clearly expressing the business value delivered, what do you want more?
What can you do? There are many ways of reducing the amount of tests to run still maintaining relevant coverage. One technique is called pairwise testing, it is very straight forward and uses tools to quickly identify the highest risk tests that should be included and eliminate the ones with less risk associated. Pairwise testing is outside the scope of this document, If you are interested in knowing more about it, check this out! http://bit.ly/T8OXjZ