Because there is not such a thing as a best practice
ATTENTION: This won’t work for everybody, all I claim is “it works in my context” and you might try to use it at your own risk.
WARNING: If you believe in best practices that should be applied to every context go away now, you might be harmed by this post
One day, a few years ago, I had a conversation with an inspirational man, he was at that time my CTO. He was talking about zero tolerance to bugs and how beneficial is to remove the annoyances of bug management from the development of software. I was listening but at that time I didn’t grasp completely the concept, because by then, I had never seen neither envisioned a situation where a development team could produce zero or for that matter even close to zero bugs. Since then a few years have passed and I have worked hard on his vision. Today I can say, yes he was right and a development team can deliver value with 0 bugs, I even learned that the project team doesn’t have to spend half of its time playing with a bug management tool to file, prioritise and shift waste.
Let me give you some context around my project and current practices:
Two co-located cross functional agile teams each with 4 developers, one tester, one business analyst, one product owner one dev-ops guy and a team lead also known as kanban facilitator (18 people)
1. We use Kanban to visualize our process and we have the ability to deliver every time we complete a user story if we wanted.
2. User stories are small and we create them so that we can deliver them in less that 3 days
3. User stories are vertical, i.e. no big bang integration is required between the teams, each team works on the full codebase that spans across multiple applications
4. We follow an hybrid between ATDD and Specification By Example that I have specifically developed for our context
5. Developers code review every line of code before every push, we also do some pair programming mainly to train junior developers
6. Developers maintain a healthy 95% or more unit test code coverage, we also enforce all of Sonar coding rules
7. Some of the developers practice TDD
8. We use Jbehave and Thucydides to automate ~100% of our acceptance tests
9. We have an internally developed “platform on demand style” build and deployment pipeline that runs all our functional tests and performance/load tests that check variations against a known baseline
10. Every automated test runs after every push to trunk in such pipeline
11. Every push to trunk gets deployed automatically and can be manually promoted to exploratory testing stage and beyond
12. After all the acceptance tests pass for a user story, we do exploratory testing on it
13. After exploratory testing is complete, we demo to our product owners that subsequently do user acceptance testing before accepting the story
14. If a bug is found during exploratory testing or user acceptance testing the developer(s) that worked on that user story, drop everything else they might be doing and fix the bug, the card will NOT progress if a valid bug is not fixed.
15. If a build becomes red, the developer who causes the instability drops everything else and fixes the issue straight away. Build must be always green.
16. When a developer fixes a bug, he writes automated tests that cover such path
You noticed I mentioned bugs in 14 and 16. Yes, of course, we are humans and we make mistakes, this doesn’t mean that we need to celebrate the mistake and make it visible by logging it in bug tracking tools that we will need to run and maintain among with the waste stored into them when the poor bug has the life span of an unlucky butterfly.
When i find a bug while exploratory testing I simply go to the developer and tell him, look pal I think there might be an issue, come to my desk and I’ll show you. If he needs to go home and can’t fix it straight away, no problem, I will stick a red post it on the user story card on the physical Kanban board with 2 words describing the bug so nobody will forget. BTW that card is not going anywhere until the bug is fixed and buried. The red post it goes into the bin after the death of the bug.
The development approach we follow, allows us to have a very good understanding of the business value we deliver as we do group discussions to derive examples for each user story, when you add a bunch of excellent developers that follow good engineering practices you will find out that the bugs you discover when exploring the software are very few, once you act upon them immediately, no excuse, bugs become something that doesn’t really
bother you.
In this situation, logging and managing bugs is simply waste and we all live happy with no bug ping pong between developers and testers, no bug prioritization meetings, no bug triage meetings, no bug statistics, no need for bug trends to identify product release dates, and guess what we deliver bugless software that delights our customers.
EDIT: I am not alone! Have a look at the great work from Katrina here http://katrinatester.blogspot.co.uk/2014/11/different-ideas-for-defect-management.html
mysoftwarequality 
Yeh, I got bored working at places where a bug tracker was needed because there soooo many problems ( and because it made people feel like they were doing ‘proper’ testing) I’m now at a better place
Couple of questions:
How do you handle issues such as usability? If you’re testing and find that the UX is clunky and awkward, how do you handle this? Can this be addressed by the dev writing an automated test?
How about some of the other non-functional stuff like performance/security?
and if you’re doing a web app, what about browser bugs? I’ve found a lot of IE8 bugs where you get javascript errors, do you get those and are those handled by writing a test?
and with your great practices, how many bugs are you finding when doing exploratory?
Thanks for your comment Phil, I am glad I am not on my own 🙂
We are currently building a back end application used by existing web clients through a restful api, hence we haven’t had anything to do with usability and browser testing as we don’t have any real user interface other than the existing clients. In the very near future we are planning to give our customers the ability to configure our app through a web application and will be facing the issues you mention.
When in the past I worked on web apps and the UX felt clunky, I discussed it with the product owner and the developers and as a team we made a decision on what to do, in some cases we involved the customers. How do you approach it?
For performance we have defined an acceptable level in an environment similar to production, created a baseline and we run performance tests within our build pipeline and we fail the build when the baseline is exceeded by a percentage. This way we can spot the impact on performance on every check in.
As per security I am not an expert I am afraid 😦
We find very little bugs in exploratory testing, I have found one in the last month, better again, no bugs n UAT or in production yet 🙂
Thanks for the quick reply and details.
We’re using Trello, bugs found in exploratory and/or usability issues get logged there and then there’s a conversation about them.
If you’re not finding anything during exploratory ( I find few bugs as well ) does it make you feel anxious in that you’re not adding value as a tester because you’re not finding anything?
I don’t feel anxious if I don’t find bugs (I used to), I like to believe that i add value in many other activities other than finding bugs. After all if I don’t find bugs I am happy because we can deliver good software built right the first time. How about you, is it a problem for you?
I’m slowly learning to be less anxious and working on expanding my capabilities to be more than just a bug finder – it’s ‘easier’ to be a tester in a target rich environment but I’m liking this new challenge
I love the low-tech Kanban board. I used it for testing one of our major enhancements and now we created our own electronic version for the testing team. We are not agile in sense of whole-team concept but at least I can adopt different practices for my team, when appropriate. I too believe context drives how you approach testing.
Thanks for the comment Bernice, I love working with the board so much more than software tools, of course it requires everybody is co-located but when you have that, it works a treat. Being agile can simply mean do what works for you, reflect and change things that don’t work, best of luck with your team!
Thank you! Do you have any examples on how agile teams might document requirements? I would like to show our B/A an alternative to the traditional requirement document.
What we do is identify goals and for each goal we write user stories. We have a huge wall that is painted so that you can write on it (a massive whiteboard) where we stick the user stories under each goal. When we transfer the stories to the kanban board we might break them down further so that it won’t take more than 3 or max 4 days to complete one. We also store the cards electronically using a tool called lean kit that is quite good as it also enforces wip limits automatically and gives management visibility on progress, throughput and flow. I am going to take a picture of the wall tomorrow and add it to the blog post.
During the first activity of our ATDD cycle, (discuss) we basically do the real analysis of the story, such analysis is a group activity so that it generates shared knowledge.
We never plan ahead for more than a month or so because we found out that while we develop and learn, our old plans are not valid anymore, so we mainly do just in time analysis and planning. It’s fun 🙂
Ah, I almost forgot the most important thing! The user stories after the “distill” phase become automated tests (acceptance tests) that describe the behaviour of the application in a business language. They have the great advantage of being always up to date because they are linked to the code under test and run every time a a developer pushes any change, unlike requirement documents that normally are out of date 5 seconds after they have been saved and sent out 🙂
Thank you so much for all the details!! Yes one problem with requirements is just as you stated. They are out dated so quickly and then no one wants to update them because they become a chore to do. I look forward to seeing your picture when it is posted. Thanks again for the info!
No problem Bernice, I have added two pictures of the project wall, let me know if you have any questions.
Well…you are still logging bugs, just using a whiteboard instead of a software tool. All that is made possible by a dev team that actually believes in fixing issues right away and integrating automation tests where needed. Good deal!
Keith, thanks for your feedback. Yes, the bugs that are alive overnight end up on a post it, but as soon as they have been killed, the post it ends up in the bin 🙂
I certainly couldn’t do it if the development team delivered bad code, the lack of logging is only the result of good lean practices.
It makes my heart glad to hear stories like yours. I tell people it is possible, but when they don’t really believe me until they see “real” teams doing it. I would like to point people to this story as an example, if I may.
I am delighted you enjoyed my story, and let me tell you, we wouldn’t have been able to get there without your precious help. We have been using a lot of what you taught us in the week you worked with us.
Of course you can share it, I write my blog posts to encourage the agile testing community, if you know of testers that need help, send them to this place 🙂
[…] Read the full article… […]
All sounds good. I’m glad you’ve found something that works in your context. It must be satisfying to the sculptor when he is able to chip away all the unnecessary stone and reveal the statue hiding within.
Is there anything about the process that could be improved? If so, what? If not, are you content “doing whatever it is that you now do” or would you prefer to go elsewhere and teach/implement this process at another organization?
Thanks for your comment dsynadinos.
We change and improve all the times, one thing at the time. It is not uncommon that we try one or 2 things every week, people come up with ideas and we try them, if they don’t work we discard and move along.
There is never a long discussion over whether it will work or not, we try it and find out for ourselves.
I love working as part of my agile team, it’s what i like the most. Part of my role is to help agile teams, so I happen to move from one to the other every 4/5 months . My current org is quite big and to get through all the teams it would take me a good bit of time anyway 🙂 The future? I am agile and only do just in time planning, never more than 2 weeks at the time 🙂
If you only do micro-developments such as you have described, it would be a rare situation in which you would introduce more than one or two bugs at a time. In that case, logging and tracking bugs makes little sense and doesn’t add value. However, I wonder what you do when you have a wrong requirement to fix, or (heaven forbid) have a new requirement added late that causes a complete redesign (e.g., adding security to a system that didn’t have it before).
Michael, thanks for your feedback.
Keeping the user stories to small size (micro development) helps reduce the complexity of each item that gets deployed hence helps the team understand better the business of the story and identify necessary changes early. I haven’t been in a situation where a wrong requirement necessitated a complete redesign of the system so I don’t know the specifics.
As written in the article it might work for a certain community. In one of the comments I read something, we never plan more than one month ahead. This puzzles me. Are the projects ready within one month or how is the deal with the customer calculate? How long will a big project last? How will its costs be estimated?
In 2003 I made the architecture for a relatively big project. I had a project plan that should have the whole thing in 2005 in production. A competitor also started at this time with a similar project.
In 2006 I decided to leave the company out of frustration because the schedule did not hold. It lasted until 2007 when I got an agreement with my company. (Meaning some money for a handshake) In 2009 the project was finished. When I called my former boss he said that it was considered a successful project, – and yes I could use it as a reference at a conference although I was not with the company anymore.
The other company now has a plan to be finished by 2020. I put this here to say that I don’t like extended runs of development. I even had established an agile development process, “crystal clear” (Alistair Cockburn)
But I had never projects where you could just plan one month ahead.
Hans, thanks for your comment. Yes, we do just in time analysis because we have discovered that doing analysis in advance creates a lot of waste as it is not based on empirical data but the further in the future it is the more it is based on assumptions.
Our customers are very happy and don’t care much about whet they are going to get in 2 years because they get a continuous stream of value every week and in some cases even every day.
Thinking about a project that gets fully specified 2 or 3 years in advance makes me shiver to be completely honest.
At that time I was working in the insurance business. (Just as criminal as gaming and betting 🙂 ) The data there has to survive for 30 or more years. And it has to be compliant with data that is already 20 years old. Platforms change, programs are renewed but the data remains to be worked upon.
Don’t worry Hans, I have bigger skeletons in my closet, I helped write software for mortgage applications during the boom, working in the gambling industry is like being an altar boy in comparison.
I appreciate the fact that the data must be in the same format in 20 years time, but what’s the reason why you can’t progressively deliver chunks of functionality every week?
Because I would need bugeting for at least one year. I would not get money, if I come each month to get some:) But we should not discuss about that. It is a different society. One of my former employees works in Berlin and he is telling me similar stories. You are just in a different business environment.
Hans, everything can change, even your budget driven environment, and you could be the change.
“Because there is not such a thing as a best practice”, brilliant, straight out of my favourite phrase books. I get very wary of anyone that tells me we need to adopt something because it is “best practice”. The focus should be on organic development of your own best practice and on going refinement.
The team I work in is 6 months into our agile transformation, just getting comfortable with some of the basics, in both an agile and agile team sense.
Great article as it represents many of the targets the agile team I am in has discussed as worthwhile achievements, and throws in a couple of thoughts that we hadn’t specifically hit on.
Have distributed this at my work to get others thinking.
Paul, thanks for your contribution. I hope this conversation can inspire your team to find its own way of success.
Hey Gus! Good article! I have a question though 🙂
What do you do if you find a bug, but it isn’t severe enough for the business to focus effort on fixing it at that point in time, but plan to fix it later?
And what if you decide to host a bug-bash session? what about the bugs that come out of something like that, which in essence don’t actually have a story ticket to put a red post-it on?
I definitely see value in doing it your way, in your context.
I’ve recently made a move a new company though that creates medical software. I’m not sure about regulations about defect tracking in my new environment, but previously I’ve normally taken the stance that I’ll have a conversation with the developers about any bugs I find and then find out whether it’s a quick/simple fix… if so then I wont log it in a defect tracker unless they actually want it to be logged. But if it’s anything that needs further investigation or is difficult to resolve, then it gets logged so that we have a record of it and that people aware of it.
Hi Dan, good to hear from you, thanks a million for your feedback.
Very good question. Say i observe a behaviour that seems wrong, I would discuss it first with the developer, if there is any doubt on whether we should fix it straight away, we call in the product owner and walk him through it. If the product owner believes that the deliverable still retains its value in the current format but understands that the change proposed will increase the value as a whole, he will decide whether to fix it straight away or create a new card for the backlog. At the end of the day, defects are working items because when fixed they unlock value as much as user stories. The separation between user stories and bugs is something artificial and often creates unnecessary prioritization issues for a product owner.
For your second question, in our context a bug bash would not be practical as we do continuous delivery and we rarely have something unreleased that is older than a week or even less, but if you compare a bug bash to an issue logged from anybody outside the development team, then the product owner would prioritise the work, this is not a common occurrence, we almost never receive bugs from external teams or customers.
Best of luck in your new company, I understand it is a regulated environment, if I could give you only one suggestion it would be: always ask “why?” when you see some wasteful activity around you, and make sure the answer is true. I worked in a bank for years and some people used regulation as an excuse for bad practices, when i started investigating the regulation rules that some people mentioned when justifying some convoluted process I often found out that they didn’t exist but were enforced by some external consultant that 10 years earlier used them as an excuse to sell his tools and services. I slowly started seeing the light and soon after left banking 🙂
[…] [11] How I stopped logging bugs and started living happy […]
[…] https://mysoftwarequality.wordpress.com/2013/09/10/how-i-stopped-logging-bugs-and-started-living-hap… […]
[…] https://mysoftwarequality.wordpress.com/2013/09/10/how-i-stopped-logging-bugs-and-started-living-hap… […]