Ultimate Guide to Reducing the Amount of Defects and other Waste in your Product


What’s a defect? I like this definition.

A defect is anything that threatens the value of the product.

Before we start, let’s agree that:

  1. we don’t want defects that threaten the value of our product
  2. we want to give our customers as much value as possible at all times.

If you don’t agree with 1 and 2 then, don’t waste your time and stop reading now.

download
Software defects aka Bugs

Testers are normally associated with finding defects. Some testers get very protective with the defects they find and some developers can be very defensive about the defects they wrote. Customers don’t like defects, developers don’t like defects, product managers don’t like defects, let’s be honest, nobody likes defects besides some testers.

Why would that be? The reason is that the focus of a lot of testers is on detecting defects, and that’s what they get paid for in a lot of organisations. If you are a tester and love your defects, you might find this article disturbing, if you decide to proceed, do so at your own peril.

 

Defects are waste

Let’s be clear from the start: defects are waste. Waste of time in designing defective products, waste of time in coding defective routines, waste of time in detecting them, waste of time in fixing them, waste of time in re-checking them. Even writing this sentence took a good while, now think how much time it takes you to produce, detect, fix, recheck defects.

Our industry has developed a defect coping mechanism that we call defect management. It is based on a workflow of detecting => fixing => retesting. Throughout the years it has become best practice (sic) to have defect management tools and to log and track defects. Defect management approaches are generally cumbersome, slow, costly and tend to annoy people no matter whether you are a tester that gets his defect rejected, you are a developer that gets a by design feature flagged as defect, or a product manager that needs to spend time prioritising, charting and trending waste.

Another dangerous characteristic of defects is that they can be easily counted and you will always find a pointy haired manager that decides he is going to shed light on the health of his product and on the efficiency of his team by counting and drawing colorful waste charts.

But, if we agree that defects are waste, why are we logging and tracking waste, creating waste charts seems even more ridiculous, wouldn’t it be easier to try to prevent them?

Oh, if only we could write the right thing first and reduce the number of defects we produce! I say we can, be patient and read on.

Software development teams have found many ways of being creative playing with defects, see some examples below.

Example 1: Reward waste

Reward for the wrong reason
Reward for the wrong reason

Some years back I was working on a business critical project in one of 5 scrum teams . Let me clarify first, that our scrum implementation was at best poor, we didn’t release every sprint and our definition of done was questionable.

Close to an important release, we found ourselves in a situation where we needed to fix a lot of defects before going into production. We had 2 weeks and our teams had collectively around 100 defects to go through. Our CTO was very supportive of the defect killing initiative and he was eager to deliver with zero defects. He put in place a plan that included free food all day and night and some pampering for the developers that needed to focus 100% on defect resolution. Then he decided to give a prize to the team that would fix the highest amount of defects.

I remember feeling frightened of the possible future consequences of this reward. I spoke to the CTO and told him that I would have liked more a prize for the team that produced the lowest amount of defects rather than the one that fixed the most. Our CTO was a smart guy and understood the value proposition of my objection, he changed his approach and spoke to the teams on how not introducing defects in the first place is much more efficient than fixing them after they have been coded. Soon after the release, we started applying an approach that focussed on preventing defects rather than fixating on detection. We never had the problem of fixing 100 bugs in 2 weeks again.

A typical defect prioritization meeting
A typical defect prioritization meeting

Example 2: Defects metrics

In my previous waterfall life, I remember when management introduced a performance metric directly linked to defects. Testers were to be judged on the Defect Detection Index calculated as (Number of Defects detected during testing / Total number of Defects detected including production)*100. An index lower than 90 would mean nobody in the test team would get a bonus. Developers were individually judged on the number of defects found in their code by the testers and business analysts were individually judged on the number of defects found by the testers in their requirements.

Welcome to the battlefield!

The bug prioritisation meetings were battles where development managers argued any bug was a missed requirement, product managers argued that every bug was a coding error or a tester misunderstanding and the test lead (me) was simply shouted at and criticised for allowing his testers to go beyond the requirements and make use of their intellectual functions outside a scripted validation routine.

Going to that meeting was a nightmare, people completely forgot about our customers and simply wanted to get their metrics right. The amount of time we wasted arguing and defending our bonuses was astonishing. Our customers were normally unhappy because instead of focusing on value delivery we focussed on playing with defects, what a bunch of losers we were!

Our customers were very unhappy.

Example 3: Defects as non-conformance to requirements

Let the nitpicking season start
Let the nitpicking season start

In the same environment as Example 2, testers, in order to keep their Defect Detection Index high used to raise large amounts of minor or non-significant “defects” that were in reality non-conformance to requirements. Funnily enough such non-conformances  were generally improvements.

 

Testers didn’t care if they were requirements, code defects or even improvements, to them they were money, so they opened them. Improvements were filed as defects as they were in non-conformance to requirements. In most of the cases, these were considered to be low severity and hence low priority defects to make the testers happy and had to be filed, reviewed, prioritised and used in trends, metrics and other useless calculations.

This activity could easily take 30% of the tester time. Such defects would not only take testers’s time, but would also affect developers, product managers, business analysts and eventually clutter the defect management tool.

Waste that creates waste, exponentially, how wonderful.

A colourful dump
A colourful dump

 Example 4: Defect charts, trends and other utter nonsense

Every week I had to prepare defect charts for management. These were extracted from our monstrous defect management tool and presented in brightly coloured useless charts. My manager got so excited at the prospect of producing useless information that she started a pet project to create charts that were more colourful than the ones I presented. She used 2 developers for 6 weeks to create this thing that was meant to wow the senior executives.

In the process of defining the requirements for wowing the big guys, she introduced a few new even more useless charts and consolidated it into an aggregating dashboard. She called it the product quality health dashboard, I secretly called it the dump.

Nobody gave a damn about the dashboard, nobody used the numbers for any reason, nobody cared that they could configure it, but my boss was extremely proud of it. A legend says that she got a big raise because of it. If you play with rubbish, then you will start measuring rubbish and eventually you will end up doing data analysis and showing a consolidated view of the rubbish you store in your code.

How can we avoid this?

1. Focus on defect prevention

Many development teams focus on delivering features fast with little consideration for defect prevention. The theory is that testers (whose time is sometimes less expensive than developers) will find the defects that will be fixed later. This approach represents a false economy; rework disrupts developers activities and harms the flow of value being delivered. There are many approaches available to development teams to reduce the amount of rework needed.

Do you want to prevent defects? You can try any combination of the below:

  1. With BDD/ATDD/Specification By Example or other test first approach, delivery teams test product owners assumptions through conversations and are more likely to produce the right feature the first time.
  2. The ability to have fast feedback loops also allows for early removal of defects, automated unit and integration tests can help developers quickly identify potential issues and remove them before they get embedded into a feature.
  3. Tight collaboration between business and delivery teams helps teams be aligned with their real business goal and reduce the amount of unnecessary features. This means less code and as a consequence less defects. Because, your best piece of code is the one you won’t have to write.
  4. Reducing complexity is very powerful in preventing defects, if we are able to break down a complex problem in many simple problems we are likely to reduce the amount of defects we introduce. Simple problems have simple solutions and simple solutions have less defects than complex ones.
  5. Good coding standards like for example limiting the length of a method to a low number of lines, setting limits on cyclomatic complexity, applying good naming conventions to help readability also have a positive impact on the number of defects produced
  6. Code reviews and pair programming greatly help reduce defects
  7. Refactoring at all times also reduces defects in the long run

Moral of the story: If you don’t write defects, you will not have to fix them.

2. Fix defects immediately and burn defect management tools

If like me years back, you are getting tired of filing, categorising, discussing, reporting, ordering defects I have a very quick solution. Fix the defects as soon as you find them.

It is normal for a developer to fix a defect he finds in the code he is writing as soon as he finds it without having to log it, but as soon as the defect is found by a different individual (a tester for example) then apparently we need to start a strict logging process. Why? No idea really. People sometimes say: “if you don’t do root cause analysis you don’t know what you are doing, hence you need to file the defects”, but in reality nobody stops you from doing root cause analysis when you find the defect if you really want.

What I am suggesting is that whoever finds a bug walks to a developer responsible for the code and has a conversation. The consequence of that conversation (that in some cases can involve also a product owner) should be let’s fix it now or let’s forget about it forever.

Fixing it now, means normally that the developer is fresh on the specific code that needs to be fixed, surely fresher than in 4 weeks, when he won’t even remember he ever wrote that code. Fixing it now means that the issue is gone and we don’t have to worry about it any longer, our customer will be thankful.

Forgetting about it forever means that it is not an issue worth fixing, probably it doesn’t threaten the value of the project and the customer won’t care if we don’t fix it. Forgetting about it forever also means that we won’t carry a stinky dead fish in a defect management tool. We won’t have to waste time re-discussing the same dead fish forever in the future and our customers are happy we are not wasting time but working on new features. If you decide to fix it, I’d also recommend you write an automatic test for it, this will make sure that if the issue happens again you’ll know straight away.

I have encountered huge scepticism when suggesting to burn defect management tools and fix just in time. Only very few seem to think this is possible. As a matter of fact all my teams were able to do this for the last 6 years and nobody ever said, “I miss Jira and the beautiful bug charts”.

Obviously this approach is better suited for co located development teams, I haven’t tried it yet with a geographically distributed team, I suggest you give it a try and let me know how it goes.

Playing with defects waste index:

defects

Epidemic: 90% – The only places that don’t file and manage defects I have ever encountered are the places where I have worked and have changed the process. In the last couple of years, I have heard of two other places where they do something similar but that’s just about it. The world seems to have a great time in wasting money filing, categorising, reporting, trending waste.

Damaging: 100% – Using defects for people appraisal is one of the worst practices I have ever experienced in my long career, the damage can be immense. The customer becomes irrelevant and people focus on gaming the system to their benefit. Logging and managing defects is extremely wasteful as well, it requires time, energy and can among other things, endanger relationships between testers and developers. Trending and deducting release dates from defect density is plain idiotic, when with a little attention to defect prevention defects would be so rare that trends would not exist.

Resistant: 90% – I had to leave one company because I dared doubt the defect management gospel and like an heretic I was virtually burned at the stake. In the second company I tried to remove defect management tools I was successful after 2 years of trying, quite resistant. The third one is the one where people were happy to experiment and as soon as they saw how much waste we were removing it quickly became the new rule. I have had numerous discussions with people on the subject and the general position is that defect management must be done through a tool and following a rigid process.

Because “that’s the way we do things here!”

Recommended Reading

Lean Software development – An Agile Toolkit (Mary and Tom Poppendieck)

https://mysoftwarequality.wordpress.com/2015/05/06/little-tim-and-the-messy-house/

https://mysoftwarequality.wordpress.com/2013/09/10/how-i-stopped-logging-bugs-and-started-living-happy/

This is a reviewed and improved version of an article I first wrote in 2015 (old version here)

Advertisements

38 thoughts on “Ultimate Guide to Reducing the Amount of Defects and other Waste in your Product

  1. Preaching to the converted? I hope we can get some more converts.
    It never ceases to amaze me that rather than solve a problem many folk want to engage a tool to increase the problem in the belief that they are actually making an improvement. Ah, the power of defects!

  2. As Augusto said: Defects are waste. Who wants to produce waste?

    At test conferences I ask the audience: “How many defects are requested in the requirements?” I assume developers are doing their best to implement the requirements and apparently there is a certain number of defects specified. Otherwise, why should they implement defects?
    The I ask the testers: “Do you check whether the developers implemented the required defects properly?”
    You can imagine their puzzled faces.

  3. Great article Augusto and I agree with all your points. Playing devils advocate here: “fix it now” makes a lot of sense. But what happens if a developer is unable to fix it now? Maybe they need to finish what they are currently working on – or it’s at the end of the day, and they have a days holiday tomorrow?

    In these situations, it makes sense to record information about the bug *somewhere* so its not forgotten – and if we need to do that (because the dev can’t fix it straight away), then why not in the ALM tool (if that is being used)?

    • Hi Simon, thanks for your feedback, it is really appreciated.
      On your questions, in the first case, what if the developer is working on something new?

      I have resolved this problem in different ways, but the best results I have obtained when preventing the problem altogether, how, you might ask.
      The way we did it was by having the developer stay on the card until it was completely tested. This meant that the developer would support the
      tester or whoever happened to test the card by pairing. The rule we adopted in a team was, you are not finished on a card until it’s released to the customers.

      In the second case, if I was the tester that found the bug, and the developer responsible was away, I would simply stick a red post it on the card that refers to the story and talk to the developer in the morning when he is back.
      I hope I answered the questions, if not, just shout 🙂

      • Thanks! Some more devils advocate questions 🙂 So the red post-it is blank and just a reminder to talk? Or does it have some info on the problem on it? If the latter, then you have, in effect, logged a bug 🙂

        If the former, what happens if, for some emergency reason, you need to be off work for a few days? No-one would know what that red post-it means…

        We can agree this would be unlikely but we would probably also agree that unlikely thing like this happen at the damnedest of times!

        • If I know I can talk to the developer tomorrow, I normally would write one or 2 words on the red sticker so that if that night I go into town to drink and in the morning I have an amnesia, those 2 words can remind me what the bug was about. As you said, the red sticker is a reminder that we need to have a conversation.

          If I have to be out for a few days, I will call any of my colleagues that are still in work, show him/her the bug and tell him/her to speak to the developer in the morning.

          We can probably find another 50 unlikely scenarios that might require writing some more information. None of these scenarios in my humble opinion creates the need for adding a tool like a defect management system. I have been doing this for over 5 years now and none of my teams decided to go back logging bugs, ever.

          In the worst case scenario, I could write an automated check that goes red because of the bug, in that case I am 100% sure that we will not forget it because unless the bug is fixed, the check won’t go green.

  4. Another nice article Gus! I agree with the premise of what you are saying, but I tend to say it a different way…
    Testers who love bugs don’t understand testing IMO. It’s not about bugs. Bugs are a bi-product of software development. It’s about the product and value. (I also use the same definitions btw!).

    For me, we can try to prevent defects, but I’ve cone to realize that “preventing” a defect tends to just mean that we are discovering the defect earlier than the product…
    And even at that, it’s impossible to think of everything and ask every possible question to uncover “all the bugs” earlier. There will always be bugs in the product. Whether we care about them or not. So what this becomes for me is a conversation of resolving the bugs. Do we resolve any bugs as soon as they are found and spoken about? If Yes, cool! No need to waste time logging, “managing”, etc…
    If no, then it might be wise to store that info of that bug somewhere IF ITS AN IMPORTANT BUG THAT WE CARE ABOUT. (I.e. A bug tool for the purpose of remaining aware… Not for process purposes).

    Using defect metrics for appraisals or any other purpose is dumb test management. Again, these people don’t know what testing is IMO. I’m not sure that’s 90% of testers though… And I’ve also seen dev managers want to use bug stats for developer appraisals too…

    • Thanks for your feedback Dan, as usual, it is very insightful and thought-provoking
      I am not going to embark on the discussion around prevention over detecting early, I have done it before and the results for my learning have been quite poor 🙂
      I am pretty sure you will agree with me that preventing or detecting earlier is positive for a product, there is a massive amount of rework (waste) that gets eliminated. Having stated that, let’s look at the rest 🙂

      You say “it’s impossible to think of everything and ask every possible question to uncover “all the bugs” earlier” I agree, and the same applies to the questions you ask the product when you are testing it. I am not by any means saying that I will discover all the defects before, what I am saying is that if I do some work before I will reduce the amount of rework, no more, no less.

      Believe me, you can prevent/discover a lot of bugs before and save yourself a lot of rework in the process!

      On storing important bugs. Why not if they are important information about the system and you will get value from it, I have no problem with that. But do you need a bug management tool for that? Can you use a wiki? How many million important bugs will you find to justify a tool? Are they really that important? Should you try to prevent some? and so on

      Thanks again for your feedback!

      • Hey! Yeah. We’re definitely on the same page with all this stuff. Was just throwing out the different angle of how I talk about it. 🙂

        E.g. I kinda get the feel that you’re implicitly talking about “bad tester mindsets” when it comes to bugs and tools. For me it can also be about bad team mindsets of testing early, communication, awareness, etc too. But you’re also covering that in the post too I guess. 🙂

        We’re definitely preventing waste by testing earlier. I’m a huge advocate of that.
        And hell yes! We can use a wiki page, or even use post it notes on a physical board. But some people might still call this a “bug board” or a “bug tracker”… And some people might want a tool like Jira (or any other wider information repository), so they have trace-ability with bugs being linked to stories along with any other artifact that is relevant to that piece of work.
        But again, I agree with you… If you are using a big tool like that just for logging bugs, then that itself is waste (of money and effort!).

        • Thanks for taking the time to clarify Dan, I now see better where we stand 🙂
          I know that many teams are not yet in the conditions to try going with my approach, I am fully aware of that. The thing that annoys me is that some things (like bug management) have become the law. All I am trying to say is, “look it’s not a law, some people have eliminated it, removed waste, and the world didn’t implode, maybe you could give it a try”

  5. ohh geez!What an article. I feel disappointed at times while reading this because I too had a lot of nightmares while preparing waste charts for my seniors. Crush all the defect management tools and let’s try to avoid them getting introduced to the system.

  6. Nice post, Noticed you duplicated the text “What I am suggesting is that whoever finds a bug walks to a developer responsible for the code and has a conversation. The consequence of that conversation (that in some cases can involve also a product owner) should be”. Feel free to delete this comment after fixing.. if you care to fix!

  7. My definition of defect is: The cause of a problem for the users. This implies that the so-called defects in those parts of the sw that is never used aren’t defects. There is only one defects: that those parts were made.

    The main purpose of testing is to provide feedback to the developers to learn how to even better prevent defects. A by-product is the repair of the defects found.

    If there are 100 defects and testers find 50 (testers are also human, I think), then perhaps 40 of those 50 are repaired properly. This means that there are still 60 defects remaining. Is that significant improvement?

    Testing is taking a sample. ‘Repairing’ what has been found in the sample doesn’t mean the sw is clean now.

    • Niels, I just wanted to add that I hope you don’t feel short-changed by my answers to your comments. I really appreciate your feedback, the reason why I give more detailed answers to other commenters than to you is simply because, by knowing you better than others, I know that on this subject, me and you, have an extremely similar view and practical approach.

  8. If we need a tool for tracking bugs, then we have too many.

    Recently I suggested a tester, who had some 600 bugs listed: “I want the number to go down to 14.”
    He asked: “Why 14?”
    “Well, once we’re there we don’t need the tool anymore and then we can make sure we get rid of the remaining 14.”

  9. Ah, I love this! This way of working is similar to effective mobprogramming and I think it has many positive side effects.

    1. It builds team spirit and tears down the imaginary wall between devs and testers.
    2. The speed of learning about the product is accelerated.
    3. That icky feeling of having a never ending backlog goes away.

    However, as already discussed here in the comments, sometimes due to bad timing you find yourself in situations where that magic flow of finding and fixing at once might not be possible. And as you mentioned, there are several ways to handle such situations.

    A bigger hurdle can be the fact that people in the team may dislike to become interrupted in their work at any moment when there’s suddenly a bug to fix. Any ideas on how to smoothen out that annoyance?

    • Hi David, thank you very much for your feedback, it is very much appreciated.
      I think you have answered the question about people not wanting to be interrupted by yourself.

      You mention mobbing. Think about mobbing, you never get interrupted because there is only one stream of value that gets examined at one time.

      I have not used mobbing extensively yet, but we have used similar concepts in which the developer working on a piece of work, never abandons that work until it gets to production and the customers are actively using it.

      This might mean that a developer and a tester might be pairing, likely from the very start, in some cases the pairing might start at exploratory testing. The developer is testing with the tester, so he will not be interrupted if a defect is found, he will simply pull out his code and fix it with the assistance of the tester.

      Does this make sense to you? Do you have the conditions to try it?

  10. he following phrase was repeated twice “Fixing it now, means normally that the developer is fresh on the specific code that needs to be fixed, certainly fresher than in 4 weeks when he won’t even remember he ever wrote that code.”

    • Thanks a million Luis, I have removed the repetition. I am not sure you noticed but you are the 4th person that identified repetitions in the post (thank you to all of you to help me make this better).
      After the 3rd request, I knew there was something wrong, I had been blogging for 4 years and this is the first time that an issue of this type has happened to me.

      So I thought about what I had changed since I wrote the last bug and it turns out that a plugin I am using (Grammarly) is causing the repetitions.
      I actually love Grammarly, I have been using it for 3 days but I find it very useful so I am in a conundrum, will I disable it or will I keep on using but with the extra checks, I will have to do when I review my blog posts.

      I have reproduced the issue Grammarly is definitely the culprit, I actually saw it adding extra text live while I was typing 🙂

      I am afraid I will have to disable the plugin until they fix it 😦

      Thanks again to all the testers that found the 4 repetitions!

  11. Hi Augusto,

    Nice post! I like the idea / practice …
    Can you share more details about approaches you had with 2nd company during those 2 years to remove dmt?
    “In the second company I tried to remove defect management tools I was successful after 2 years of trying, quite resistant”

    Did you try to remove the dmt from waterfall companies/teams (yes, there are still some 😔)?

    Thanks,
    bvccaneer

    • Thanks for your feedback bvccaneer, you bring back some memories with your questions 🙂
      The second company was a brand new scrum implementation, management were terrified of losing control and we had a CEO that came into the office at 4:30 am and left after midnight every day. Imagine how such behaviour impacts on a group of people that with agile are trying to show the focus on people and wants to encourage sustainable development. The fear of losing their illusion of control didn’t allow me to help them see the uselessness of the systems. It took 2 years, at the end one team I was working part of started targeting zero defects with prevention and as the improvements brought some results, removing the defect system was easy. Also back then I did not have the same experience in change and failure I have now (I failed more times than you possibly can conceive) and enabling change was a brand new and alien discipline, I knew I loved it but I didn’t know how to do it 🙂

      The waterfall company was a systematically toxic environment, the best thing that happened in my life was talking to a person that opened my eyes and made me see how I was killing my career, and my health staying there. Thank you, you know who you are anyway 🙂

  12. Nice article. I agree with all the points mentioned.
    I like the emphasis about the value of the product is ,
    by not fixing the defects, but by not writing the defects
    by not verifying the defects, but by finding the defects

  13. Great stuff Augusto, like so very often!

    I could see how this (the quest for removing bug trackers) requires extra effort, when e.g.
    1) Bugs are raised by clients and they first go through Client Services/Support before they reach the team
    2) Bugs need a lot of data for reproduction (think user accounts, environment configurations, dumps from various databases etc.)
    3) Bugs are slow to fix (days, not minutes or hours)
    4) There are other processes and procedures that slow down (think of mandatory code review for a distributed team, for example)
    5) Management doesn’t care that much about quality and prefers the team to deliver features, but the team still wants to tackle the bugs

    I am not saying those are very good reasons not to burn down bug trackers, but I am saying they do make it more difficult, because more people need to be convinced of the idea. In my experience, bug trackers are needed when working with weak product management. (They say, for example “let’s deliver this new account setup screen first and then fix some bugs from the registration, because we need to get this live for customer feedback”.) As a Scrum Master, I feel it’s my responsibility to help management learn out from thinking that we can just leave bugs to be fixed later.

    Thanks for sharing your experiences!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s