The Grand National is a National Hunt horse race held annually at Aintree Racecourse in Liverpool, England. First run in 1839, it is a handicap steeplechase over 4 miles 514 yards with horses jumping 30 fences over two laps. It is the most valuable jump race in Europe, with a prize fund of £1 million in 2017 .
In the UK and Ireland, the Grand National is the most watched race of the year. It is an amazing spectacle to watch for the excitement, the fierce competition, the danger that comes with the massive fences and obviously to see if the horse we backed wins.
Every year, I put a small bet on one of the horses, but I don’t remember ever winning in my last 20 attempts. Predicting the winner on a field of 40 horses is not a simple matter, it is indeed very complex.
So many things can happen during the race that can throw even the most experienced and knowledgeable horserace expert. Among the things that happen during the race, changing weather and track conditions, the form in which the horse woke up in the morning, the training or lack thereof he had the week preceding, the horse in front of yours falling at the fence and bringing you down with him, the amount of drinks that the jockey had the night before, how distracted he might be from other thoughts in his head, after all his wife is divorcing him, the effect that the noise from the crowd has on your horse, and just about another million random events that could happen in those 5 minutes of racing.
Predicting a winner among the 40 participants is hard
But what if we could bet while the race has already started? Would that help us with our prediction?Well, yes. We could discard the horses that have fallen at the first fence, the unseated ones and the ones that have fallen too much behind.
That’s interesting, does that mean that now our chance of backing the right horse has increased?
Yes indeed and this does not only work for horse racing, it works for product development and innovation too.
Let’s imagine an organisation with €1M budget. Based on their understanding of the market they decide to invest the full budget into delivering “my shiny project” that again according to their understanding of the market will give them a return on investment when finished of €10M.
In gambling terms they are making a $1M bet on the “my shiny project” horse at 9/1 odds.
So what do they do? They use the €1M for assembling a team getting them the necessary resources to get the work done and do the work to deliver the project. While they are delivering, the race is on, but they are not watching it, they are busy building the project. Once they are done, only then they will know if their bet was a winning one.
If things go well, they might even get better odds as return, they might make $100M instead of 10, but history has thought us that this happens only in statistically negligible cases, while most of the time the result is that we have lost most of our money if not all of it. There are more fallers than winners at the Grand National year after year; and indeed there are more losers than winners, in fact on a field of 40, 39 lose, and one wins.
What can we do to help our organisation win the bet? Is there a possible approach that gives us the edge over other companies?
Oh yes, there are ways to help you win the race, let’s look at one example.
We take our €1M budget and allocate 9% of it to bet on the product we believe will win based on our understanding of the market and we pay an extra 1% to watch the race.
How do we watch the race? We invest 1% of our budget in tracking how our customers use our product with the intent of understanding what our customer really needs.
This would equate to spending €90k on betting on one horse and €10k on a live stream of the race we are betting on.
While we are watching we find out that our original horse has fallen at the first fence, and we also observe that another horse, we never thought could be the winner, is jumping very well and is at the centre of the leading group, a very good strategic position.
Then obviously we decide to invest another €90k betting on that horse and another €10k to keep on watching the race.
You can clearly see where I am going with this analogy. By limiting the risk of the initial bet and investing in observability I am gaining a clear advantage on the people that made their big bet before the start of the race.
In product development we can do this by making sure we create customer observability in our product to have a live stream view of the customer behaviours that informs our future bets.
Systems are all around us, and inside us. Understanding the systems we live and work in is fundamental if we want to navigate the problems that they carry.
Let’s think about our body as a system. It is a pretty complex and advanced one as it went through millions of years of evolution to get to the current state. Its working is truly fascinating and even though we have been studying it for millennia, there are parts that we still don’t fully understand.
Our body, as all systems, contains feedback loops. One we are very familiar with, is our body temperature regulating system. In order for our body to work well, its temperature needs to be between 36.5 and 37.5 degrees Celsius. In extreme circumstances, if it goes under 25 degrees we can die and a rising body temperature can damage our organs very quickly.
How do we make sure that this doesn’t happen without a thermometer that measures our temperature constantly?
Our body has a very sophisticated yet simple feedback loop system that allows us to regulate the temperature to the optimal range. In fact if we go out naked in the snow, we immediately feel a cold sensation, decide to get back into the house to put on some heavy clothes and then go out again. If we are running or exercising for a prolonged period of time and our temperature raises, our system will automatically start sweating to reduce and regulate it to its optimal range.
What happens every day of our life is that our body temperature will oscillate within a range of acceptable temperatures because, through evolution, it has designed a defence system based on fast feedback loops. By continuously knowing when it’s too hot or too cold we are guaranteed we won’t die frozen or for too much heat. A slower feedback might have caused our body to freeze, imagine for example if we felt the cold only after 24 hours in the snow, but a feedback loop fast enough prevents all that.
What a great system our body is. But even such a wonderfully evolved system has its Achilles heel, in fact it can be damaged by slow feedback loops.
Let’s think for example about the effects of smoking tobacco on our body. How come we only recently found out that smoking is so dangerous for our body? Why did it take so long?
When we smoke our first cigarette, chemicals such as tar and formaldehyde penetrate the cells in the lung tissue and change them. At first our body may be able to repair the damage. With repeated exposure, normal cells that line our lungs are increasingly damaged. Over time, the damage causes cells to act abnormally and eventually cancer may develop.
The problem is in the fact that the time that will take for the cancer to develop is so long that only recently we could only start hypothesising tobacco smoke was a potential cause. You can easily imagine why. In ~30 years our body absorbs an incredible amount of chemicals coming from all sources, the food we eat, the air we breathe and the tobacco smoke we inhale. How could we have guessed? It was very difficult and only through data driven medical studies we managed to identify a clear link between tobacco smoke and cancer.
Tobacco smoking is a very recent habit in human history if we consider humans evolution and our body has not yet developed a defence from it. What happens unfortunately is that our body will alert us of the cancer (feedback loop) only when it has damaged one of our organs. At that point, unfortunately, it is often too late to repair the damage.
A slow feedback loop can often look like the this picture. No impact for a long period of time, then a massive impact all of a sudden.
Slow feedback loops are everywhere around us, look at how some policies that seemed harmless at first sight, have caused migrations, desertification and destruction, years after, look at how the excess co2 in the atmosphere is damaging our planet slowly but nevertheless unequivocally.
Not being able to predict the effects of policies and changes to a system is troubling, but how does this fit with modern organisations and product development?
The link is in the feedback loops. We have seen earlier how the fast feedback built in our body temperature regulating system has helped our body defend itself from changes in temperature for millions of years, can we learn something from it?
A system that oscillates within a healthy range, like our body temperature, is a balanced and healthy system. What produces the healthy oscillation is the speed at which the feedback is delivered. Our body senses the changes in temperature very quickly and allows for regulating it in a way that appears to us to be continuous.
Now let’s think about organisations that rather than regulate their own temperature are trying to achieve an outcome.
Let’s look at product development for example.
Traditional organisations spend quite a lot of time studying the market to identify new solutions or products that can help them achieve an outcome. This work is often done by internal experts. Such experts have been involved in delivering previous products and are naively expected to have the wisdom that is required to design the product that will achieve the goal.
A lot of work happens inside the organisation and a lot of time passes before the new product is given to the customers. The more time the idea stays within the organisation the more we build assumptions upon assumptions instead of satisfying real customer needs. After one year or two we finally have a new product on the market, but very often we realise that the outcome we had originally thought we would get is not materialising. We often are very far from the expected outcome. We are objectively surprised, sometimes astonished at the results.
Our plan didn’t survive contact with reality.
Reality indeed, this is the first time we get feedback from our customers and it’s not what we thought it would be.
Organisations rarely understand the cause of the problem and often an unsuccessful product causes blame games. Everybody is to blame, excluding the people that designed the product. “The market has changed!”, “The latest regulatory policy makes it more difficult to use!”, “The downturn of the economy killed us!” Et cetera. I even heard leaders blame the customers, saying things like “they don’t understand!”
Slow feedback loops for product development cause something like this picture. The delta can be very big in the presence of slow feedback loops.
What can we do to avoid the truly sobering experience of showing our customers the fruit of years of our work and seeing it rejected? Well, we can’t avoid it completely but we can make it much less painful if we introduce a balancing feedback loop like the one
that regulates our body temperature.
The first thing to do is to identify an outcome we want to reach over time. This outcome is for us now the ideal body temperature we want to maintain.
The second thing we need to do is to plan regular and frequent feedback loops with our customer to verify directly in the real world how we are doing against the outcome.
The third thing is to measure how we are doing against the outcome and design the next bit of functionality taking in consideration the learning from the last feedback loop.
A fast feedback loop on product development causes a behaviour like the one in the picture.
By doing this we make sure not to diverge too much from our desired outcome. We know that we will make bad decisions that might move us further away from reaching it, but we also know that through frequent feedback loops we will be able to discover our mistakes early and take actions to repair them in a cost effective way.
It looks like we can learn a lot from our body temperature regulating system and treat our customers like our body regulates our temperature, with care and continuously.
I have been using the Shift Left and even Shift Right terminology, as we say in Dublin, for donkeys years. I am not sure who I heard it from, I seem to vaguely recall a conversation with Gojko Adzic, or was it Dan North? I am getting more and more forgetful, whomever I pick I am very likely wrong, so forget about where i heard it from. and let’s move on.
I love it, it’s leaner!
I have to say I love Shift Left, it has helped me and my teams understand where we needed to take our attention to. In my mind it was never only about redistributing testing activities, but it brought great emphasis on prevention over detection, going towards a leaner testing, some people like to call #NoTesting.
I am all for prevention and I was always going to end up with the Shift Left party.
Now, reading what Janet says, I understand perfectly where she is coming from and there is no denying that the continuous infinity loops models describe the new reality much better than Shift + direction does.
Was it always like this?
Software development today is certainly non linear 100% agree.
Now correct me if I am wrong here, but waterfall is linear.
The water goes in one direction and there is no going back unless we are on some planet with different gravitational laws, which would be pretty cool to see now that i think about it 🙂
The key is “what” are you shifting from?
My take is that, maybe, in the early days, testers suffering with a bad waterfall hungover, would have understood Shift Left with more ease than if presented with the new continuous models (that in fact, didn’t exist yet).
We are improving
Think about being a waterfall tester in the mid late 200x. You are in a room and I talk to you about how to think and use Shift left. Somebody else maybe Janet, talks to you of either of the (excellent) continuous models we have today. What do you reckon you are going to do next?
If I had to guess, I’d say you’d Shift left. It is easier to understand if all you know is waterfall.
There is a left, a right and a centre in waterfall and people get it when you ask them to go left.
It was the first step to move from waterfall to agile, at least for me.
I see the continuous models as the reflection of the growing maturity of modern testers.
We are ready
We are now ready, bring the new continuous models on and let’s leave the shifting to rally drivers.
The models are good now, because testers are ready to understand, embrace and apply them.
Most of us involved in software development are familiar with the term “technical debt”.
As a quick reminder, it was introduced by Ward Cunningham to describe the phenomenon that occurs when we use code that is easy to implement in the short run instead of applying the best overall solution we have identified.
It is by definition a conscious decision to take a shortcut for short term gratification (like taking out a new credit card for a holiday) that in thew long term will cost us to spend extra time in development when improving the system (like paying interest on the credit card debt).
Until the capital is repaid in full we will always pay interest when adding features to the system as the debt creates obstacle to adding new code efficiently.
Ward suggests that when we are in a situation of rushing something out we need to be conscious that we will have to pay the capital in the future or the interests will cripple us.
Ward suggests refactoring as a solution to paying off technical debt.
I really like Ward’s vision and I want to expand the metaphor one level up.
Now if we change the word “software” in my description of technical debt above with “processe” we can define Process Debt.
Replacing we get:
Process Debt is the phenomenon that occurs when we use a processes that is easy to implement in the short run instead of applying the best overall solution we have identified.
Let’s look at one example
You notice that lately the system seems to be more unstable than usual. You know this because there are more calls from customer care and more defects get raised.
Option 1: You want to get to the bottom of the situation and believe that a root cause analysis with 5 whys could get you there. If you use this approach you will probably identify a change in your process to help you prevent some defects in the future.
Option 2: You implement a better policy for developers to select the defects to work on reducing the time the defects are in the unresolved queue while maintaining new features creation throughput relatively stable.
You know Option 1 is better in the long run because it will generate a change in the process to reduce the amount of rework. This will mean less time spent fixing and more time spent on new features that as a consequence means higher throughput and lower lead time for new features.
Option 1 requires some investment. You need to hold one or more root cause analysis sessions, identify the problem(s) experiment with solutions until you find a solution that mitigates your problem.
If you are under pressure, because the new product needs shipping and the old defects need fixing, you are likely to choose option 2.
By doing this you have introduced process debt
The capital of this debt is the lack of change in the process you could have identified if using Option 1.
Oh, yes, I forgot, change is difficult.
The worst part of it is the interest on the debt you will pay forever until you pay the capital. This interest will appear in the form of.
1. bugs keep on coming, we seem to be fixing bugs all the time
2. new features get delayed because now the queue is in the new features to be delivered
3. the lead time of any new feature is expanded by a factor X
4. the customers continue on complaining because of your defects in production
5. your product owner starts being annoyed at the amount of work spent by the development team on defects
6. developers are tired of always fixing defects
n. need I say more?
This interest will be paid for the rest of your life if you don’t fix the problem.
Months after you are covered in defects, are stressed by your customers and change job.
Is this healthy? Certainly not.
Is there a cure? Oh yes, indeed.
Continuous improvement has the same effect on process that refactoring has on software. It repays the capital of your process debt.
With small changes in the form of experiments you will be able to clearly discuss the problem that the team is having and make small tweaks continuously.
My esteemed colleague and fervid innovator Claudio Perrone presents his continuous improvement model called PopcornFlow saying
If change is hard, make it continuous
I could not agree more.
By making process change a continuous activity, we enable behaviours that will stay with teams forever and we unleash the power of people’s creativity in improving their own way of working.
Lats Sunday, after reading multiple “Agile is Dead” articles, I posted this short update on linkedIn
As you can see from the stats, in less than 7 days it has received a lot of attention (I am not an influencer and my updates generally do not receive such large feedback)
My contacts in linkedIn include a lot of Agile or Lean Coaches and as expected, initially the message got some positive comments. Soon after some agile detractors joined the conversation and made it much more interesting as generally feedback that comes from different perspectives enriches the conversation adding dimensions that sometimes cannot be expressed by a biased mind.
I noticed 3 interesting trends in the messages.
When agile is not driven by technology, agile fails
When agile is driven by technology and not the business stakeholders, agile fails
Agile is only useful to deliver something nobody wants quickly
The first 2 are extremely interesting, in fact they say exactly the opposite thing but they both come to the same conclusion “agile is dead”. I read and reread those messages and then I saw it
If agile is driven by one part of the organisation, whichever it is, and trust is not built within the whole organisation, it will fail. Do it like this and agile is dead before you even start.
If you try to own something that will change your organisation and run with it, you better make sure you share your vision, your responsibilities and your success with the rest of the organisation. How do you expect people outside your little world to want to follow you in this difficult change if they don’t know, understand, own and help you change. Agile/lean transformations are not driven by a department, they are driven by the whole.
And the result might be that you even stop talking about departments and only talk about the whole.
Now on objection #3.
Agile is only useful to deliver something nobody wants quickly
I have seen this very often and honestly makes me sad. A lot of scrum implementations have a Product Owner that is seen as the heart of the product, the person that understands the vision of the product and that takes the responsibility to take the important decisions for the future of the product in regards to strategy, prioritization and so on.
If you look at it this way, you might think that the PO is a single point of failure, in fact what if he is not able to make good decisions, how about his bias, is he a dictator?
As an agile coach I make sure that any product owner that works with me will have the tools for making good decisions. He in fact will know how to manage flow using WIP limits, he will be aware and become proficient in UX techniques, he will learn how to monitor, gather and use feedback from his customers, he will understand the importance of small experiments, he will be aware of cost of delay and when prioritising his features and user stories will have access to many advanced prioritization techniques.
Being agile does not mean automatically ignoring lean startup, lean UX, research. No that is not being agile, that is being a scrum master after 2 days training.
The owner of a window shades company, was asked by a business consultant “What business are you in?” He initially answered “We are in the window shade business”. Then he was asked “When someone walks into your store, why do they want a window shade? What are you really selling?” he had to pause and think before he saw it
“We’re in the light-control and privacy business! – not the window shade business.”
The implications of this discovery were that by understanding the company real purpose, he was able to introduce new products that his customers loved.
Reading this story reminds me of the struggle of the testing community to get buy in from business owners. Years and years of hearing complaints that business owners don’t understand the importance of testing, they don’t appreciate the hard work of testers, I have grown tired of it.
I believe it’s not the business owners not to understand testers, I believe it is testers not understanding the business they are in.
The majority of the testers I know believe that they are either in
“The business of finding bugs”
“The business of sourcing information to help make decisions”
The testers I want to work with are in “the business of delighting their customers”, that is what we all need to be in.
If we make our business “delighting our customers” believe me, we will find a lot of innovative ways to do it as testers.
Forget about defects, forget about information, focus on delighting your customers and
you will find innovative ideas to improve the quality of your product
the business owners will love your work – and most of all
Previously on “A knowledge worker’s tale“: Team X discovered that lack of clarity on what to build, was creating a backlog in their exploratory testing work queue. They realised that adding new testers would have only fixed the symptom of the problem but not addressed its root cause.
Team X met to talk about a possible solution to the fact that too many misunderstandings in the user stories were slowing down the work dramatically and driving the testers crazy.
When Gus walked in, they were all there, nobody was late, the team really wanted to resolve this problem.
Mike took charge and said “We need to ensure that the user stories contain all the details, we can’t be blamed for problems in the requirements”. All the others nodded in unison, the bloody user stories were the problem…
Peter added “Absolutely Mike, we need Fritz to sign off the user stories so that when we start, if there is a problem, we know who’s fault it is and we don’t get the blame every time, I am tired of this!” he added “we need to make sure we know who’s fault it is”.
Before Fritz could answer, Gus interjected “OK, let’s see. Am I right in saying that our goal is to deliver value to our customers? Right. How is identifying who to blame going to help our customers? How is blaming somebody going to help our customers at all?”
He continued “Customers will still get the product late because of the misunderstandings, I bet if you asked them, they will tell you that they don’t give a rats arse who’s fault it is. What they care about is to get a solution to their problems, through our software”
Then he went “Once there was a little man in Japan, many years ago. He said that every activity that does not benefit the final customer is waste and needs to be removed from the process. He went to define 7 categories of waste that can be found in manufacturing. This guy’s name was Taiichi Ohno, he revolutionised the motor industry and his learnings have been used to improve manufacturing all over the world, he worked at Toyota. I strongly believe his lessons can be used very much in every context, including software development, let’s not introduce waste, let’s focus on value add activities”
“It seems reasonable” said Mike “but how do we make sure the defects in our stories don’t get caught only at exploratory testing, we have seen that when that happens our flow gets to a standstill, we can’t allow that”
“Taiichi would be proud of you Mike” said Gus. “One of his 7 categories of waste was exactly what you described, ‘defects’.” he continued “Quality was fundamental in the Toyota production line, one tenth of a millimetre difference in a bolt could bring the line to standstill, Taiichi knew it and made sure the workers knew it too so that they could find new ways of avoiding it”
Gus stood up and went to the whiteboard, he wrote “Prevention over Detection”. Then he drew a circle that had 3 stages “1. Write a failing test” – “2. Write enough code for the test to pass” – “3. Refactor”.
He went “Ladies and gentlemen, this is TDD, the one most effective ways of preventing defects in your code I have found in my 20 years as an engineer. In one of it’s more modern evolutions it is called BDD. Through high collaboration and conversations it allows teams to deliver code that has a fraction of the defects of code written without it. If you like to know more, I can organise a one day workshop to talk about BDD and how it can help us prevent defects”
Peter said “I am a bit confused about all this stuff, little Japanese people building cars and tests that are written before code, it doesn’t make sense. But in all fairness Gus has been right before with even more nonsensical stuff like ‘do less to do more’, I think we could take a leap of faith and trust something good will come out of this, what do you think guys?”
The team nodded in silence, they were puzzled. Accepting something brand new and counterintuitive can be scary, but true agile teams are courageous in their decisions.
Gus smiled and left the room so that the guys could make a decision freely.
Will Gus’s nonsense help team X? Stay tuned and discover what happens next
I took my first steps in technology over 20 years ago. If we exclude the year off I took in 2006 I have always been employed in many different companies and I can say with certainty that I worked on a “shitload of products” and I use that specific term with purpose, read on to find out.
During all these years, being somebody that loves learning, I ended up working as system analyst, developer, tester, business analyst, manager, leader, coach, change agent, plus some other short term hats.
If I exclude the last 6-7 years where I had the skills and the ability to influence decisions I can go back and say for sure that the products that were initially envisioned, before any customer feedback was used, can be called a shitload of useless stuff and one or two good ideas that resolve real customer problems.
Very often the product envisioned was very similar to the product that we delivered.
Am I saying that in my first 13 years I worked I mainly produced waste?
Pretty much YES.
Another thing I remember in those early years was that I was never in a team where we could say, oh thank god we are busy but it’s not too bad, we can do our work, go home and have a balanced life. Invariantly there was pressure. We need all this by that date, come on! Work faster!
To me, it always felt like we were told by Dilbert’s boss that if we shove some more paper in the printer it will print faster. Also, when we were not busy, then managers will fill our capacity with a new shitload of useless products created for the purpose to make people sweat, very often no thought on the customer whatsoever.
Am I saying that trying to deliver fixed scope, fixed date shitloads of products creates big problems to the workers that build them?
YES, not even the pretty much is needed this time.
Another thing that I have noticed through the years is that without exception, companies older than 3 years have already built a shitload of products that are now impeding their ability to respond to change and survive. We will call this “shitload of legacy systems”.
This specific shitload is used as the excuse for not being able to change, as if saying “yes sure, we can’t compete with the market and we will die soon, but it’s not our fault it’s the fault of the legacy system (that by the way we built)
Am I saying that the shitload of products are also causing the slow death of the companies that created it in the first place?
Next time you start a product, think twice before rewarding people for the delivery of all the scope, in time.
I have been helping organisations deliver products that matter to their customer as soon as possible, I am not in the business of delivering projects or shitloads of products.
I am researching new ways of demonstrating THE VALUE in MONEYof the “non built shitload of products and features”, if you are interested, let’s do this together.
Previously on A Knowledge Worker’s Tale: A large bottleneck appeared in the exploratory test column of team X’s kanban board, but only Rick and Jane, the testers, seemed to give a damn about it working hard to unblock it. Gus saw the problem asked the team to take off their headphones, go to a room and get instructions from the testers on how to unblock the flow of work.
And the story continues…
Team Xers reappeared in their open space with renewed energy. Rick and Jane had explained what they needed to get their work unblocked and soon people started working in pairs. Rick and Peter were debugging together an issue that seemed difficult to track with the assistance of Fritz the business analyst that seemed positive he remembered something important that could help them. Jane was showing to Mike and Jimmy how, what she thought was the correct customer journey, was not what Zach the customer thought it was.
It was great to watch, a group of skilled professionals working in groups, using all their skills and knowledge to troubleshoot a complex problem.
As expected, the pairing and collaboration gave its fruits, defects were getting fixed and retested and the bottleneck was reducing fast. Flow was being reestablished thanks to the team swarming and focusing on the biggest problem in the system, it was great to observe.
Once things got back to normal Mike called the team to the board and said “Folks, it has taken only 3 hours to fix that bottleneck that had basically stopped our flow for two days, I think we have learned a very big lesson today and the lesson is that when we see a bottleneck, we all have to chip in and help unblock it!”. The other guys nodded with satisfied faces finally seeing their work getting to completion again. “Well done all!” Rick the tester added “without your help, we would still be where we were 3 hours ago, collaborating has been fantastic” and the lads gave themselves a small round of applause.
“Let’s go back to work!” said Mike.
“Hang on, hang on” intervened Gus, that had been lurking from distance. “It is great that you guys found a way to reestablish flow, I was delighted to see how you collaborated and forgot about your roles for the good of the customer, I say well done for that”. He continued “The fact is, if we go back to work without addressing the problem that created the bottleneck in the first place, it will be only a matter of time before it happens again, don’t you agree? Now that we have it fresh in mind, I suggest we try to understand the real problem that created the bottleneck, let’s do a quick retro, who’s with me?” The team nodded and followed Gus in the retrospective room.
“So WHY do you guys think the bottleneck formed at exploratory testing?” asked Gus. Immediately Peter jumped in “Because we need more testers!” to which some of the other team members nodded in unison. He added, “Rick and Jane are always super busy, they never have a minute free and at the same time they are always behind with work, the guys need help, we need more testers!”
“Adding more testers is a potential solution, but we haven’t identified the problem yet” said Gus. “I am going to ask a tester now, Jane, WHY do you think the bottleneck formed at exploratory testing?” Jane, slightly caught by surprise turns to Rick, mutters something, then says “Because exploratory testing was taking too long”. “Ok this sounds like a reasonable problem, and WHY was it taking too long?” added Gus. “Well, running tests would have been fast, but we found a lot of defects that needed fixing and that created a never ending back and forth with the developers and Zach” added Jane.
“Very good Jane, thank you, we are getting somewhere now” said Gus, and continued “so it appears we had too many defects in the stories you were testing, so WHY were we having so many defects?”. A long silence ensued.
Peter broke the silence and said “For my user story, it turns out I didn’t understand clearly what Zach (the customer) wanted, so I had to change it four times”. Jimmy added “Yes, the same for me, this feature is complex and the work me and Mike were doing had to be changed many times, every time Jane or Rick would find something wrong because they’d ask Zach and he changed his mind, yet again”
Gus smiled and went “Oh Oh Oh, I think we might be getting somewhere here folks, what do you think Mike?” Mike was deep in thought, head down. He stood up and said loudly “I get it! It looks like a communication problem early in the process presented itself as a visible problem in exploratory testing. It’s got nothing to do with testing!” He was truly excited by the revelation, then he added “The root cause of the problem is the bad communication between the team and the customer and the symptom shows up in exploratory testing.”
Gus said “That’s an excellent observation Mike, do you guys think that if you clarified your doubts with the customer before you started working you would have saved the time you have spent reworking?” Everybody in unison went “Yeah!”. Well, now we know what the real problem is, I’ll chat to you tomorrow morning at the stand-up and we’ll find a solution together, it’s getting late now. Great work everybody and well done for getting to the root of the problem with only four why’s, I normally need five…”
Gus stood up to leave but stopped mid-track and asked “Guys, so do we still think we need more testers?” Peter was first to answer “No Gus, you’re right that was a knee-jerk reaction, thinking about it ,we need more clarity, not testers”
Asking why enough times to get to the root cause of a problem is a great technique, thought Gus while leaving the office and heading to the pub.
A defect is anything that threatens the value of the product.
Before we start, let’s agree that:
we don’t want defects that threaten the value of our product
we want to give our customers as much value as possible at all times.
If you don’t agree with 1 and 2 then, don’t waste your time and stop reading now.
Testers are normally associated with finding defects. Some testers get very protective with the defects they find and some developers can be very defensive about the defects they wrote. Customers don’t like defects, developers don’t like defects, product managers don’t like defects, let’s be honest, nobody likes defects besides some testers.
Why would that be? The reason is that the focus of a lot of testers is on detecting defects, and that’s what they get paid for in a lot of organisations. If you are a tester and love your defects, you might find this article disturbing, if you decide to proceed, do so at your own peril.
Defects are waste
Let’s be clear from the start: defects are waste. Waste of time in designing defective products, waste of time in coding defective routines, waste of time in detecting them, waste of time in fixing them, waste of time in re-checking them. Even writing this sentence took a good while, now think how much time it takes you to produce, detect, fix, recheck defects.
Our industry has developed a defect coping mechanism that we call defect management. It is based on a workflow of detecting => fixing => retesting. Throughout the years it has become best practice (sic) to have defect management tools and to log and track defects. Defect management approaches are generally cumbersome, slow, costly and tend to annoy people no matter whether you are a tester that gets his defect rejected, you are a developer that gets a by design feature flagged as defect, or a product manager that needs to spend time prioritising, charting and trending waste.
Another dangerous characteristic of defects is that they can be easily counted and you will always find a pointy haired manager that decides he is going to shed light on the health of his product and on the efficiency of his team by counting and drawing colorful waste charts.
But, if we agree that defects are waste, why are we logging and tracking waste, creating waste charts seems even more ridiculous, wouldn’t it be easier to try to prevent them?
Oh, if only we could write the right thing first and reduce the number of defects we produce! I say we can, be patient and read on.
Software development teams have found many ways of being creative playing with defects, see some examples below.
Example 1:Reward waste
Some years back I was working on a business critical project in one of 5 scrum teams . Let me clarify first, that our scrum implementation was at best poor, we didn’t release every sprint and our definition of done was questionable.
Close to an important release, we found ourselves in a situation where we needed to fix a lot of defects before going into production. We had 2 weeks and our teams had collectively around 100 defects to go through. Our CTO was very supportive of the defect killing initiative and he was eager to deliver with zero defects. He put in place a plan that included free food all day and night and some pampering for the developers that needed to focus 100% on defect resolution. Then he decided to give a prize to the team that would fix the highest amount of defects.
I remember feeling frightened of the possible future consequences of this reward. I spoke to the CTO and told him that I would have liked more a prize for the team that produced the lowest amount of defects rather than the one that fixed the most. Our CTO was a smart guy and understood the value proposition of my objection, he changed his approach and spoke to the teams on how not introducing defects in the first place is much more efficient than fixing them after they have been coded. Soon after the release, we started applying an approach that focussed on preventing defects rather than fixating on detection. We never had the problem of fixing 100 bugs in 2 weeks again.
Example 2:Defects metrics
In my previous waterfall life, I remember when management introduced a performance metric directly linked to defects. Testers were to be judged on the Defect Detection Index calculated as (Number of Defects detected during testing / Total number of Defects detected including production)*100. An index lower than 90 would mean nobody in the test team would get a bonus. Developers were individually judged on the number of defects found in their code by the testers and business analysts were individually judged on the number of defects found by the testers in their requirements.
Welcome to the battlefield!
The bug prioritisation meetings were battles where development managers argued any bug was a missed requirement, product managers argued that every bug was a coding error or a tester misunderstanding and the test lead (me) was simply shouted at and criticised for allowing his testers to go beyond the requirements and make use of their intellectual functions outside a scripted validation routine.
Going to that meeting was a nightmare, people completely forgot about our customers and simply wanted to get their metrics right. The amount of time we wasted arguing and defending our bonuses was astonishing. Our customers were normally unhappy because instead of focusing on value delivery we focussed on playing with defects, what a bunch of losers we were!
Our customers were very unhappy.
Example 3:Defects as non-conformance to requirements
In the same environment as Example 2, testers, in order to keep their Defect Detection Index high used to raise large amounts of minor or non-significant “defects” that were in reality non-conformance to requirements. Funnily enough such non-conformances were generally improvements.
Testers didn’t care if they were requirements, code defects or even improvements, to them they were money, so they opened them. Improvements were filed as defects as they were in non-conformance to requirements. In most of the cases, these were considered to be low severity and hence low priority defects to make the testers happy and had to be filed, reviewed, prioritised and used in trends, metrics and other useless calculations.
This activity could easily take 30% of the tester time. Such defects would not only take testers’s time, but would also affect developers, product managers, business analysts and eventually clutter the defect management tool.
Waste that creates waste, exponentially, how wonderful.
Example 4:Defect charts, trends and other utter nonsense
Every week I had to prepare defect charts for management. These were extracted from our monstrous defect management tool and presented in brightly coloured useless charts. My manager got so excited at the prospect of producing useless information that she started a pet project to create charts that were more colourful than the ones I presented. She used 2 developers for 6 weeks to create this thing that was meant to wow the senior executives.
In the process of defining the requirements for wowing the big guys, she introduced a few new even more useless charts and consolidated it into an aggregating dashboard. She called it the product quality health dashboard, I secretly called it the dump.
Nobody gave a damn about the dashboard, nobody used the numbers for any reason, nobody cared that they could configure it, but my boss was extremely proud of it. A legend says that she got a big raise because of it. If you play with rubbish, then you will start measuring rubbish and eventually you will end up doing data analysis and showing a consolidated view of the rubbish you store in your code.
How can we avoid this?
1. Focus on defect prevention
Many development teams focus on delivering features fast with little consideration for defect prevention. The theory is that testers (whose time is sometimes less expensive than developers) will find the defects that will be fixed later. This approach represents a false economy; rework disrupts developers activities and harms the flow of value being delivered. There are many approaches available to development teams to reduce the amount of rework needed.
Do you want to prevent defects? You can try any combination of the below:
With BDD/ATDD/Specification By Example or other test first approach, delivery teams test product owners assumptions through conversations and are more likely to produce the right feature the first time.
The ability to have fast feedback loops also allows for early removal of defects, automated unit and integration tests can help developers quickly identify potential issues and remove them before they get embedded into a feature.
Tight collaboration between business and delivery teams helps teams be aligned with their real business goal and reduce the amount of unnecessary features. This means less code and as a consequence less defects. Because, your best piece of code is the one you won’t have to write.
Reducing complexity is very powerful in preventing defects, if we are able to break down a complex problem in many simple problems we are likely to reduce the amount of defects we introduce. Simple problems have simple solutions and simple solutions have less defects than complex ones.
Good coding standards like for example limiting the length of a method to a low number of lines, setting limits on cyclomatic complexity, applying good naming conventions to help readability also have a positive impact on the number of defects produced
Code reviews and pair programming greatly help reduce defects
Refactoring at all times also reduces defects in the long run
Moral of the story: If you don’t write defects, you will not have to fix them.
2. Fix defects immediately and burn defect management tools
If like me years back, you are getting tired of filing, categorising, discussing, reporting, ordering defects I have a very quick solution. Fix the defects as soon as you find them.
It is normal for a developer to fix a defect he finds in the code he is writing as soon as he finds it without having to log it, but as soon as the defect is found by a different individual (a tester for example) then apparently we need to start a strict logging process. Why? No idea really. People sometimes say: “if you don’t do root cause analysis you don’t know what you are doing, hence you need to file the defects”, but in reality nobody stops you from doing root cause analysis when you find the defect if you really want.
What I am suggesting is that whoever finds a bug walks to a developer responsible for the code and has a conversation. The consequence of that conversation (that in some cases can involve also a product owner) should be let’s fix it now or let’s forget about it forever.
Fixing it now, means normally that the developer is fresh on the specific code that needs to be fixed, surely fresher than in 4 weeks, when he won’t even remember he ever wrote that code. Fixing it now means that the issue is gone and we don’t have to worry about it any longer, our customer will be thankful.
Forgetting about it forever means that it is not an issue worth fixing, probably it doesn’t threaten the value of the project and the customer won’t care if we don’t fix it. Forgetting about it forever also means that we won’t carry a stinky dead fish in a defect management tool. We won’t have to waste time re-discussing the same dead fish forever in the future and our customers are happy we are not wasting time but working on new features. If you decide to fix it, I’d also recommend you write an automatic test for it, this will make sure that if the issue happens again you’ll know straight away.
I have encountered huge scepticism when suggesting to burn defect management tools and fix just in time. Only very few seem to think this is possible. As a matter of fact all my teams were able to do this for the last 6 years and nobody ever said, “I miss Jira and the beautiful bug charts”.
Obviously this approach is better suited for co located development teams, I haven’t tried it yet with a geographically distributed team, I suggest you give it a try and let me know how it goes.
Playing with defects waste index:
Epidemic: 90% – The only places that don’t file and manage defects I have ever encountered are the places where I have worked and have changed the process. In the last couple of years, I have heard of two other places where they do something similar but that’s just about it. The world seems to have a great time in wasting money filing, categorising, reporting, trending waste.
Damaging: 100% – Using defects for people appraisal is one of the worst practices I have ever experienced in my long career, the damage can be immense. The customer becomes irrelevant and people focus on gaming the system to their benefit. Logging and managing defects is extremely wasteful as well, it requires time, energy and can among other things, endanger relationships between testers and developers. Trending and deducting release dates from defect density is plain idiotic, when with a little attention to defect prevention defects would be so rare that trends would not exist.
Resistant: 90% – I had to leave one company because I dared doubt the defect management gospel and like an heretic I was virtually burned at the stake. In the second company I tried to remove defect management tools I was successful after 2 years of trying, quite resistant. The third one is the one where people were happy to experiment and as soon as they saw how much waste we were removing it quickly became the new rule. I have had numerous discussions with people on the subject and the general position is that defect management must be done through a tool and following a rigid process.