Testers: what’s your strategy in a continuous delivery context?

smallbatchIn my last stint as a tester from October 2012 to Jan 2014, I helped my organisation at that time moving from delivering once every month, to delivering multiple times a day.

Let me first clarify that we didn’t move to multiple deliveries per day just for the fun of it, but because we needed it.

 

Your organisation might not yet know it needs this level of agility but more than likely it will at some stage in the future.

How did this transform the role of the testers within the organisation?

Massively

The start

When I joined I found scrum teams that delivered either once a month or once every 2 months. The teams had 3 different defects management databases full with old and new defects. Testers were doing the following activities:

  1. automation (~30-50%)
  2. exploratory testing (~50-70&)

The batches were big, the exploratory sessions were long and found a lot of defects. The automation was not effective, as it was slow and unpredictable, its value was negative.

When I left

When I left, we were using kanban, delivering multiple times a day, defects were more or less a myth of the past, no defect management tool existed. Testers were doing the following activities:

  1. Three amigos BDD sessions with customers and developers
  2. Exploratory testing (1~5%) – never longer than 10 minutes per card, more often than not reporting no defects
  3. Pairing with developers
  4. Coaching developers on testing
  5. Writing automation (0%)
  6. Talking to the customer and the team
  7. Improving the system
  8. Designing the product with the team and the customer
  9. Helping define what to monitor in production
  10. Any other valuable activity the team needed them to do

As you can see the activities that before occupied 100% of testers time, now occupy from 1 to 5% of testers time.

Were testers busy before? Yes, absolutely

Were testers busy after? Yes, absolutely

Were testers complaining because they weren’t doing automation or enough exploratory testing? No, believe me. Most testers I worked with saw the new activities in the role as a learning activity and an opportunity to broaden their skills and become more valuable to any company.

If a tester didn’t want to adapt to the new reality and embrace the change and new ways of doing things, he would have been busy for 10 minutes  a day (~2%) and he would have not been useful to the team.

Did we get there with the touch of a magic wand? No, the end stage was the result of many experiments. It was, back then, a good recipe for that context at that time (it is continuously changing)

So, tester, what’s your strategy for working in a company that releases multiple times a day?

 

Limiting WIP should wreck your head

Previously on “A knowledge worker’s tale”: team X were able to convince Zach the customer that he could not have new tickets worked on until the ones he had in progress were completed, unless the new ones were more urgent and replaced the older ones.

stop-starting-and-start-finishing-6Having the ability to talk sense into Zach with little effort felt good, the team was energised, they had a new tool that could help them talk to their customers, but the initial problem still stood, the board still contained 60 in progress tickets and it was not going to clear on its own.

They were still in front of the board after the conversation with Zach when Gus said, “guys, do you remember when I suggested we visualise the work in progress and you didn’t want to waste time doing it because you were too busy?” “Yes” said Rick the tester.

“You didn’t believe me back then, but you made an effort and trusted me. Things worked out well back then” said Gus. “Well I need you to trust me again, because what I am going to tell you now will feel very strange and some of you will immediately think that I have gone insane”

The lads looked at each other and silently agreed to listen, after all, he was right before, let’s give him another chance.

Gus started “How many people are in this team excluding me?” The lads looked around and mentally counted when Rick said “8, we are 8, 5 developers, 2 testers and 1 analyst”.

Gus said “Fine, then let’s start with a Work In Progress Limit of 5, this means that the team can only work on 5 cards at the same time, no exceptions, I mean you can’t pick up a 6th card if you haven’t finished one of the first 5, never, absolutely never. Did I say no exceptions already?”

The team looked around to each other with quizzical faces, they could not believe their ears, from the board they saw they were working on 60 items at the same time, and that was certainly too much, but 5? Five cards was ridiculous.

insane
How team X saw Gus when he said “well, let’s start with a WIP limit of 5”

Rick went “Come on Gus, are you joking? Five cards? It’s ridiculous, we’ll never finish, will we?” Jimmy added “Also, if we can work on 5 cards only there will be 3 of us that do nothing every time, how do we justify this to our customers?”

“Very good questions lads” Gus said “Let me answer them in order. Rick, you say we’ll never finish, instead I say we will finish more cards. In fact setting a WIP limit low, forces the team finishing a card instead of starting a new one. In order to start a new one, you need to finish one before that, this will force you to complete the work. Your new motto will be “Stop starting, start finishing”. Does this make sense?”

“As per Jimmy’s question, not exactly. There won’t be 3 people doing nothing at the same time, because you guys will be pairing and working together on tasks. Pairing on a card, no matter what activity you are doing, has 3 great effects, you share knowledge, the quality of the card is higher and the card gets done quicker. And remember, it is not important to be busy, it is important to be doing the right work at the right time. Stop focusing on whether you are busy or not and start focusing on the most important work to be done.”

Gus went to the board and wrote on the very top a big 5, that meant only 5 cards could be at the same time on the board NO EXCEPTIONS.

The team started to understand what that meant but still felt uneasy. Rick said “this is not going to work Gus, I am very worried about this”

“That’s perfectly OK Rick” said Gus “if you weren’t feeling uneasy I would be surprised and also worried, WIP limits are a traumatic change for a team, you will find it difficult at first and it will cause a lot of discussions within the team, even some heated ones. Believe me or not, these discussions are necessary, because you need to change the way you work so that you can start finishing and stop starting”

“You know what?” added Gus “let’s give this a try, if it doesn’t work for you, we will go back to picking up as many stories as you want, but you need to give yourself 2 weeks with this experiment, are you with me?”

“Let’s do this” the team said.

 

The magic talking board

episode2newz
This is the second episode of the adventures of team X, if you haven’t already, read the first part of the story

Things looked slightly less grim now. We had visualised the work that was coming through and at least we could see how deep the hole we had dug for ourself was.

For a team of 8 (programmers testers and analysts) we had something like 60 work items in progress, it was gigantic, daunting and scary.

But that was huge progress, because now we knew it was there. Now that we could see it and show it, we could do something to stop it from happening again.

messy-deskLittle we knew that what we had created was not only a board, what we had created was a living and talking entity, you don’t believe me? Read on.

The morning after we built the board, Zach appeared, he was the most unreasonable customer in the Northern hemisphere, known for his famous saying “take this ticket, if you finish it last week you are late”. He walked hurriedly to the team area with his usual bunch of new unbelievably urgent tickets to work on.

While he was walking, I swear, I thought I heard the music from “The good the bad and the ugly” such was the tension in the air.

Jimmy, a developer, told him: “Hey Zach, yes, we will work on them but we need to finish these first” pointing at the board.

Zach wasn’t happy, squinted and said, “but this is urgent! I need it now!”.

Jimmy was relentless, looked at the cards that Zach had already in progress on the board and said “that’s fine Zach, no problem, have a look at these cards on the board, they are your 6 tickets already in progress, if your new two are more urgent then let’s replace them”.

Zach eyes widened with disbelief, looked at the other cards and said, “but, but, but… I, I, I… need, need , need… …well no, those are more urgent, work on these new 2 only once you are done with the old ones”, he said goodbye and went back to his office.

goodbadandugly
From Top to Bottom: Jimmy, Gus and Zach

Jimmy the developer had defeated Zach, the fastest ticket-slinger in the company!

The team X guys looked at each other, high fived Jimmy and noted that the board had told the customer what they had been trying to say for the last 2 years, i.e. “we are too busy we can’t take more work”, but for some magic reason now that the work was visible, that conversation was much easier and Zach was not shouting anymore, miracle!

This small event made the team determined to always make sure that their work was visible on the board, the board was a silent ally, they won’t forget to feed it!

If Zach had been brought to reason, what else could the board help them do?

Gus said, now the fun starts, let’s talk about setting a WIP limit…

Wanna know what happened next? Is the WIP limit going to make the team even more powerful? Stay tuned for the 3rd episode to find out what Jimmy and the team X lads get up to!

The new episode is out!

What you are measuring is wrong, let me tell you why

idreamofthedaywhen0aorganisations0awillbeabletofocuson0athereal0aproblemstheyface-default

There is one measurement that is commonly accepted when examining the effectiveness of delivery teams. Lead Time.

This is a definition from Wikipiedia.

A lead time is the latency between the initiation and execution of a process. For example, the lead time between the placement of an order and delivery of a new car from a manufacturer may be anywhere from 2 weeks to 6 months.

If software delivery team “LightningFast” is able to give to its customers product X in 1 week while software delivery team “SlowAndSteady” requires 4 weeks to deliver the same product we can state without error that LightningFast is better at software delivery than SlowAndSteady. In fact it is 4x better.

(For the purpose of this article let’s consider the quality of the product delivered by the 2 teams to be equal)

At this point we are considering the starting time to be when a development team has some form of requirements to start working on and finishing time when the bits are deployed to a production environment.

As a business owner you would want to hire team LightningFast and try to avoid SlowAndSteady, I agree with you.

Let’s imagine that team “LightningFast” works for “BestProduct.com” and “SlowAndSteady” works for “WeAreTheBest.com”

l

The CTO at WeAreTheBest.com would willingly spend a lot of money for either replacing SlowAndSteady with LightningFast or coaching SlowAndSteady to lower lead time and become more like LightningFast.

But the business world is not a software delivery speed contest, it is much bigger than that.

Follow me and you’ll discover that even if you have team LightningFast in your organisation you might have bigger problems to solve.

Let’s expand a little our concept and start looking upstream (left).

Team SlowAndSteady have a very effective way of transforming their roadmap into actionable requirements and are able to complete this specific task for product X in 2 days. Team LightningFast don’t have a lot of skills on this department and spend a lot of time upfront to create the requirements. In the specific, for product X it took them 4 weeks.

So let’s calculate lead time one more time.

LigntningFast => 1 week + 4 weeks = 5 weeks

SteadyAndSlow => 4 weeks + 2 days = 4.4 weeks

Okay, does this mean that SteadyAndSlow are more effective than LightningFast? It would seem so. The starting point for the calculation of the lead time changed completely the judgement we had built over the effectiveness of the delivery teams.

This seems interesting, what’s next?

Next is a different question. The question is what is the lead time that we should be measuring? What’s the lead time that is important to our business?

The answer, unsurprisingly, is the whole. From start to finish.

Now, let’s walk backwards to see where the start is.

Next bit I would add is the time that product X took before it got prioritised and given to the delivery team to work on. In many cases, companies have an Enterprise Portfolio from which the Products get selected from when prioritising.

WeAreTheBest.com (the home of SlowAndSteady) have mastered a very good prioritisation process and priorities are continuously assessed based on market conditions, customer feedback and active monitoring so when a product becomes the highest priority for the company the process signals it clearly and they can start creating a roadmap within 1 week.

BestProduct.com (the home of LightningFast)  are struggling with prioritisation, they are not aware of how their customers use their products and have no intention of using customer insights to make decisions on what to build. They rely on the CEO that is the smartest person in the company to decide what gets prioritised. The CEO got it wrong this time and should have started working on product X last year when  it was added to the enterprise portfolio. It took 1 year for the product to get picked up.

Ok. Now things are interesting.

BestProduct.com => 1 week + 4 weeks + 1 year = 1 year and 5 weeks

WeAreTheBest.com => 4 weeks + 2 days + 1 week = 5.4 weeks

Wow, more than a year difference between the 2 companies, this is incredible!

We could go on and on going back to the moment the idea appeared in that company for the first time and add the time lag between the idea appearing and the corresponding product materialising on the Enterprise Portfolio or even go back further and research when for the first time social conditions emerged a problem to be resolved for customers that will be eventually resolved by product X.

I have seen many enterprises and many delivery teams. Don’t take my word for it, look out there, delivery teams effectiveness is equated to how good they are at doing the Requirements to production transition.

This creates a need for improvement localised to that context. A lot of money is invested and spent to resolve the last bit of the ride, while monstrous inefficiencies slow down delivery in measures orders of magnitude bigger.

the narrow focus on a sub-system is called sub-optimisation and it is a very well known concept in systems thinking and lean but it seems to have flown over the heads of most of the agile community.

Still today, a lot of agile experts focus almost exclusively on technical aspects and are not concerned with resolving the real problems in the system.

Do you want to know more about this?

Do you want to know how to identify the real problems in your system?

Give me a shout and let’s have a chat!

 

3 Simple Steps to Help Overloaded Teams

dublin-rain-491-390x285It was a wet and dark morning in Dublin when I sat with team X for the first time.

I had been told: “Gus, these guys are struggling, they really need your help. They are always late, and everybody complains about the quality of their work, it’s bad, very bad”.

I expected a demotivated team of technically poor developers with its members playing video games or youtube videos instead of doing their job, but I was wrong.

No developer was slacking or pretending to work, on the other hand, I saw people with worried faces shuffling around, some listening to angry customers, some head down into code, some testing and quietly cursing.

To a newcomer, they looked super busy and doing their thing.

I perceived the first hint of trouble later in the day. I went to talk to some of them, just introducing myself and telling them that I was going to work with them to try to make their life easier. Each and every one of them was too busy to talk. All of them had something urgent, being a production issue, being a customer waiting behind their backs, or a release they had to finish by 5pm. Sorry, sorry Gus, I really can’t now, I have this blah blah…

I took it in and let them alone for the day. The day after, I kept on observing without saying a word. The amount of pressure that I saw applied on to these poor kids was frightening. In the same day I saw 8 different people go to the same developer and demand he finished something he was meant to have finished already. The requesters were different people and asked for 8 different things, all quite angrily.

The subsequent days, my observations stressed more and more this situation, the people were pushed over the limits and by trying to finish things quickly and relief some of the pressure were skipping steps and introducing new problems, new issues to work on, and so on.

I kept on observing and asking questions here and there, but the people saw me as somebody that couldn’t help them with the code, hence quite useless. I had to do something different.

On the 4th day, in the morning, after their standup, I told them to stop doing what they were doing and come and have a chat with me.

Believe it or not, It was almost impossible to get them off their seats.

no-thanks-were-too-busyThey felt they could not leave the sinking ship, they needed to continue trying to flush the water out with their hands. Did I have a powerful drain pump and could save the boat easily? They didn’t have time to learn how to use it, they needed to keep on shovelling with their bare hands.

I eventually got them into a room. We had a retrospective. I disguised the retro to be something else, because I had heard from one of the developers: “we don’t do retrospectives anymore, what’s the point if we don’t have the time to change anything?”

That’s a very valid question. Very.

The team was obviously overloaded and didn’t have the maturity and the necessary negotiation skills to express that to their customers and stakeholders. They had reacted to the overload with what I call the “headless chicken approach”. It works like this: “Just do all you can, forget about quality, process, product, customers, just run for your life and even if your work product is terrible, people won’t be able to blame you, as you work like a donkey every day.”

fuck-it-let-s-just-panic-and-run-around-like-headless-chickensDo you think this is uncommon? Well think again.

The first step we took was to visualise the work in progress. The guys were using Jira with no physical board and didn’t realise how much work they were really taking on. The board was scary, full scary I mean.

As the work came from many different products, another thing that we did was separating the products into different classes of service.

By doing this we immediately saw that one product had most of the tickets, why? We don’t know yet, but at least we are aware of it now.
After this very simple step  the team members started having conversations that were beyond the need to finish MyNewFeature, they started looking at current priorities and talking about the whys of the bad situation they where in.

Then they started talking about experiments to fix it.

They were improving the system.

I looked outside the window, the sun had come out.

p9220189_dublin_hapenny
What were the 3 steps we made that enabled the behavioural change?

  1. Acknowledge we were in an unsustainable situation that needed changing
  2. Agree the team had the responsibility and the authority to improve it with my support and management agreement
  3. Visualise the work they were already doing

 

Just this small change had transformed a group of intelligent people acting like headless chickens into people that were trying to improve a complex system, not bad for a weeks work I’d say!

If you are curious stay tuned and I will tell you what happened next, no more chickens, I swear.

The new episode of the story of Team X is out

Can you remove a Tester and get better quality?

whaatHave you ever been in a conversation like this?

Manager: “The last release is very buggy, why didn’t we spot the issues before?”

Tester: “We did all we could with the resources and the time given to us”

Manager: “What can we do so that it doesn’t happen in the future?”

Tester: “We need more testers”

Manager: “OK let’s get Michael form the 4th floor help us from Monday”

I have been hearing this conversation for more than 20 years. I have even been the tester and the manager in these conversations.

But what happened the following release? Did Michael from the 4th floor manage to help improve the situation? Not at all, and the same would have happened if also Frank and Jane from the 5th floor helped, and so on.

One day, over 10 years ago (before I even knew what agile or lean was), I thought that insisting with the same approach would have been insane, so I started to look at what different possibilities we had and I tried a crazy move.

When challenged with the same “too many bugs in this release what do we do?” I said:

Why don’t we remove a tester instead of adding one?

People laughed at me, but I had enough influence to be able to try this.

Against all odds, things got a lot better. The one tester remaining in the team was swamped with work and developers felt they needed to help to avoid getting everything on hold.

Guess what, they started to help.

Some of the activities that were “tester zone only” before, became accessible to developers. A different perspective helped identify some inefficiencies in such activities and developers adapted some of their own process to help.

All of a sudden giving developers visibility over the test activities had triggered a lot of curiosity and interesting hypothesis. Some developers, challenged with repetitive tasks, created small tools to make them faster, some other saw how sharing the test plans with the developers could help prevent issues that otherwise would have been caught much later, and so on.

This was many years ago, it wasn’t perfect, but taught me a great lesson.

If people don’t have skin in the game they won’t improve the system. If they are affected by a problem they will resolve it instead.

If you ask your developers to write code, that’s what they will do, defects and all.

If you empower them with delivering customer value they will look beyond the code and make sure that what they deliver is also useful for their customers.

This is one of the foundations in today’s agile software development teams where role silos are gone, collaboration is fundamental and team members care about customer value rather than lines of code or number of tests executed.

We’ve come a long way

 

The art of doing 1/15th of the work and getting earlier business outcomes

imagesIf you are an agile practitioner you are likely to have read the book “Scrum – The art of doing Twice the Work in Half the Time” from Jeff Sutherland. I am a fan of Jeff and I believe that what he has done for software development in the last 20 years is great.

I do have issues with the book’s title though.

That title is what makes people think that being agile means being able to go fast and deliver more stuff.

Is this what real agility is?  Let me tell you a story.

At a client of mine a couple of years ago, I was asked to coach a product team and help them with a new product they were starting to work on. I was excited about it and asked the product owner to meet for a coffee and initial chat.

He kindly agreed and told me: “before we meet, have a look at our requirements document so you will know what the topic is”. He also told me that a high level estimation had been done and that the product would take from 6 to 8 months based on one agile team and that there would be licensing costs associated with an automated scanning system we had to acquire.

Attached to the mail there was a document of about 50 pages with detailed workflows, low fidelity prototypes of the screens and quite a lot of explaining text. I skimmed through it looking for a description of the problem that the product was meant to resolve, but couldn’t find it.

I read and reread the document and I couldn’t find the original problem that the product had to resolve, it was not there.

When I met the product owner, after agreeing the weather was miserable (that’s how we start any conversation in Dublin regardless of the season) he started describing his solution. I let him explain to me the beautiful features to build and the amazing technologies we were going to use.

When he was finished, I asked him: Why are we building this?

The initial reaction (completely normal) was a defensive stand for his solution. When I probed more, it was clear that after having worked for so long on the product on his own he had forgotten the nature of the initial problem that triggered the decision to build this product.

Using the 5 WHYs technique, in about 10 minutes, we eventually got to the initial problem that we needed to resolve.

At this point the conversation became different.

I asked what he thought were the features we should prioritise to resolve the problem so that the customer could have something earlier than in 6-8 months. That triggered the interest in the PO that identified 5 features (about 30% of the total described in the document).

I then took each feature and asked him if it was necessary to resolve the problem we just identified. After another 10 minutes we agreed that 1 single feature, that accounted for about 1/15th of the initial solution, would resolve the customers problem. To avoid making the PO feel bad about having done so much unnecessary work, I told him that we would build the other features incrementally, and that it was a great success that he had identified a single feature that would be useful for the customer almost immediately.

We then agreed to identify outcomes of success for the full product and start measuring immediately as soon as we released the first feature.

These read something like:

“We will know we have succeeded when 30% of customers do X instead of Y” or

“We will know we have succeeded when we will have 40% less support calls related to problem X”

et cetera. I made sure the business outcomes were related to the initial problem, not any product.

What ended up happening is that we built the first feature in 2 weeks (opposed to 6-8 months) and the measurements told us that we had reached what we wanted already.earlierbusinessoutcomes

We never built the other 14 features, we stopped because the business outcomes we had set were reached.

And yes, you guessed, we didn’t have to buy the licenses for the automatic scanning system either.

As a coach, my mission in organisations is to guide teams and navigate problems, maximising value with minimum work. This is always welcome when meeting CTOs or CIOs because – most of the time – less work means lower costs and earlier delivery.

We did 1/15th of the work and got business outcomes earlier, a big improvement IMHO from “doing Twice the Work in Half the Time”, what do you think?