Testers: what’s your strategy in a continuous delivery context?

smallbatchIn my last stint as a tester from October 2012 to Jan 2014, I helped my organisation at that time moving from delivering once every month, to delivering multiple times a day.

Let me first clarify that we didn’t move to multiple deliveries per day just for the fun of it, but because we needed it.


Your organisation might not yet know it needs this level of agility but more than likely it will at some stage in the future.

How did this transform the role of the testers within the organisation?


The start

When I joined I found scrum teams that delivered either once a month or once every 2 months. The teams had 3 different defects management databases full with old and new defects. Testers were doing the following activities:

  1. automation (~30-50%)
  2. exploratory testing (~50-70&)

The batches were big, the exploratory sessions were long and found a lot of defects. The automation was not effective, as it was slow and unpredictable, its value was negative.

When I left

When I left, we were using kanban, delivering multiple times a day, defects were more or less a myth of the past, no defect management tool existed. Testers were doing the following activities:

  1. Three amigos BDD sessions with customers and developers
  2. Exploratory testing (1~5%) – never longer than 10 minutes per card, more often than not reporting no defects
  3. Pairing with developers
  4. Coaching developers on testing
  5. Writing automation (0%)
  6. Talking to the customer and the team
  7. Improving the system
  8. Designing the product with the team and the customer
  9. Helping define what to monitor in production
  10. Any other valuable activity the team needed them to do

As you can see the activities that before occupied 100% of testers time, now occupy from 1 to 5% of testers time.

Were testers busy before? Yes, absolutely

Were testers busy after? Yes, absolutely

Were testers complaining because they weren’t doing automation or enough exploratory testing? No, believe me. Most testers I worked with saw the new activities in the role as a learning activity and an opportunity to broaden their skills and become more valuable to any company.

If a tester didn’t want to adapt to the new reality and embrace the change and new ways of doing things, he would have been busy for 10 minutes  a day (~2%) and he would have not been useful to the team.

Did we get there with the touch of a magic wand? No, the end stage was the result of many experiments. It was, back then, a good recipe for that context at that time (it is continuously changing)

So, tester, what’s your strategy for working in a company that releases multiple times a day?


16 thoughts on “Testers: what’s your strategy in a continuous delivery context?

  1. In the end was someone else writing the automation? Were the developers writing it? Were the testers pairing with them on the automation? I guess I don’t understand what that 0% automation figure meant. Did you not do any automation any more?

  2. I’m moving to towards a similar model Gus with emphasis on building my understanding of the context(really understanding the people that matter & what they value), test first using bdd(collaborative with me implanting the code), exploratory testing followed by monitoring and automated synthetic testing in production.

    Having pointers on how you managed to successfully transition would be great.



  3. Good different perspective on testers role and the overall strategy …
    Looks like a magic, no exploratory testing, no defects(written). Automation is still done (by developers) so are we really saving bandwidth ? Would be curious to know the impact on the quality of product and time line impact cause of automation done by developers.

    • Hi Prashant, thanks for your feedback.

      You say no exploratory testing, well beyond the fact that the traditional exploratory testing is still there, even though the percentage is very low, if you look at the other activities I list you will find that they are mostly exploratory. They explore ideas, they explore assumptions, they explore the way the customer uses the product, etc.

      As I have mentioned on the linkedin connected discussion, Agile teams are not simply made by their individuals, they are one entity and act as such . Empowered teams will discover through continuous improvement what the most effective and efficient way of doing a specific task is.

      To your productivity question, when teams follow good engineering practices, systems will be easy to change and as a consequence teams will be able to release more often. The automated tests have a big role in this.
      But I have to say that, it is not about the productivity (number of lines of code) but about delivering the most important thing with high quality.
      The focus is on flow and not on being busy at all times.

      To respond to your other question, the quality is higher

  4. I find this interesting. I like it but can see a few holes. You say up to 5% (10 mins) of a testers time was spent on exploratory testing the product (or feature). Were other people doing this too (devs)?
    Also was this 10 minutes per small feature? How many features per day? (I’m assuming they perhaps went from a small number of BIG features, to a very large number of small features, so is this 10 mins per small feature?)

    If not, and it literally is 10 minutes per day, then how do you know there was a reduction in bugs, if you are only spending 10 mins per day to investigate the product (or features)?

    Playing devils advocate a bit here because I’ve seen companies try to move to models like this, but have failed as they think that exploratory testing isn’t required at all because they claim to have uncovered all the info upfront to “prevent” all the bugs at the beginning, which is impossible due to the 5 orders of ignorance. 😉

    • Hi Dan, thanks a million for your feedback, really appreciated.
      You say you see a few holes, if you didn’t you wouldn’t be the good tester you are!

      Let me answer each of your very valid questions and please let’s keep this up, I love getting my ideas tested!

      Dan: “You say up to 5% (10 mins) of a testers time was spent on exploratory testing the product (or feature). Were other people doing this too (devs)?”

      Yes, anybody who had the skills to do exploratory testing performed exploratory testing. In the first period in that context, I spent most of my time pair exploring with developers to make sure they were gaining the skills required. Once I saw that they could do it, I let them do it on their own (always in pairs) and was available for help. After a while no more hep was needed, or it was only in some specific situations that they needed some extra support.

      Dan: “Also was this 10 minutes per small feature? How many features per day? (I’m assuming they perhaps went from a small number of BIG features, to a very large number of small features, so is this 10 mins per small feature?)”

      Every story we exploratory test is a small batch, very small, so small that we would ask ourselves questions at retrospectives if we could not deliver it from story to customer in less than 3 days. There is no big feature. On a normal day a tester would run exploratory testing from 0 to 2 stories. In the same day developers might do the same.

      Dan: “If not, and it literally is 10 minutes per day, then how do you know there was a reduction in bugs, if you are only spending 10 mins per day to investigate the product (or features)?”

      We looked at the bugs in production. I remember that in 2012 before we started working in small batches, we had a release (one month batch) that had 24 production releases. In the second part of 2013 (6 months) we had 3 bugs in production on that product.

      To conclude on your devil’s advocate part, I can assure you I am very aware of the 5 orders of ignorance, I actually blogged about it a few years back 🙂
      Fact is, you can reduce ignorance in many ways, not only through exploratory testing.
      Look at the other activities that I mention in the article, do you see how many of them are exploratory activities?
      We test ideas, we test assumptions, we test customer behaviours, all of this without having to focus on testing the finished product.
      Also, look at lean startup reduces ignorance, it’s very much in line with a lot of the things I do.

      Thanks again!

      • Great response! Thanks Gus! 🙂
        Yes. All the activities will increase our awareness and will uncover unknowns. I agree.
        I think you’ve been in a great position to have people with the right mindsets, eager to learn and conduct testing activities (including the ones you mentioned in the list) from both backgrounds of devs and testers.

        Looking at bugs that crop up in production is risky IMO. If customers find even just one bug bug, they might be put off the application. And also, the software might be mission critical, so you may not be able to afford the attitude of finding bugs in prod. (e.g. if its medical or clinical trial software, or automobile driver assistance software, etc).

        My TaaS model currently still focuses on the specialist testing skills, but does emphasise collaboration and raising awareness of testing in a move from “software should meet requirements” to “lets investigate to uncover more unknowns”.
        What I’ve seen in my experience is that devs want to spend most of their time writing code, but in software teams today, out of all the activities at each time throughout the creation of the software (from idea to production), the writing code bit is a small bit of the overall picture, but gets the biggest attention. Testing (not just of the product too, but of the idea, the design, the release process and all the bits in between) is many big bits that get much less attention IMO, and the people who want to code all day have really faught back against doing these other activities. It’s different skills too, and our education system gets it wrong completely when thinking about this (developers dont learn about this stuff at uni, and testers cant even study testing at uni at all!!).

        I do like your mindset with this though. But I have one last question 🙂 …
        With all this in mind, do/did you have a period of time where you waited for your devs to really become /skilled/ exploratory testers (as in maybe a few years of experience with fluent knowledge about ET, Heuristics, Oracles and to 100+ different types of software risks that we should be investigating)?
        Or did you hand over the reigns to have the devs perform the ET straight away after only a few weeks/months of pairing?

        • Thanks again for helping me get deep dan.

          You say “you’ve been in a great position to have people with the right mindsets, eager to learn and conduct testing activities”
          I heard this many times. Other variations I have heard are: “you are lucky to be in that situation” or “you were fortunate to have the right people around you” and also “you have had the fortune of not having my situation where blah blah…” et cetera.

          I don’t know if you are a fan of golf, but legend says that back in the day the South African golfer Gary Player sank 2 consecutive long putts for 2 birdies. His playing partner sad: “Wow, you are lucky!” to which Gary replied “I am a great believer in luck, the harder I work the luckier I get”

          Honestly, I don’t think I have been lucky. Could I be lucky every time? No, I made my own luck.

          I learned to treat developers with respect, show them my appreciation for their work, empathise with their fears and expectations. At that point, developers, or for that matter any category of people, become more interested in our own needs, more willing to help.

          If we keep on saying that developers can’t test, that if there are no testers the world is going to end because developers are not able to do their job properly, how do we expect to have developers listen to us and help us? First we shut them out of our little testing world and then we expect they want to learn what we do and help us, this is not reasonable.

          I am not talking about you, but you will have to agree with me that this attitude is very much mainstream among testers.

          To answer to your last question, no it didn’t take years, it took a few months.

          Thanks again Dan, I think I am going to create a blog post for this, you are my muse today 🙂

          • Oh, I definitely agree. 🙂 I think anyone can learn to be great at testing. But my gripe is that people think that you can spend 2 weeks coaching someone and then they are experts…

            To expand: I see lots of companies moving in a similar way that you have, with the extra step of getting rid of the testers in the company. They have a 2 week handover from testers to dev, then think that’s enough for the devs to become testing experts and go it alone.

            I think that good testing skills can only come with continuous learning and great experience. Experiences can be shared for others to gain knowledge, but that individual should use the shared knowledge and put that to practice to build up their own experience and it can take years of practice to become a good tester.

            “Everyone can test, but not everyone can test well” springs to mind with what I’m trying to say.

            I’m definitely not saying it cant work if you let the devs start to control the exploratory testing, with limited learning… I think there would be an element of luck here for the project (it’s a bit like letting junior testers do all the testing IMO).

            But this gripe of mine isn’t just related to exploratory testing though. If you think of BDD or CD or even TDD… People read a blog or two and then think they know everything and start down a path that’ll cause lots of problems. IMO, people should be more stringent in studying their craft and all the things that are evolving the software world that goes along with it.

            I think this is what you are advocating too, and I hope more people take this on board!

  5. Augusto,

    The timing of this post is superb as it aligns well with what I’m currently experiencing as we’ve transitioned somewhat into Continuous Delivery. During Q1 of this year, we transitioned away from 2 week sprints and into CD using Kanban for minor releases (bug fixes, minor to moderate enhancements, etc.), but still use small waterfall releases (sprints) for new product integrations and other major enhancement initiatives. We are working towards eliminating or further minimizing these small waterfall releases in favor of CD.

    I’m interested to know… what have you done to succeed in breaking up these major initiatives into workable pieces that can be delivered via CD?

    I’ve found that to succeed with this transition to this point we’ve all been required to vastly improve communication in its frequency, in its content, in its openness, and in its compassion. IMO and IME, rigid process is the response to communication gaps and gets more rigid with more communication gaps. I’ve found that as we improve communication, we lighten the process controls naturally because fear diminishes. I’ve also found that Mentoring is a critical activity for everyone to be engaged in. I believe that we all should mentor each other in our activities and responsibilities and have observed that as we do we vastly improve product and process quality.

    As testers, we’re gradually becoming communication advocates and this has served us well.

    What experiences did you have related to communication that helped you to succeed?

    • Hi Terren, thanks a million for your feedback, it is really interesting.

      You hit the nail on the head, communication, and transparency I might add, are key to being able to make the leap.

      We started by acknowledging that we had too much rework to be able to deliver continuously and introduced BDD. This helped us prevent many of the issues that we would have found at exploratory testing. It also helped bond the team BDD is mainly a team activity.

      We also focused on having a very strong automation strategy that follows the testing pyramid pattern, with as little UI E2E tests as possible and a lot of unit and API tests.

      Communication is also helped by shared activities. Whenever possible we tried to pair on every activity from analysis to coding to testing, this helped also trust and transparency.

      Last, but certainly not least we were allowed a free to fail environment, we made loads of mistakes in the process of improving but we didn’t get scalded for trying, this is very important.

      To break the big things into workable CD items we used a vertical slicing pattern, you can have a look at it here (https://mysoftwarequality.wordpress.com/2015/04/07/vertical-slices-of-value/) with feature teams and eliminated all component teams.

      Breaking down stories in vertical slices of value is not the easiest thing and it requires practice and resilience, because at the beginning you might feel like going back to the old ways that feels easier.

      I hope I gave you some idea, if you have any other question, just shout!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s