Ultimate guide to use metrics to distract your teams and destroy your company

I have had a lot of conversations, with agile people and not, around the topic of measuring success of an agile team. I have heard all sorts of metrics thrown around, from velocity, throughput, number of bugs or lack thereof, and so on.

The fact is that those metrics are completely useless, let me tell you why.

Imagine that your team in a period of 3 months has increased its velocity from 24 to 48. What does that mean? Some people will tell you they are 100% better or even 100% more successful!

I say that they are 100% better at delivering stories (assuming they didn’t game the metrics)

More than likely they work in an organisation that measure success based on the old Budget-Time-Scope paradigm.

Unfortunately in your search for speed you are sub-optimising your system and not achieving the real goal of your company.

buildeverythingandpray
A team that bases success on the Budget/Time/Scope triangle I call it BuildEverythingFastAndPray)

What is a successful team?

A team is successful if they help the organisation they serve be successful, regardless of how many story points they deliver.

Let me tell you what a successful team measures.

A successful team measures business outcomes. What are business outcomes? Let me give you some examples:

1) x% increase week to week on downloads of your mobile app
2) y% increase in signups month to month
3) z% reduction of customer support calls month to month

or any similar outcome depending on your context where x,y,z>0

Why?

Because an organisation that obtains those outcomes is normally successful.

Even more importantly, the team will continuously monitor how their actions affect the business outcome metrics they have set to achieve so that they might decide to:

  1. Stop writing that feature, we have obtained the result and any further bell and whistle wont give us ROI
  2. Do more of this, the metrics are going in the right direction but not as expected
  3. Stop doing this and do something else, this feature is not producing the results we were expecting
  4. Ah, look at what the customer is doing instead of doing what we thought he would do! Let’s help them do it in an easier way…

 

 

buildmeasurelearn.jpg
Build Measure Learn

Now compare this to delivering all the 1 zillion stories in the fixed scope at the velocity of 48 per sprint.

What’s a successful team for you?

 

What you are measuring is wrong, let me tell you why

idreamofthedaywhen0aorganisations0awillbeabletofocuson0athereal0aproblemstheyface-default

There is one measurement that is commonly accepted when examining the effectiveness of delivery teams. Lead Time.

This is a definition from Wikipiedia.

A lead time is the latency between the initiation and execution of a process. For example, the lead time between the placement of an order and delivery of a new car from a manufacturer may be anywhere from 2 weeks to 6 months.

If software delivery team “LightningFast” is able to give to its customers product X in 1 week while software delivery team “SlowAndSteady” requires 4 weeks to deliver the same product we can state without error that LightningFast is better at software delivery than SlowAndSteady. In fact it is 4x better.

(For the purpose of this article let’s consider the quality of the product delivered by the 2 teams to be equal)

At this point we are considering the starting time to be when a development team has some form of requirements to start working on and finishing time when the bits are deployed to a production environment.

As a business owner you would want to hire team LightningFast and try to avoid SlowAndSteady, I agree with you.

Let’s imagine that team “LightningFast” works for “BestProduct.com” and “SlowAndSteady” works for “WeAreTheBest.com”

l

The CTO at WeAreTheBest.com would willingly spend a lot of money for either replacing SlowAndSteady with LightningFast or coaching SlowAndSteady to lower lead time and become more like LightningFast.

But the business world is not a software delivery speed contest, it is much bigger than that.

Follow me and you’ll discover that even if you have team LightningFast in your organisation you might have bigger problems to solve.

Let’s expand a little our concept and start looking upstream (left).

Team SlowAndSteady have a very effective way of transforming their roadmap into actionable requirements and are able to complete this specific task for product X in 2 days. Team LightningFast don’t have a lot of skills on this department and spend a lot of time upfront to create the requirements. In the specific, for product X it took them 4 weeks.

So let’s calculate lead time one more time.

LigntningFast => 1 week + 4 weeks = 5 weeks

SteadyAndSlow => 4 weeks + 2 days = 4.4 weeks

Okay, does this mean that SteadyAndSlow are more effective than LightningFast? It would seem so. The starting point for the calculation of the lead time changed completely the judgement we had built over the effectiveness of the delivery teams.

This seems interesting, what’s next?

Next is a different question. The question is what is the lead time that we should be measuring? What’s the lead time that is important to our business?

The answer, unsurprisingly, is the whole. From start to finish.

Now, let’s walk backwards to see where the start is.

Next bit I would add is the time that product X took before it got prioritised and given to the delivery team to work on. In many cases, companies have an Enterprise Portfolio from which the Products get selected from when prioritising.

WeAreTheBest.com (the home of SlowAndSteady) have mastered a very good prioritisation process and priorities are continuously assessed based on market conditions, customer feedback and active monitoring so when a product becomes the highest priority for the company the process signals it clearly and they can start creating a roadmap within 1 week.

BestProduct.com (the home of LightningFast)  are struggling with prioritisation, they are not aware of how their customers use their products and have no intention of using customer insights to make decisions on what to build. They rely on the CEO that is the smartest person in the company to decide what gets prioritised. The CEO got it wrong this time and should have started working on product X last year when  it was added to the enterprise portfolio. It took 1 year for the product to get picked up.

Ok. Now things are interesting.

BestProduct.com => 1 week + 4 weeks + 1 year = 1 year and 5 weeks

WeAreTheBest.com => 4 weeks + 2 days + 1 week = 5.4 weeks

Wow, more than a year difference between the 2 companies, this is incredible!

We could go on and on going back to the moment the idea appeared in that company for the first time and add the time lag between the idea appearing and the corresponding product materialising on the Enterprise Portfolio or even go back further and research when for the first time social conditions emerged a problem to be resolved for customers that will be eventually resolved by product X.

I have seen many enterprises and many delivery teams. Don’t take my word for it, look out there, delivery teams effectiveness is equated to how good they are at doing the Requirements to production transition.

This creates a need for improvement localised to that context. A lot of money is invested and spent to resolve the last bit of the ride, while monstrous inefficiencies slow down delivery in measures orders of magnitude bigger.

the narrow focus on a sub-system is called sub-optimisation and it is a very well known concept in systems thinking and lean but it seems to have flown over the heads of most of the agile community.

Still today, a lot of agile experts focus almost exclusively on technical aspects and are not concerned with resolving the real problems in the system.

Do you want to know more about this?

Do you want to know how to identify the real problems in your system?

Give me a shout and let’s have a chat!