Measuring product development performance
What do we want to achieve?
We want our product development process to achieve a fast flow of change, so that we can deliver value to our users effectively. We can define that value as solutions that solve, or contribute towards solving, problems that our users are facing. We can deliver value through the work we make available for our users, whether it’s a large piece of project work or a small piece of content or process change.
How are we going to achieve it?
We have to define what work we want to measure. We probably don’t want to measure all the work that we deliver but could start with the new work that goes through our product development process. We can decide to add other types of work later.
Based on Team Topologies approach, there are four measures that are indicators of organisational performance in delivering value to users:
- Lead time – How long does it take us to go from starting work to making it live?
- Deployment frequency – How often does work go live?
- Change fail percentage – How much work goes live that doesn’t solve the problem it set out to?
- Mean time to resolution – How quickly is work that doesn’t solve the problem fixed?
How long does it take us to go from starting work to making it live? (Lead time)
How might we measure it?
- Record start date.
- Record live date.
- Count the number of days, divide by the number of projects to get an average lead time.
What could we do with the measurements?
- Decide whether the benchmarked average lead time is where we want it to be.
- Decide whether we want to reduce lead time.
How often does work go live? (Deployment frequency)
How might we measure it?
- Record the live date of all projects.
- Count the number of days between each project, divide between the number of projects to get an average deployment frequency.
What could we do with the measurements?
- Decide whether how often we make new projects live is where we want to be.
- Decide whether we want to decrease our deployment frequency.
How much work goes live but doesn’t solve the problem it set out to? (Change fail percentage)
How might we measure it?
- Set goals and measurements during the project.
- Regularly measure live products/services against the goals.
- When a product/service drops is evidenced to not be achieving its goal, request work to fix it.
- Record the date of the request.
- Count the number of products/services against the number that are achieving their goals to get a percentage.
What could we do with the measurements?
- Decide whether the percentage of products/services going live and not achieving their goals is where we want to be.
- Decide whether we want to reduce our change fail percentage.
How quickly is work that doesn’t solve the problem fixed? (Mean time to resolution)
How might we measure it?
- Record the date of the request for work to fix.
- Record the date the new solution goes live.
- Count the number of days between them, add to the total of all projects, divide by the number of projects to get an average number of days to mean time to resolution.
What could we do with the measurements?
- Decide whether the average number of days we take to solve problems is where we want to be.
- Decide whether we want to reduce our mean time to resolution.
What behaviours can we expect these measures to drive?
Measuring, and responding to measurements always drives behaviour change in a system.
If we sought to improve on all these metrics, we can expect to see work reduce in size and scope to tackle smaller specific problems faster.