Measuring product development performance

What do we want to achieve? 

We want our product development process to achieve a fast flow of change, so that we can deliver value to our users effectively. We can define that value as solutions that solve, or contribute towards solving, problems that our users are facing. We can deliver value through the work we make available for our users, whether it’s a large piece of project work or a small piece of content or process change. 

How are we going to achieve it? 

We have to define what work we want to measure. We probably don’t want to measure all the work that we deliver but could start with the new work that goes through our product development process. We can decide to add other types of work later. 

Based on Team Topologies approach, there are four measures that are indicators of organisational performance in delivering value to users: 

  • Lead time – How long does it take us to go from starting work to making it live? 
  • Deployment frequency – How often does work go live? 
  • Change fail percentage – How much work goes live that doesn’t solve the problem it set out to? 
  • Mean time to resolution – How quickly is work that doesn’t solve the problem fixed? 

How long does it take us to go from starting work to making it live? (Lead time) 

How might we measure it? 

  1. Record start date. 
  1. Record live date. 
  1. Count the number of days, divide by the number of projects to get an average lead time. 

What could we do with the measurements? 

  • Decide whether the benchmarked average lead time is where we want it to be. 
  • Decide whether we want to reduce lead time. 

How often does work go live? (Deployment frequency) 

How might we measure it? 

  1. Record the live date of all projects. 
  1. Count the number of days between each project, divide between the number of projects to get an average deployment frequency. 

What could we do with the measurements? 

  • Decide whether how often we make new projects live is where we want to be. 
  • Decide whether we want to decrease our deployment frequency. 

How much work goes live but doesn’t solve the problem it set out to? (Change fail percentage) 

How might we measure it? 

  1. Set goals and measurements during the project. 
  1. Regularly measure live products/services against the goals. 
  1. When a product/service drops is evidenced to not be achieving its goal, request work to fix it.  
  1. Record the date of the request. 
  1. Count the number of products/services against the number that are achieving their goals to get a percentage. 

What could we do with the measurements? 

  • Decide whether the percentage of products/services going live and not achieving their goals is where we want to be. 
  • Decide whether we want to reduce our change fail percentage. 

How quickly is work that doesn’t solve the problem fixed? (Mean time to resolution) 

How might we measure it? 

  1. Record the date of the request for work to fix. 
  1. Record the date the new solution goes live. 
  1. Count the number of days between them, add to the total of all projects, divide by the number of projects to get an average number of days to mean time to resolution. 

What could we do with the measurements? 

  • Decide whether the average number of days we take to solve problems is where we want to be. 
  • Decide whether we want to reduce our mean time to resolution. 

What behaviours can we expect these measures to drive? 

Measuring, and responding to measurements always drives behaviour change in a system.

If we sought to improve on all these metrics, we can expect to see work reduce in size and scope to tackle smaller specific problems faster. 

Retrospective April 2022

This month’s lesson: I need a lot of space for my thoughts to soar, without it they get stuck and it’s the exploring of ideas that is important to me.

Contributing to the digital transformation of the non-profit sector

Working at a national non-profit organisation to embed product thinking and practice

It’s been an interesting month. I’ve made more progress in some areas and much less than I’d like in others. Some things I thought I understood I found out I didn’t, and some things I assumed to be a given actually aren’t. But I’m still reflecting on what I can learn from it.

Participating in online communities for social good, innovation, product and digital

Nothing. Zip. Nada. Still haven’t figured out what I can do towards this goal.

Continually developing my knowledge, skills and practice

Formal education

Didn’t do anything on my BSL course. I did a little on the Gitlab Remote Working course but not enough

Informal learning

A started a few new projects: timeline of modern work, ambigoality, superpowered, dividual me, and the app-ification of work.

Reflective practice

Irregular Ideas is going well and has 33 subscribers. It’s a good opportunity to explore some of the more far-out ideas I have along with helping me practice writing and sticking to a schedule.

I wrote weeknotes on schedule every week.

Leading an intentional life

My nomadic life along the coastline was on pause. Back soon.

I’m still not doing enough to improve my physical health.

Financial independence

Runaway at 51 months.

Defining a product

How do you define what ‘a product’ is?

The short answer is: “a product is a means of facilitating a value exchange between a user and an organisation”.

The long answer is:

A product is a means of facilitating a value exchange. It takes organisational resources and makes them available to a user, typically in a way that aims to create a behaviour change for the user and return greater benefits to the organisation than it took to produce it.

So, a bag is a product. An organisation uses it’s resources, consisting of raw materials, knowledge, infrastructure & machinery, etc., to produce the (in this case physical) product. The organisation sells the bag for more than it cost to produce and distribute the bag. The customer buying the bag can use it to carry things, which is behaviour change the bag manufacturer and customer both want because it signals that the product is meeting a need.

A product is successful where the value exchange results in both parties getting something more valuable than it cost them. For the organisation this could be money (in the case of selling a bag) or it could be some other outcome such as brand exposure. For the user, they get to carry stuff around in a more convenient way, which they value over the money they paid for it. The important thing about value exchanges is that they are never like for like. The customer values being able to carry things more than they value some money, and the organisation values that money more than the time and resources it took to produce the bag.

We could also go into ‘potential value’, where the customer buys the bag but doesn’t use it, but the bag retains its value because the customer could use it but they haven’t, either for reasons outside the product’s influence or because the product didn’t meet their needs.

The same value exchange exists for digital products. You might be reading this in a browser that you use without having to pay money because the company providing it values the data they get from your usage more than the small amount of money they could charge. You get to read random things random people write on the internet, and the company gets the data you generate by doing so. That’s the value exchange.

A charity product is no different. It takes the charity’s resources (knowledge, website, hosting, etc.) outside of the organisation in a way that is useful to the user. The value return the charity is/should be looking for is in the social change using the product creates.

Where products and services differ is in the ‘means’ of exchanging value. A service is only exists and is of use at the time the user interacts with it, whereas a product exists and retains utility even when it isn’t being used. The best bus service in the world is of no use to you unless you’re using it, but owning the best car in the world is useful even if you aren’t driving it right now (because could sell it, for example), but both can achieve the same outcome for you of transporting you to the best beach in the world.

How I picture services and products interacting is almost like a service blueprint where the user experience happens over a length of time and is expressed horizontally, and the product is expressed vertically as the value chain of taking those organisational resources (CRM, API, website page, etc.) from within the organisation to outside where the value exchange can take place, which is shown as where the two (horizontal and vertical) intersect.

So, in deciding whether a user is interacting with a service, for example being on the phone with a knowledgeable advisor, or a product, for example reading the same information on a web page, we should think about which maximises the value for the user and the organisation. Both provide the same information but a phone service is likely to be more costly to the organisation because people have to be employed to work on it, but if the information would be applied in lots of complicated scenarios which aren’t easy to explain in text on a web page then the service is also likely to be more valuable to the organisation as more people will act on the information.

Products and services are successful where both the users and the organisation get more out of them than they put in.

Why User Stories are so powerful: coordination without consensus

Forget about formats, at it’s foundation a User Story is a Boundary Object.

A boundary object is information which is:

  • both plastic enough to adapt to local needs and constraints of the several parties employing them and yet are robust enough to maintain a common identity across sites,
  • weakly structured in common use, and become strongly structured in individual-site use,
  • may be abstract or concrete,
  • has different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable,
  • means of translation,
  • used in different ways by different communities for collaborative work, and
  • interpretable differently across communities but with enough immutable content to maintain integrity.

The creation and management of boundary objects is key in developing and maintaining coherence across intersecting social worlds, such as between users, product managers, designers and developers.

Going to the source of the idea of what a User Story is helps us understand more deeply what they can do and what they can achieve.

Get the reps in

The only way to get good at something is to build a repetitive practice that uses the skills you want to improve and forces feedback to make changes to the practice.

Organisational stigmergy is far more interesting than organisational strategy

Organisational strategy is top-down, specific, defined and visible.

Organisational stigmergy is bottom-up, loosely-coupled, always emerging and mostly invisible unless you know what to look for.

Stigmergy arises when people figure out useful things to do within an organisation and implement them without permission. Their actions often leave subtle signals for others to follow and create their own stigmergies.

Stigmergy can sometimes run counter to strategy but generally it’s a good thing and should be encouraged.

All my domain names


Idea branding

Riffing off ‘catchphrase comms’ from Greenaway et al in Digital Transformation at Scale, treating an idea as a brand, giving it identity, a known and accessible touchpoint.

Ideas can have domain names. No longer does ‘a thing’ have to be big enough or significant enough to warrant an online location.