Those who define KPIs love the mean. It allows you to reduce the chaotic space of human behaviour down to a single number, which you tend to increase or decrease. This compression is tempting, but reductive: the most important—and fascinating—behaviour lies in the tails of the distribution.
I have worked for a few startups at this point. Almost every graph relating to human behaviour that I’ve seen looks like the following.
This is a power-law, or the the Pareto principle in action. When you think power-law, think skew: the vast majority of customers contribute negligible revenue, a small proportion of customers contribute almost all the revenue. Switch revenue for engagement, cost, referrals, you name it. As an aside: I would encourage every person with access to data in a business to study the behaviour of their top 1% of users by some metric (cost, revenue, engagement). What you find will fascinate you.
Let us view power-laws through the lens of a business thinking about their customers.
The good side of power-laws: if you pick your battles correctly, you can get a lot done with limited effort. Some tweaks may be 100/1000x more impactful than others. The work is in figuring out which tweaks to make.
The bad side of power-laws: skewed revenue distributions introduce fragility. Power-laws, by definition, concentrate power in the hands of a few. You are beholden to a small number of over influential customers—if those customers leave, you might be hosed. If my memory serves me correctly, King (of Candy Crush fame) is a good example of a highly skewed business: the vast majority of their revenue comes from an extremely small percentage of their total user base. 40% of Slack’s revenue comes from less than 1% of the user base. In both these cases, they have managed to mitigate that fragility by scaling the user base by volume. This often isn’t possible for early stage companies: e.g. business-to-business companies that focus on higher-value individual contracts. As a buyer in this situation, this gives you massive leverage.
So Stephen, what do you suggest? If you’ll excuse the pun, means are meaningless in power-laws. I encourage you to consider how you can represent the richness of the underlying distribution when setting KPIs, targets and goals. For example:
These targets are far deeper than the mean. The former allows you to ensure that more than half of your customers by number are profitable: this says nothing about how much your worst case customers lose you, and whether you are profitable overall. The latter allows you to shape your revenue to a target distribution, whilst still allowing some skew at the tail for the ‘whales’ of your business.
Should you use these as metrics for your primary goals? Probably not: they’re not particularly catchy, and can be complicated to understand. However, avoid the temptation to squash all behaviour into a single number, and understand the underlying distribution wherever possible.