27 August 2009

The Flaw of Averages

I have heard it said before to trust the law of averages. In the present day and age, everyone wants to predict the future and to be able to prepare for it. In doing so, we often try to use averaging only to be disappointed / shocked / outraged at the results when we compare reality with our predictions.

If we look at a popular index, being CPI (Consumer Price Index) which measures how much more expensive things are from year to year. In Australia, we are used to an annual CPI of 3% - 3.5%, however anyone that has been inside a grocery store in the last 6 months knows that 3.5% would be a wonderful dream come true. We often see grocery prices rise by anything from 10% - 25% in a given year, not counting seasonal items for which prices spike and dip according to the supply-demand equation. I am most certain this is not unique to Australia, as having lived in the USA and South Africa for some time, I recall the same shocked realization, even though the periodic statistics that were released assured me of a CPI well below my own rudimentary findings.


Sam Savage recently published a book titled "The Flaw of Averages" which takes us into a full-blown study of the dangers of decisions based on averages. You can also find more information here.

There is the story of the statistician who drowned while crossing a river that was on average 0.6 m (2 feet) deep. As you can image, the deepest part of the river may well have been several times deeper than the man was tall. Or perhaps the hapless soul lost his footing and remained horizontal (making him maybe 0.4 m tall) for too long and not able to breathe.

In the same way, in business we know that if we experienced a 5% overall growth or contraction, it is never assumed that all pro
ducts / services grew / shrank by the same as the average. So be careful when business decisions are made based on average figures.

Let's take an example KPI: assuming we had 100 products to sell and we sold 100,000 of each product for the year (10 million items sold).

If our QA policy allows for 2% returns due to quality problems (clearly not Six Sigma compliant), this means that 95,000 items returned in the year (0.95 % returns) means we are doing OK and well within the allowed range. Therefore as long as returns stay under 2% (gosh we even managed to get it under half of that), then the Production manager should have no reason to jump up and down ranting and raving like a lunatic!

However, if we dig further, and find that of the 95,000 returns, as much as 94,000 are on one single product and it just happens to be a very profitable line (excluding
returns), then we can start to understand the reason the Production manager is constantly screaming and about to have a cardiac arrest, as this represents a 94% deficiency rate.

Here is another example with a report (please pardon haziness due to resizing of image to fit the page) to show the outliers both good and bad.







Here we can clearly see that not all sales reps are pulling their weight equally in terms of the dollar value of products sold. However, this also shows that a single aggregate measure in and of itself should not lead to knee-jerk reactions. While this report indicates turnover, the individual sales reps profitability may look completely different, and if we combine this with returns / rejects, and possibly customer service scores, the skewing might not look nearly as bad as initially perceived.

Thus we conclude that averages by themselves are a dangerous measure and to get a better overall picture, we need multiple measures. KPI's should not be restricted to a single measure, but an indicator could be a complex algorithm that include multiple measures to derive a single indicator value that immediately spells out to its audience whether the picture of events is good, bad or neutral.

Have an awesome week!


No comments:

Post a Comment

Note: Only a member of this blog may post a comment.