Author: admin

Us humans love a story that has a nice beginning, some events in the middle and a positive end. And the people who are trying to sell us something know this all too well. These are known as “just so” stories. Everything you need to know about the outcome is assumed to be contained in the story. Here’s an example:

“Company X was losing customers at an ever increasing rate. The CIO and COO got together and decided to create a Data Lake using [place your technology/software here] to analyse all of the various disparate data sources in a single repository. They also hired a Data Scientist who used the technology to identify a cohort of at-risk customers. A churn prevention program was developed to target the at-risk group and Company X recovered its lost ground and began to grow its customer base again.” 

The implication is that “if your business has customer churn issues then you should consider talking to [place your vendor here].”   

The problem is that the reality of the above use case is probably a lot messier than the just-so story leads us to believe. This is the far more likely scenario:

“The CIO of Company X hired a Data Scientist who managed to obtain some IT resources out of the datawarehousing team to germinate an unstructured data repository. The Data Scientist, as a side project, poured data from multiple systems into the new repository. The Data Scientists official project was developing some scorecards and dashboards for the COO, based on some updated efficiency KPIs. As a result of building the dashboards, the Data Scientist became aware that churn was high and, on a hunch, decided to hit the new repository to see if there were any unique characteristics of the churning customers. The Data Scientist happened to get hold of an unused license of [vendor product here] to do the analysis. Turned out that there were indeed unique characteristics of churning customers, and the Data Scientist proudly told the CIO and COO about it….etc”. 

Firstly the CIO’s reason for hiring the Data Scientist was not deliberately to tackle customer churn. Also it was the Data Scientist who managed to, almost covertly, divert IT resources to create the embryonic Data Lake. So investment in a Data Lake was not a deliberate decision by executive trying to solve a particular business problem. The Data Scientist could easily have looked into the unique characteristics of the churning customers and not found any, simply because there was nothing there to see. In fact, had the Data Scientist been developing some financial reports for the CFO, instead of a dashboard for the COO, the customer churn problem may not have been noticed at all. The just-so story could just have easily turned into an accounts payable fraud detection story instead. Most of all, the technology used to determine the customer characteristics could have been any product that was capable of the type of analysis the Data Scientist decided to try and was only utilised because of the unused license.

So you as the reader of the Use Case, and potential buyer of the vendor/software/technology, need to keep in mind the possibility that the story has been given to you in a “just so” form. This is particularly important as you will find that those companies that try to start a Data Lake promising to “increase sales by x%” or “decrease customer churn by y%” will be sorely disappointed. Despite the impression given by just so Use Case stories, Data Science is more akin to diamond mining than constructing a bridge. You need to add data capability to help uncover the gems that are no doubt there… but how big, how many and what type of diamond? No-one will know until you’ve uncovered the gems, a long time after the decision was made to invest.

Postscript: I’m aware there is a Gartner Report which fundamentally disagrees with the above (i.e. 90% failure due to uncertain use cases). However I submit the analysts may have fallen for the trap described in this article, looking at results after the fact. Were the 10% successes really designed specifically for the issue that eventually rendered them a success. Or were they they just the lucky 10% who found something. More likely the 10% figure itself comes from a Woozle Effect.

When designing enterprise level information technology solutions, a new dichotomy is becoming increasingly apparent: transaction generators vs information consumers.  When thinking in terms of single applications we have traditionally thought in terms of business functionality: e.g. an accounts receivable function needs to be supported by an accounts receivable system.  It has been standard to rely on the individual application’s reporting system to satisfy the analytics requiring portion of the user base. For more esoteric requirements we have started to rely on extracting data into data warehouses and, more recently, data lakes to mash up data from the various functions and also with 3rd party data (like data streams).

However, it is becoming more clear that there is indeed a set of users who do very little transaction generation and really don’t need to be users of our transaction system at all.  This includes executives, analysts, consultants, auditors, data scientists, and so on: information consumers. This class of users can be better served if they can instead obtain almost all of their requirements from a comprehensive data lake.  Indeed such a design is preferable as this can be a single source for all of their analysis, removing the need to get access to and learn how to use all of the various data transaction systems which generate the source data. The transaction generation systems can then concentrate on capturing transactions efficiently and effectively, freeing themselves from needing to accommodate the needs of these fundamentally different types of users.

Aggregating all of the required analytics data into a single enterprise repository has many synergistic benefits.  In a single repository, data from multiple functions are able to be seen in light of whole-of-enterprise and even whole-of-supply chain and whole-of-market contexts.  User familiarity with their analytics interface is dependent not on the content of the data, but on the analytics task: Visualisation (e.g. Tableau, Qlik, PowerBI) for building interactive dashboards, continuous monitoring for alerting and compliance (e.g. ACL, Logstash, Norkom), data discovery for investigations (e.g. SAS Analytics, Spotfire), reporting (e.g. Cognos, BusinessObjects), search (e.g. elastic), data mining (e.g. Oracle Data Mining, EnterpriseMiner) and statistical analytics (e.g. SAS, R). 

But doesn’t this approach violate the concept of single point of truth (SPOT)? No, the transaction system might indeed be the SPOT, but our data repository can be a replica of the SPOT and differ in known ways (e.g. is a copy as per midnight the previous night or as per 30 minutes ago).  For the vast majority of information analysis needs, this level of “fresh enough” is perfectly fit for purpose.  

Functional systems have the most rudimentary analytics capability (likely just some basic reporting) which is only a small fraction of the analytics capability of value to information consumers.  Executives and other information consumers have really been short changed by the rudimentary data analytic tools provided by functionally focussed transaction systems up to now.  

Modern enterprise solution architects need to split the data analytics functions from the transaction systems: free the data and then deploy the analytics power of Big Data.  

Previous articles in this series about the pragmatic use of KPIs in organisations, are available at the links following this article. This particular article looks at KPIs as part of an overall framework of organisation control.

KPIs are part of a whole
KPIs, especially when widely published and/or used to calculate personal bonuses, are an amazingly powerful set of behaviour modifiers. As explored in previous articles, KPIs can set and/or communicate direction to staff and other stakeholders, encourage hard work and innovation and provide feedback on the success or failure of past efforts. When viewed from this angle it becomes obvious the potential consequences of setting KPIs in ignorance of the many other influencers of behaviour in an organisation.

Managements use a great range of levers to influence and shape behaviour of the individual parts of their enterprise including:

Corporate Policy
Risk Mitigation Plans
(Approved) Budget
Process Manuals
Organisational Structure
Computer System (automated) Controls
Contracts/Agreements with Partners, Suppliers, Customers and Regulators
Project Mgt Plans
Position Descriptions
Vision/Mission/Strategy Statements
Water Cooler chats
Public and media proclamations
Executive or Board Orders/Decisions
C-Suite meetings/roadshows with staff
Reward and Recognition Programs
Review Report Recommendations
Training and Development Programs
etc
Staff, suppliers, partners and other stakeholders glean what is expected of them from the above and use their understanding of the levers to guide their own actions and behaviour. The Performance KPI, when tied to personal bonus schemes, will often be seen by staff as the most important of these, as that is where the organisation is putting its hard cash.

KPIs are more important than what the CEO says
This is how some of Australia’s largest insurance organisations recently sent mixed messages to their staff: The Executive and Board emphatically stated that they wanted an ethical institution, but the KPI driven personal reward systems were totally based on profit-focussed measures. Decreasing costs (like minimising claim payouts) and increasing revenue (like making policy sales to any buyer) will become a very large focus for those measured and rewarded exclusively on profit. Essentially staff thought: “I hear what they are saying, but if they really wanted that, they’d change my KPIs”.

It becomes easy to see that KPIs must be in close synchrony with the remaining behavioural signals being sent to staff. If you want different behaviour to the past, you must look at all behavioural controls and ensure they are sending a synchronised message of change or else your staff will choose which of the conflicting messages to follow.

Not only must KPIs remain in synch with other influencers of behaviour, it is also important that they remain in synch with each other. If the KPI system is rewarding one set of management in one direction and another set of management in an entirely different direction, it is likely the two executive will come into conflict. For instance, if some executive’s KPIs and bonuses are tied to completion of a capital plan, and another set of management have KPIs about minimising budget expenditure, it is likely the two set of management will clash.

Don’t Ignore the Knitting
As with Strategy generally, if KPIs concentrate exclusively on what needs to change without including maintenance of core activity, it is likely that core activity will be sacrificed or, at best, neglected. Once we provide contingent remuneration, we are asking our staff to focus on those issues we are measuring and rewarding, which is implicitly asking them to not focus on others. A KPI developer must keep this in mind when developing a portfolio of KPIs for a team or individual. Once again this is another reason why some KPIs may not need to be “stretch”. They may act as a floor threshold (i.e. do not let this figure go below this level – e.g. do not allow employee churn to rise above 12% p.a) which are easy to achieve but ensure that core activity is still being completed whilst the individual strives for excellence in other areas of continuous improvement.

The Takeaway: What to Do
As a takeaway, KPI developers must be cognisant of the many ways that behaviour is influenced within an organisation and must keep in mind that the KPI and bonus are just one of these influencers. Do you have any examples of where KPIs have been inconsistent with other indicators of corporate intent leading to unintended dysfunction?

Previous KPI Series Articles
A Tale of Two Managers – a hypothetical where two managers react differently to a revolutionary idea. The story highlights the dangers of annual % increment KPIs to overall organisational performance.

The Parable of the Wet Driveways – an alternate reality were scientists need to figure out how to determine if rain occurred at night when everyone’s asleep. The story warns of the dangers of confusing correlation and causation when measuring performance/outcomes.

Oils ain’t Oils & KPIs ain’t KPIs – the many reasons KPIs are currently used in practice and how easy it is to accidentally start using the ones you have in ways for which they were not originally designed.

KPIs ain’t KPIs: Part 2 – exploring the ways that humans interact with KPIs once they begin to be rewarded based on the results. It’s not always as expected/planned.

https://www.linkedin.com/pulse/kpis-aint-everything-jeffrey-popova-clark/

KPIs are like fire: they are very powerful, but if you don’t use them carefully they will burn you and your organisation.    An honest look at the use of KPIs in practice leads to a number of pragmatic suggestions which can vastly improve the positive impact of these powerful creatures and avoid their many pitfalls.

What are they for again?

 KPIs are for a great many things:

1. To provide an incentive to staff to strive a little more (employee incentive)
2. To send a signal to staff about what goals the executive would like achieved (communicating staff direction)
3. To allow executives to see if their staff need to be paid a bonus (remuneration calculation)
4. To help align staff action to organisational goals (staff alignment) and to each other
5. To efficiently monitor progress on how the business is progressing (monitor business performance)
6. To quickly identify any issues that need addressing (problem identification) and how big those issues are (problem assessment)
7. To allow comparison to benchmarks (either compared to the past, other parts of the organisation, other organisations or some gold standard ) to determine remaining potential (benchmarking)
8. To communicate to external stakeholders what the organisation intends to focus upon (stakeholder communication re intent)
9. To tell external stakeholders how the business is performing (stakeholder communication re performance)
10. To assess the current state of the environment that impacts on the performance of the organisation

The big problem is normally we start unconsciously creating KPIs for one or two of these purposes. More often than not, though, our KPIs end up getting (ab-)used  for the other purposes as well. 

For instance, KPIs that are focussed on aligning staff may be subtly different to the ones that are meant to stretch them.    An “align staff” KPI target does not need to be achievable, just able to be understood to help staff understand the direction management intends for them to progress.  This could be an impossible target like “No imperfections in any final product”.   However a stretch KPI may be very difficult to achieve but is still possible: e.g. “no less than 6 sigma quality across the year”.  If the intent of management is to convey a focus on product quality then a “No imperfections in any final product” KPI target is very useful.  Just make sure that bonuses are not tied to achieving it.

Some measures are very useful for monitoring the performance of a business, but are not actually intended to change behaviour in any way.  For instance, one may monitor employee churn rates to ensure that they are not increasing or decreasing.  If they are decreasing it could signal that  employees are feeling job insecurity or that the staff pool is not valued in the external labour market.  If it’s increasing it could be because of better job opportunities elsewhere, or an ageing workforce that is increasingly retiring, or poor supervisory practices.  Essentially these kinds of KPIs may be a watch and see type: Changes warrant attention but no one is trying to “hit a number”.  Such KPIs tend to be underutilised because there’s no stretch aspect to them. So the confusion is that it can’t be a KPI because it can’t be used to incentivise or stretch a team or staff member.  But the KPI is useful because it will help monitor business performance and also identify issues that require business action. 

The Take Away

The key is to overtly label each KPI based on its raison d’être.  If a KPI is for business monitoring and not for remuneration…label it so.  Contact me for a useful KPI purpose assessment tool that can assist with this process.

Next week we’ll look at how KPIs tied to incentives can sometimes have unintended consequences.

Once upon a time there were two managers, from different firms, who each ran a section expending $20M per year to produce their outputs.  They both wore glasses, they both spoke well at large gatherings and both were well liked by their respective senior executives.  Not rising hot shots, but solid performers that no doubt will eventually earn their way into executive ranks in the not too distant future. 

The Idea

One morning in the shower, whilst pondering ways to improve their respective areas, they both independently come to the same insight.  This idea was awesome…revolutionary…absolutely fabulous: it would save their sections not only the 5% stretch efficiency target in their personal KPIs, but a whopping 25% saving.  They could reduce the costs of their section from $20M per year to just $15M per year without interrupting production and without any capital expenditure. They both could smell the executive suite leather….what an idea!

However this is where our two managers began to differ.  One, named White, is totally committed to benefiting the organisation, a total team player, whilst the other, Black, thinks more in personal terms.  White immediately implements the idea and is absolutely overjoyed to discover that the plan is working perfectly: costs have dived 25% and the bottom line for the company is a full $5M per year better off and will be for every subsequent year. White enjoys hearty congratulation from the executive and a full bonus for the year.

 Black however decides to implement the idea but simultaneously makes other changes that decrease the efficiency of the section by $4M. This means that in the first year Black meets the stretch target of 5% by saving only $1M, but does get the full bonus and also the same hearty congratulations from executive enjoyed by White.

End of Year One

At the end of the year White sits down with the boss to negotiate next year’s KPIs.  “Wow, you smashed that 5% out of the park.  25%! Well done.” Says White’s boss.  “Thankyou”, says White. “So, look we can’t expect 25% every year, so I’m happy to leave the target as 5% again. That shouldn’t be a problem for a hotshot like you.”  But White says “Well we are of course enjoying the 25% reduction again this year you know.  Another $5M in the bank!”. But White’s boss says “Can’t rest on your laurels though…continuous improvement and all that.  Should be able to find another 5% surely”  White walks out a little dubious, not confident at all that another 5% is really available after cutting 25% out of the expenses last year.

Black also sits down with the boss.  “Well done. You not only beat the target of 2.5%, you even hit the stretch of 5%.  That’s like $1M.  Well done. I hope you enjoy the bonus. So do you think you can do it again? Another 5%?”  Black says “Sure, lets see if we can’t do it again.” Black walks out knowing that next year’s bonus is in the bag.  Just remove some of the inefficiencies deliberately put in last year and the initial saving idea will do the rest.

End of Year 2

White tries everything during Year 2 but just can’t make a further dent in the costs:  $15 M expenses again.  Black, meanwhile, merely removes some of the deliberately planted inefficiencies and brings the costs down from $19M to $18M, beating the target again for the second year in a row. 

White’s boss says “Well…you couldn’t even hit the base target this year.  A bit slack, White. Look we all remember last year, that was great, but you’ve got to keep that up. And remember the CEO is retiring in a couple of months and the new CEO won’t remember your great year. You’ve got to at least get the base target of a 2.5% reduction. OK?” White says “But we’ve only spent $30M over two years now, when we used to spend $40M. We’ve saved $10M..which goes straight to the bottom line!” But the boss says “Look White. We can’t just have one hit wonders around here.  You’ve got to show us you’re able to get continuous improvement.” White leaves the meeting feeling a little shaken, determined to find another efficiency in the section.

Black, meanwhile, is having a great meeting. “Well done, Black, you did it again.  2 years in a row hitting the stretch target.In fact a bit more this time.  Another bonus.  From $20M to $19M to $18M.  Surely you can’t do it again? Another 5%”. Black starting to get cocky says, “Look we’ve hit 5% twice lets go for 5.5% this year as the stretch”. “Really,” says Black’s boss, “You really think you can do it”. Black says “Continuous improvement shouldn’t only be for operations, but management should get better as well”. “Ok then, if you really think you can do it, 5.5%”.

End of Year 3

In Year 3 you can guess what happens. White makes no inroads on the $15M and Black shaves another $1M off the cost base.  White has missed base targets two years in a row and Black has now hit stretch 3 years in a row.  Black is a hero, with growing fame and talk of fast track promotion, whilst White’s job is in jeopardy.  White’s boss says “The new CEO can’t understand why I keep a manager who misses base targets two years in a row. You’ve simply got to improve White”.

End of Year 4

By the end of Year 4 White is sacked and Black is promoted, even though Black’s unit never once became as efficient as White’s.  In fact, over the 4 years, White’s unit outperformed Black’s and managed to produce the same outputs for $60M (a saving of $20M), whilst Black’s unit used $70M (a saving of only $10M).  Why is White sacked even though she has performed twice as well as the hero Black?

Indeed the “hero” Black has deliberately spent $10M of the company’ s money being deliberately inefficient purely for their own gain (see the lighter red section of Black’s chart). Why are the rewards for the individual so totally out of whack to the benefits to the company?

This is a simple example and the likelihood that there are sociopaths like Black deliberately manipulating performance to optimise their personal long run gain at the expense of the overall business is probably low. But it does illustrate how seemingly corporate aligned KPIs can lead to unintended counter-productive decision making amongst those measured and rewarded by KPIs.

 Have you seen KPIs encourage counter-productive behaviour? Let me know your examples.

All throughout the land, scientists laboured to find a way to detect when it had rained. If it rained only sometimes, it was useful: crops would grow, dams would fill with drinking water and house roofs would be clean. But, if it rained too often: floods would come, crops would rot and houses would grow mouldy and mildewy. It would be very useful to everyone to know how often it rained. But sometimes it rained at night! How were the scientists to know if it rained when everyone was asleep?

On one rainy day, Sally, one of the land’s great scientists, noticed something exciting: Every time it rained, the driveways became wet! In fact they stayed wet for hours afterward, particularly at night! A wet driveway might be a great way to tell if it had rained. Tests had to be done!

Sally applied for funding and over a period of a year research assistants stayed up all night in random geographic locations watching for rain and recording the wetness of a random selection of driveways. Pebble, concrete, bitumen, flat, steep, curved, straight…it didn’t matter. It really worked! 100% of the time that it rained, the driveway became wet. 100%! This was sensational. Even the proportion of patients with heart conditions having high cholesterol was not 100% and yet cholesterol tests were a trusted test across the globe. Driveways were available all over the world…the Wet Driveway test was marvelous!

Sally became an international sensation. She was interviewed about her great idea, invited to lecture at universities and at business functions. Soon she was being asked her opinion about all sorts of things with little relationship to either driveways or rain because polls show that she was one of the world’s most trusted “smart people”.

After a while, other scientists were able to publish useful guidelines above which various conditions occurred. On average driveways were wet about twice a week. However, if the driveway was wet more than 5 times in a week then flooding risk was increased by over 250%. Less than 0.5 times a week indicated an 80% increase in the likelihood of drought. Similar thresholds were calculated for plant rot and mildew house. Soon everyone was using the wet driveway test, and the new thresholds. Television stations started to include the wet driveway averages in their newscasts. Even governments and corporations started to use the new statistic in their budget deliberations.

Sally was very pleased that her idea was being developed in such a useful fashion. Some of the applications being developed were ones she had never considered herself. Being a trained scientist she used her frequent television appearances to educate the general public about the proper ways to use the wet driveway test:

“2 wet driveways a week is only an average of course, and a global average at that. Some driveways are located in places where rainfall averages higher or lower than the global average.”
And
“You shouldn’t get too worried if your driveway doesn’t get wet for an entire week, it doesn’t mean you’re in drought straight away. It’s when the average over the year gets below 0.5 that you can say you’re in drought …and don’t forget to adjust for your local averages”.

Despite Sally’s sterling efforts, untrained amateurs continued to use the test based on the simple guidelines only. Still, on balance, Sally felt that using the test simplistically was still an improvement over not using it at all. But the people and even other trained scientists continued to use both the tests and thresholds in an uninformed and simplistic fashion. One week of no wet driveway and people started to install water tanks and buy water. On the 6th day of rain, some people began packing up their houses ready for a flood. “Well it is 100% accurate isn’t it?” Sally tried to explain that the thresholds were based on averages and that “above 5” only indicated an “increased risk” of flood. The 100% referred to the indication of rain and then only in the direction of rain producing wet driveways, not wet driveways producing rain. The 100% did not refer to the flood risk at all. Despite her efforts, some companies started to sell flood preparation kits: “For that dreaded 6th day”.

Other companies began to sense a market opportunity. Market Research showed that people not only wanted to predict the future of floods and droughts using the Wet Driveway test, they wanted to be able to influence it. The companies set their scientists to work to:

“Find a way to keep the wet driveway average between the safe levels of 0.5 and 5 days a week,” said the CEO of one company.

Soon, a famous hairdryer company, managed to develop a “driveway dryer”. “As soon as you get to 4 Wet Driveways, you turn on the Renington Driveway Dryer and you’ll stay below 5, guaranteed!”, claimed the internationally televised advertisement. Renington sold millions of the their patented Driveway Dryers. Renington was also reportedly working on a driveway wetting device which could push the average above 0.5 and, presumably, thereby avoid a drought. Householders started using their garden hose to wet their driveways in an effort to “Ward off the drought.”

Sally immediately saw the problem. People thought that drying or wetting their driveway would influence the weather and therefore their risk of flood or drought. In fact she’d even heard some other trained scientists (possibly working for Renington) saying scientific sounding things like: “evaporation from aqueous covered non-porous surfaces contributes positively to the local humidity, which in turn increases the probability of rain.”

Sally knew she had to educate the puiblic and immediately used her guest television spot on Ellen:
“The Driveway test only tells you if rain has occurred, much like a cholesterol test just tells you how much cholesterol is in your blood. There is no evidence that wetting or drying your driveway will change your risk of flood, plant rot, drought, or whatever,” Sally explained.
Ellen asked her, “So are you saying you were wrong about your test?”. A little shocked by the question, Sally stammered a feeble “No-o, but…”. Ellen interrupted her to announce a commercial break.

Later that day the CEO of Renington offered Sally a highly paid position as Executive Vice-President of Research and a generous research budget guaranteed for the next twenty years. The Driveway Dryer is still a hot seller.

*****************************************

When I catch myself making this typically human correlation-causation error, I find it useful to “Remember the Wet Driveway Test”. Also it helps me identify when people generally or even entire industries are making the same common error. As we know much competitive advantage is found through not making the same mistakes as everyone else. Have you got any classic examples of this common error?