The quest for the Ultimate KPI for Corporate L&D – Business Impact

What is Business Impact?

For years I have been a strong advocate for business impact measurement of L&D. Even before I got really into learning analytics. 

The sole purpose of L&D (or so I thought back then) was to improve business KPI’s. And the ultimate KPI for L&D was ‘return on learning’: the money made, or saved, as a result of your learning program divided by the total investment in that program. I was always very surprised that so few (or rather near to none at all) L&D programs and investments worked that way. Hardly ever, the full ROI as described above was calculated. And for me, that ROI was the ultimate L&D KPI….

This is part 4 of the series on discovering the ultimate KPI for Corporate Learning & Development. If you’re interested in reading more, here’s part 1, part 2 and part 3.

Can you calculate the ROI for Learning?

I think it is not to hard to calculate the ROI of learning. As a business consultant, I’ve had my fair share of calculating ROI’s of varied types of investments (although mostly IT related). And as an analytical person and trained Master of Science in Aerospace, I learned to unravel very complex systems (try a spacecraft for a change!) into smaller manageable elements that were adequate for all sorts of calculations.

The ‘easiest’ examples to calculate learning ROI are related to well structured, static, output or product oriented processes. Manufacturing is one example of such environment, call centers a much used second. Everything is well described, well measured, monitored, tested and due diligently reported through well established and proved KPI’s. Delivering a learning program to improve one of those KPI’s’s and then calculate the ROI is not a big challenge.

Calculating benefits

Step 1 is to define your desired benefits, your intended improvement and gather all the relevant data.

Some examples:

Customer retention: Will the % customer retention increase after you better train your customer success managers? Customer Services should be able to tell the additional revenue related to each percentage point increase in customer retention. And sales would certainly be able to provide data around % customer retention of employees who completed the program against the achievements of those who have not.

Compliance: Will the risk for data privacy leaks reduce after deploying that mandatory data privacy training for the whole organization? The costs associated with compliance is typically expressed as a function of the average cost of an incident times the likelihood of occurrence. So the (potential) cost of the risk of a data privacy incident would be for example 10 million (potential fee) time 0.01 (1% chance of it happening, typically per year), which is equal to 100.000. So if you would bring down the chance of an incident back to 0.005 (0.5%), it would potentially save the organization 50.000 per year.

Cross Product Sales: Increasing product training of Sales professionals to increase cross selling is a common business case. In this example, Sales should be able to provide evidence that cross selling increases overall revenue and ideally they know the exact relationship between cross selling volumes and revenue increase. So if we can develop a program to efficiently train sales people on other products (and hopefully also on generic skills on ‘how to cross sell’) and they start increasing cross selling after completing the program, we have a clear base for our ROI.

Calculating Costs

We regularly develop models for total cost of ownership of training programs. They can simply be build in excel using certain assumptions like average costs per units, type of training etc. Sometimes these cost models are used as a base for outsourcing training, sometimes to build a solid financial L&D plan for next year.

The basic elements of these models are always similar.

learning design, development and delivery: The cost models naturally contain the costs associated with learning design, development and delivery. Based on specific criteria like topic, complexity, type of format, training duration, translations, length of the program (how many years will you expect it to run), required maintenance, total audience size and locations you can make an estimate of the total expenditure.

But it does not stop there. Ideally you also include the following:

Overhead costs: You will have a certain amount of overhead costs. Think of the use of IT systems (your LMS!), HR and L&D staff costs (wages and facilities) and other overhead elements that typically go hand in hand with large organizations. They way to bring overhead into the calculation can be done by taking the sum of annual overhead costs and divide it by a certain unit of volume. This could be for example content & delivery spend. Let’s say you spend 10 million per year on content development, licenses and delivery. And you have a total overhead of 5 million (sounds a lot until you realize this also includes all L&D Staff!). You could make a decision that for every 10.000 investment in content design, development and delivery, you add another 5.000 for overhead. Now this would only be a very simple first approximation, and it’s a model, so always a simplified version of reality, but it would get you started. More complex models exist if you want to be more accurate..

SME costs: Another element we tend to ‘forget’ when calculating the total investment in learning is the valuable time contributed by SME’s. Typically SME’s are internal, sometimes they are even more expensive external experts. As an SME for learning analytics, I helped building learning analytics programs for customers and know from experience that it can take considerable time and effort depending on the maturity of the learning design and development process. I think it’s always a good idea for L&D to look into ways to minimize the time required from SME’s as their time is very precious and thus very expensive. By taking this into your cost calculation you can compare solutions that require less SME time, but have a higher D&D cost, with solutions that have a lower D&D cost element, but require more SME time!

Employee costs: I’ve said before in the discussion on learning hours not being the key L&D Metric, that time is our most precious commodity. So the time your employees spend in learning is very precious, and equally costly. Taking the cost of this time due to lost productivity or opportunity is something that I find essential for any L&D budget, but rarely see it included. I was therefore very pleased to see a recent article about the retention and L&D challenges at Amazon that called out the unbelievable high 715 million dollar potential waste through employees spending valuable time on zero value add learning activities! It so rightfully shows that these costs often if not always far outweigh the costs of learning D&D!

It also makes a great business case for adaptive and personalized learning. The more efficiently (in terms of minimal time spend) you can get an employee to the desired knowledge or skill level, the lower these costs. And for many cases that more than justifies the higher costs for developing adaptive and personalized learning and the additional IT costs for more advanced tools!

So why are we not calculating ROI?

It’s almost impossible to say how much (or how little) L&D teams structurally calculate the ROI of learning. Some reports say less than 10%, others have a bit more (18%). Honestly, I think both percentages are too high and many organizations who claim to calculate the ROI of learning are overestimating their own ability to do so, or have an incorrect understanding of learning ROI calculation. I’ve been working for and with learning and development teams who won prestigious awards for their level of quality and maturity, and never once I saw a satisfactory learning ROI calculation that would pass the level of scrutiny that a good accountant or auditor would use.

For years I encouraged and advised L&D teams and leaders to do so, but there is a reason that measuring the impact of learning is a steady top 5 or top 10 priority for L&D every year; it is really difficult and expensive to do.

And the last couple of years I have become less adamant that every L&D investment should have direct impact on business performance. Here are some of the reasons:

Brain Health: Although we still do not really know exactly how our brain work, evidence suggests that learning new things is good for the overall health of your brain. John Medina is a scientists who popularized this notion through his best sellers book ‘Brain Rules’. One of my favorite non-fiction books and definitely worth a read. I always used to make a ‘joke’ saying to learning executives that you do not want your employees to learn how to knit. It turned out I was completely wrong, as the process of learning new knowledge or new skills in itself is valuable, irrespectively of what knowledge topic or skill you are learning! So having a popular almost foundational layer of scalable learning opportunities available for your employees that they can use on any topic of their choosing will not have a direct impact to the bottom line, but it will have indirect business benefits!

Indirect impact on Business Performance: In most cases however, it’s not possible, nor required for that matter, to have a direct impact on business performance. At best we could establish an indirect impact, but calculating a direct impact on business performance in todays complexity is impossible. The key reason for this is that business performance is a more often than not a very fuzzy concept. It relies on so many different and unrelated (and therefor seemingly random) parts like the weather, a persons mood, financial markets, equipment etc. Business performance can never ever fully depend on the single notion of a person’s ability to successfully or unsuccessfully execute a task. That ability is a requirement for business performance as without a person’s ability to execute a task correctly and within required parameters like quality and speed, you will not achieve your desired business performance. But it is by far not the only factor of influence nor a guarantee!

Other factors are have much more impact, like:

• Quality of input: if you have to work with low quality of input, your performance will be impacted even if you are the best in the world! Data scientists still spend around 80% of their time fixing poor quality data that is caused by processes and people entering data in systems. You cannot expect a data scientist to deliver more studies per unit of time if the quality of data grows worse.
• Quality, efficiency and effectiveness of the process: There is a reason so much is being invested in lean and process excellence practices (and training!). Building a high quality, efficient and effective process is a really really complex thing to do! If a process is not tailored for the right business performance, you can have all the knowledge, skills and behavior you want, but you will reach boundaries caused by process inefficiencies.
• Incentives: I have seen too many times that incentives are not aligned with performance objectives, or realigned after performance objectives changed (triggering the development and deployment of a learning program…as it mostly does). You can try to make your leaders the best people managers in the world by ‘sending them’ to the best leadership programs in the world….but if leaders are not being incentivized for taking care of their people’s wellbeing, curiosity and development, I firmly believe that even that world class leadership development program will have little effect.
• Organization: Like process design, organization design is hard to get right. Many companies re-organize periodically as a result of changing market circumstances that require a different organization, or simply because they are not delivering on what shareholders are expecting. An organization that is operating suboptimal could prevent individuals from performing at the best of their abilities, or even prevent employees from applying skills they have learned in training! Think of time spend on unnecessary meetings that you must attend. Think of people being double hatted because of vacancies, think of unnecessary paperwork, or too many layers of management and unclear decision making. Or workday politics.
• Culture: According to Peter Drucker, culture eats strategy for breakfast. I believe the same goes for learning: culture eats learning for breakfast as well. If your learning programs to improve performance go against the company culture, they will not fly without serious additional effort to change the culture accordingly.
• Technology: I put technology last. Even though in many cases technology is often the first ‘blocker’ on the list when it comes to preventing people from improving their performance (or maybe second after ‘knowledge and skills’?). I often think that technology is blamed while it’s actually innocent. Admittedly there is a lot of technology out there that has a huge potential for improvement (to put it nicely), but more often people have a tendency to either not really invest in understanding what the technology can do, or try to push old school processes in new technology (spoiler…that will not fit well) and even many times both of these are true. Still technology (and the way we use it) is often a factor that limits people’s (and business) performance.

With all these additional factors impacting people and business performance, it will be, from a data analytics perspective, extremely complex to isolate the (potential) contribution of learning. Any attempt to do so either requires a really solid model that takes all these factors into consideration, has plentiful data available in each of these area’s and is able to exactly define the contribution of each of these factors. OR you must make assumptions for those area’s where such a model or the data required to build a model is not available.

It’s my experience that such a model does not really exist anywhere. It is simply to complex to build and we do not have the data required to fully test and validate it. That leaves us with one option, and one option only: make a large number of assumptions.

As I have shared in part 1 of my data interpretation for L&D series (link), making assumptions in analytics is perfectly acceptable. But only if:

1. You limited the number of assumptions to an absolute minimum
2. You carefully record each assumption and be transparent on what assumptions you have made
3. You spend effort to test assumptions to see if they actually hold up (and correct if they aren’t)
4. Your stakeholders agree with your assumptions

The challenge I always provide to many ‘ROI measurements’ is that they do not share the assumptions made, so we have no way of validating if the calculation holds true. And therefor the outcome has limited value.

What should I do now then?

So, if calculating business impact is not possible, should we stop trying all together?

Well, I’m not sure. I see 2 options. First is to take the path of full business impact analysis. Second option is to let go of trying to link all learning with (direct) business impact and set different objectives.

Business Impact with Assumptions

The ultimate goal of defining business impact of L&D (with assumptions) is to be able to confirm that a training investment (this could be a single program, a set of programs or even a combination of programs, knowledge mgt and tools) contributes a specific % or number to a business KPI.

To demonstrate this, you would require the following 4 things:

ONE: A clear definition and specification of that KPI (or KPI’s) combined with periodic or continuous tracking of the performance of that KPI over time. This is something L&D cannot do. It should be the full accountability of the business to such a KPI and monitoring in place.

If no such KPI exist, you will have to make your first assumption and define a proxy for the KPI that can be specified, measured and tracked. But, honestly, this is one assumption you do not want to make. It would in this case always be better to refrain from using business impact as the definition of success of your learning investment!

TWO: A specification of all elements that contribute to business performance, or as many as you can identify. I typically use a ‘value tree‘ structure that is mostly based on 3 core contributions: People, Process, Technology. I’ve shared a high level example of such a value tree below (note, this is just for inspiration, not a template!) with typical elements in each of the 3 core components. You can immediately see that ‘knowledge & skills’ in this example are only a small part of the tree, which is supporting my points on the complexity of impacting business performance through L&D earlier in this post. Also note that L&D could (with help) impact behavior, but only rarely this is achieved and never only through L&D (behavioral change will always need to be supported by for example leadership (lead by example) and incentives (reward desired behavior).

THREE: For each of the elements in the tree, you need to establish its contribution or impact to performance. Conceptually this could be as easy as putting a % towards each box and making sure it all adds up to 100%. However, the real challenge is to establish the correct % of each box!

So in this example, what % of performance is driven by ‘Knowledge & Skills’. Is this 2%? 5%? or 50%? The answer to that questions depends fully on the business KPI and context! Is manufacturing environments where most of the process is automated and the contributions of people are rules based, this % will be low. In sales related situations, especially sales of complex products or services, where the use of tools is very limited and the process very loose, this % will be much higher!

Please do note that “the % contribution of skills and knowledge” to business performance IS NOT the same as your L&D ROI! But it will hugely help to calculate a most accurate ROI!

Now, in reality, you will never be able to put an accurate ‘%’ against each performance factor. So this would be where most of your assumptions would come into play. An example of an assumption could be that ‘people behavior’ contributes 10% to performance. You could back up this assumption with scientific research (please make sure you clearly reference to all scientific research used). That way you do not have to analyze this yourself, as long as you are aware that the scientific research most likely cannot be 1-1 ‘copied’ to your organization and context. But that is exactly why this is an assumption, and not a fact!

FOUR: Establish the contribution of your L&D Investment towards total performance improvement.

The final step depends on a number of things:

• What performance factors were you targeting with your L&D investment? Was is knowledge? Skills? Both? Or also Behavior? Note that as a rule, L&D can only influence ‘People’ related performance factors. Improvements to the UX/UI of essential online tools should be established by IT, and process efficiency improvements should be done by the process owner.
• Did performance actually improve? Do consider that even if performance decreased, your L&D investment could still be showing a positive impact! This would mean that without the L&D investment, the performance decrease would be even worse!
• Did other performance factors change? And if so, which ones and how did they change? This includes any performance factors being added, or removed
• Where there any significant external factors that influenced performance and should be taken into the equation? Think of Covid, think of economic downturn, think of hostile takeovers, divestments or changes in strategy.

In a perfect world, none of the other performance factors changed during your time you deployed and delivered you L&D solution(s). This would be similar to the laboratory conditions in which car manufacturers the range of their electric cars: we all know it’s is not the actual range, but it does provides an indication.

So for example: assume that your L&D investment was targeted at sales skills, and skills contribute 20% towards sales performance. If sales went up from 100 to 110 millions, the modelled business impact of L&D was 20% of 10 million (increase in sales) or 2 million.

In reality the number of 2 million will be different. It could be higher or lower. We made an assumption right? BUT…..the beauty of this approach is first that it is much more accurate that contribution all of the increased sales to your L&D investment. And secondly, if you would introduce this method, validate it with your key stakeholders (they will have to agree with your assumption, remember?), like the electric car range, it could become your standard for ‘estimated impact through L&D. And like with electric cars, could become the de-facto number that everybody uses, everybody recognized and everybody accepts, knowing that it is an approximation and not a fact.

I believe that implementing such a model across your L&D investments would be accepted by even the most data driven and finance oriented CLO’s, CHRO’s, CFO’s and CEO’s in a more or mess stable business environment. (with sudden disruptions, the model will loose much of the value as the assumption made relies so much on stability…)

Off course you could refine the model by NOT making the assumption that “all other performance factors remained stable” and bring all of them in. But that would exponentially increase the complexity of the model and significantly reduce the feasibility of using it. Most likely you would not be able to get your hands on the data you would require.

Another method of testing the validity of the assumption “all other performance factors remained stable” would be to do A/B testing and compare data between sales people who have completed the learning experience, vs sales people who have not started. This is a much used method in marketing to optimize the UX and UI of webshop pages. As long as you keep both a high level of uniformity between group A and B (meaning they must be constructed out of sales people with similar characteristics) you can analyze the significance of their different sales performance.

Letting go of business impact measurement

I sincerely hope you were able to follow my line of reasoning in the previous section. I was probable one of the (if not the) most complex explanation I’ve written down so far in my blogs.

But that complexity if a reflection of the complexity of L&D impact measurement itself! If you want to do it right, it will take effort, time, considerable brain power and money. So that raised the question; do we actually require business impact measurement for L&D investments? Or for all L&D investments?

My opinion is NO.

I do believe that we can apply the model I described above to much more of our programs than we currently do and I would encourage everybody to start doing so. But realistically, we will always remain with a significant part of our investments that can never be connected directly to business impact. this because of the complexity of business performance measurement itself, the lack of data in many relevant area’s, the costs of doing the actual calculation and the fact that if we would require so many assumptions that the reliability of whatever number coming out of the exercise will be extremely low.

That is why I no longer believe that ‘business impact’ is the ultimate KPI for L&D.

Instead, I think it is sufficient and much more beneficial to define and track if we are meeting our objectives. Or in other words, if we deliver on the things we set out to achieve when we asked for the L&D investment. But that is the topic of the next post!