To Measure or Not to Measure? That is the question…

In any quest for the ultimate KPI for corporate learning, the question on to measure learning hours or not needs an answer. Learning hours are for some reason controversial, and once in while the same discussion pops up again on LinkedIn: should or should we not measure learning hours? As a response to some of these LinkedIn posts (including for example the one from Myles Runham) and a discussion at one of my customers, I decided to explain my view on why measuring learning hours can be very insightful and valuable…and even strategic. I’ll provide a few simple examples of how to use learning hours to your benefit. Could they be the ultimate KPI for learning? Probably not. But my point is that monitoring learning hours can actually be a very strategic differentiator if done well!

So if you agree with me…you could skip this post entirely and wait for part 3 of the series…unless you have a challenge convincing others of the extreme usefulness of measuring learning hours and are looking for additional reasons why it is so useful (I might mention some you have not considered before). If you do not agree with me, and think measuring learning hours is a waste of time and effort, please have a read and share your thoughts in the comments (or on LinkedIn) and tell me if you still think it is a waste, or if you might have changed your mind. I would be very curious. I’m especially looking for feedback that might convince me that I am actually completely wrong about this!

Before we start, lets make one thing clear. Measuring learning hours without context has no value. But honestly, any metric without context is pretty meaningless, learning hours is no exception to that universal rule.

That having said, I have been measuring, analyzing, and reporting on learning hours since about 2014 at a number of large international companies and have always found it extremely useful (and challenging, but that is a different story!). There are different ways to measure learning hours. The most ‘simple’ and most common way is to record for each training the expected or planned effort (in terms of time) to complete it, multiplied by the number of completions. As long as this definition and calculation is consistently applied, it can be useful in a specific context. As long as we all realize that (a) we’re only looking at formal learning/training, (b) some people will complete faster, others need more time, (c) we make a solid estimation and (d) we record all the assumptions mentioned clearly and ideally validate some of them periodically (especially validating or simply maintaining the ‘expected time’ with program updates seems to be a challenge).

I’ve seen examples of where things can go horribly wrong because some people put learning hours in hours, others in minutes and some even in seconds. I’ve also seen people taking the time between start date+time and end date+time, which is incorrect and will give you a huge overestimation when it comes to trainings that cross a number of days, and especially 3 or 6 month long leadership training programs! I’ve also seen systems being used incorrectly and give very high learning hours values. So please check the quality of the data….your analytics and insights fully depends on it!

A more accurate, but also more complex, way of measuring learning hours is taking the actual time. For traditional elearning, this is possible in most cases and with most LMS and LXP systems. The challenge is (1) people can complete their training over the course of a full working when they do a lot of other things in between, and (2) many other forms of learning do not easily provide this data. Or not at all when it comes to F2F training. So most likely you will need a hybrid approach. The more learning happens online, the more data you will be able to harvest to this purpose. Just be mindful of the effort required. Especially with social learning, or on the job training and performance support, the trouble to get the data in such a way and format that you can bring it into your learning analytics model and dashboard might be much higher than the improved accuracy of the values it brings! It’s perfectly OK to make assumptions, we make assumption all the time. As long as you clearly record the assumptions and validate them once in a while.

Whatever method you choose, learning hours data will always be inaccurate to some extend. And we will never be able to measure the accurate time people ‘do learning’. I have my share of reflective moments under the shower, and when I’m going to bed, but I would feel a bit reluctant to record these as ‘learning experiences’!

Despite the inaccuracy, I think learning hour based metrics are extremely useful. I’ll give 3 examples.

1. Learning Hours to calculate learning volumes

I once explained the head of learning at one of my clients that I look at learning hours as a leading indicator for learning in an organization. And whatever method for calculating learning hours you take, the basis of this statement is the assumption that if people spend more time in learning activities, they learn more compared to when they spend less time in learning activities. There’s an interesting and persistent story going around (amongst others socialized by Adam Miller, former CEO of Cornerstone on Demand) that companies where employees are spending at least 5% of their time on learning (the equivalent of around 2 hours a week, 1 day a month, or 96 hours a year) outperform their peers. This has led to some CLO’s adopting target for learning hours. Unfortunately, I’ve never been able to validate this (if you know of additional sources, I would be very interested to have a look). Without validation, this assumption as well as the overall assumption that says that more learning hours lead to more learning can (and should) be challenged, and we should do that often. Although pretty much our entire educational system is build on it as educational credits are mostly (if not exclusively) linked with ‘time spent’, so it must hold some merit. But even if we would question the assumption, monitoring the learning hours to understand volume of learning is a solid indicator of success. Here’s some examples:

First, learning hours is a much better way to monitor volume compared to capturing learning program completions. Tracking completions can give very skewed insights. The illustration below shows 3 very different programs that have very different completion volumes. Looking at completions alone (left chart) could suggest that Course A is the most impactful one. However looking at total learning hours generated by each course gives a completely different picture.

Secondly, monitoring changes in learning hours provides you insights on trends. Imagine you want to stimulate and facilitate learning in your organization. If you keep track of total learning hours consumed, you can start generating insights on for example

1. Where in the organization are most hours consumed?
2. What is the balance between push and pull?
3. Are learning hours in a upward trend or downward trend? And is this a good or a bad development? (spoiler: it depends on your strategy and targets. See also the next part of this article)

Below are a few charts as examples of what I am referring to. The first chart on the top row tells you that the number of learning hours is growing month over month. Which is usually a good thing. The second chart then splits the learning hours between push and pull. In my experience most learning is still pushed and you can see that this is increasing over time. While pull is much lower and growing much slower. The 3rd chart of the top row then provides how the hours are distributed over 3 departments. You can already see that each department is ‘behaving’ differently. Finance is fairly stable. IT has a clear downward trend, while sales has a clear upward trend. To be honest, the measure that should have been used here is average learning hours per employee as Sales could simply be top of the list because they have the most people, but its more the trend that matters for this example rather than absolute numbers.

The bottom 3 charts are each a further level of detail of the pull/push chart, but then split per department. Again we can see very different trends per departments here. Pull learning in Finance is clearly growing. Push learning is IT is clearly decreasing, while pull learning remains stable. Push learning in Sales is growing a lot, while pull learning is on the decline.

Looking at these charts and figuring out if this is what is to be expected depends 100% on your L&D Strategy, plans and context. I could imagine that a learning campaign in Finance was launched in the beginning of the year, explaining the increase. I could imagine that IT restructured their IT compliance portfolio, merged a couple of topics or even started to introduce a more adaptive learning design for their courses; reducing the average time people spend on each IT compliance course. However, different scenarios are possible. How about IT started to outsource activities at the start of the year and as a result has a much lower number of FTE’s?

A possible sales scenario could be that they started a major upskilling program for the sales force at the beginning of the year, and decided that this program was so important that they pushed it to all sales professionals. Eager to learn, the sales professionals picked it up really well, but were left with less time to do training out of their own initiative. As said many times….data and charts need context to make sense and tell an accurate story.

Again, the data alone cannot tell if these trends are positive or negative. If you can just sit back and relax because everything is going to plan, or require to take measures to turn a negative trend around. That fully depends on your situation, strategy, plans and context.

2. Learning hours to calculate time & money spend on learning

Or should I say ‘time=money’?

In todays world, time is regarded by many as our most valuable asset. And there is a good reason that ‘Time Management’ is consistently in the top 20 most popular courses on LinkedIn Learning in 2019. 2020 saw it even on number 1, but that was also due to the Covid situation, so not entirely representative. Putting a price on personal time is sheer impossible. But putting a price on an hour of time spend by an employee is easy to calculate. You could take the average loaded costs per employee per hour, or even divide total annual revenue’s by total employee hours per year. The first is typically in the 50-150 USD range, depending on the type of company and geographical spread. The latter can really differ but is significant higher for healthy companies and could be a factor 2-5 times that amount.

It’s therefore very strange that only few (if at all) learning teams and organizations include the time element (and related value) of participants in any budget or ROI calculation. Imagine an organization with 25.000 employees, spending 25 million a year on L&D (a nice round ~1.000 per employee on average). If the average learning hours per employee per year is only 25 (this is little more than 2 hours per month!) and the average loaded cost per employee is 50, the actual total investment in L&D is over 56 million, more than twice the 25 million mentioned earlier. If your employees spend more on learning, and/or cost more per hour, the difference will be even more. For fun try to calculate the total costs related to time spend on learning for this company if they would raise the average learning hours to 100, and the average employee cost per hour is also 100…..These are serious numbers.

So I could be a very lazy teacher, force my 1.000 participants to read a 25 euro book that will take them on average 8 hours to read, and have office hours scheduled to answer any questions. That way my ‘L&D spend’ is low (basically only 25.000 euro plus some of my time) and I can demonstrate a very effective L&D organization to senior leadership. But the actual costs of the learning experience, made in the organization are remain hidden.

Or I can invest 50.000 euro’s in making a world class summary that brings across the essence of the book in just 15 minutes, enriched with relevant context and examples in a highly engaging way. 50.000 might seem twice the L&D investment compared to just buying the books. But not if you include the value of the time spend by the participants! Do the math and you will find out quickly which of these solutions will provide you with the highest ROI. Assuming naturally that the learning transfer of the summary is equal to that of the book!

On a side note: The business case for Adaptive learning

The above example also provides a strong business case for adaptive learning. For me the benefit of adaptive learning is not just providing a better learner experience and increased learning transfer. Adaptive learning also reduces time spend on learning as it minimizes, or fully removes the irrelevant elements of the course, and the parts that the learner already knows.

So take 2 courses:

Course A is a traditional course, 90 minutes so rather lengthy. and is costing around 50k.

Course B is an adaptive course that significantly reduces the average time to complete to only 30 minutes, but costs twice as much. Course B covers the same topic as Course A and has at least the same learning transfer.

If you would only look at costs associated with the design and development of the course, the financial case for adaptive learning that is twice as expensive as traditional learning will be hard to make. Even if you would be able to quantify the improved learner experience and learning transfer.

If you include the costs related to people spending their time completing the training, the picture will be very different. Assume that each hour spent costs the company 75,- you will see that the increased initial investment makes a big difference with low completion numbers, but that difference will reduce with more completions until it reaches a point where the total costs of course A and B are the same. In the example below that point was reached in May. When completions continue to increase, the adaptive course will become overall cheaper than the traditional course.

Scrap Learning

There’s more on time…. We still have a tendency as L&D departments to mainly push learning to employees. I admit, some of the push learning is related to compliance or other mandatory subjects and had a valid justification. But a lot of push training goes well beyond that. We simply love to define for others what they should learn in order to be successful in their role.

So we go off designing roles and defining exactly what skills people require for these roles and what learning content relates to these skills. But then we forget to take into consideration that people already have knowledge and skills. So in essence we push training to employees on things they already know, or on skills they already have. In addition, we assume all these skills are required for the role. But we have no actual evidence that that is the case. So chances are that we push learning to employees that they do not apply on their job. This is also referred to as ‘scrap learning’: “it was fun, the lunch was very nice, but I can’t actually use any of this in my role”. A somewhat dated article from 2016 by ATD refers to a study that shows that scrap learning could be as much as 85%, and where CEB (now Gartner) sets it at 45%. The interesting observation from the calculation of how much a 45% scrap learning rate costs per employee, is that the highest cost is related to the ‘cost of time wasted’, NOT on the L&D spend

We also love to create big training events. elearnings of 1 or several hours, a few days of classroom training, you name it. Within these long trainings, we have the tendency to cover so much content, that the chances of a learner spending reading, practicing or testing knowledge and skills they already have is almost guaranteed to be a 100%. I’ve earlier argued why we should create more modular and bit sized learning content (see the article on the Lego Caste Paradox) to (amongst other benefits) prevent people going over their 6th introduction of machine learning (“what is machine learning and why is it important to know how machine learning works”).

If you do not capture and measure learning hours, you will not be able to analyze, track and improve the element of time and cost of time wasted by scrap learning even if it make up the biggest part of the costs….

3. Time as strategic advantage

My 3rd and last reason why I think measuring, tracking and analyzing learning hours is critical and should be part of any L&D strategy: The time it takes to upskill, reskill or in any way or form enable employee to perform their role correctly and safely. Sometimes this is referred to as ‘time to autonomy’, or ‘time to proficiency’.

With all the discussions on the ‘future of work‘, before or after Covid, and the ‘future of skills‘, and the crisis emerging to build the much needed new skills, I see 2 strategic challenges for any company that wants to stay relevant:

1. Being able to build skills good and fast
2. Being able to adjust and change what skills are build in the organization to adapt to emerging needs

So any company that can build new skills fast, will have a strategic advantage over companies who take longer to build these same new skills. That I think is also the reason why companies like amazon are making skills building a strategy (“educate 29 million people by 2025“): They take control over the skills pipeline and will be able to pick out individuals who build these skills faster and better than others

So when L&D plays a crucial role in building these new skills, it’s worthwhile tracking how much time on average it takes employees to go to the next proficiency level of each (strategic) skill. Imagine it takes your company 80 hours to build ‘data interpretation’ skills from a beginner to an intermediate level, while your biggest competitor can build it in 60 hours. Not only will you competitor spend less, it will also be able to reap the benefits of this new skill much faster that you. It could even go to the level where top talent joins your competitor as they, on a personal level, can skill up faster.

One of the challenges on tracking upskilling and reskilling is that skills are not the same. Excel skills are ‘easier’ to learn than Python. So each approach to start measuring this would require a sufficient level of granularity of data and insights to better understand where your time to upskill is solid, and where you time to upskill is in need of improvement.

Without using a time or effort related metric like ‘learning hours’, or ‘time spend on learning’, you will be completely in the dark around how well and fast you are upskilling or reskilling. Without such a metric you will not be able to claim any successes when you manage to do it well and faster than anyone else, nor will you be able to identify area’s for improvement and take corrective actions when you are falling behind.

Measuring time and effort

By the way, I would always recommend to measure both effort and time for the simple reason that if not carefully planned, even low effort learning experience can be spread over too much time which will not just impact your ‘speed’ to upskill, but most likely also reduce any transfer of learning due to the Ebbinghaus Forgetting Curve!

Conclusion

No, Learning Hours are for sure NOT the ultimate KPI for learning. But that does not mean we should just ignore it for 2 good reasons. One reason is that learning hours brings the time element in our conversations. This enables us to have a much more balanced discussion around investment that also include the time and effort participants invest in learning experiences at a time when ‘time’ is a most valuable asset. The second reason is that in todays extreme dynamic business environment the ability to upskill and reskill faster and better that the competition is a strategic advantage.

But before we dive into that topic a little bit more. Next we’ll cover another popular metric in learning: Engagement. We’ll explore if engagement could be the ultimate KPI for L&D!