How to Measure the Success of Training
It is long. It is dry as dust. And it is outstandingly boring. But it’s a classic: Donald Kirkpatrick’s Evaluating Training Programs. And it’s a classic for a reason! Together with Will Thalheimer’s Learning-Transfer Evaluation Method, Robert Brinkerhoff’s Success Case Method and Jack Phillips’s Five Levels of Training Evaluation Model, Kirkpatrick’s Four Levels model remains the fundamental evaluation method for training – it has been for decades, and it’s here to stay. So, let’s take a detailed look at how the model has evolved over the years.
According to Donald Kirkpatrick, effective training evaluation is based on four levels:
- Reaction
- Learning
- Behavior
- Results
Every supposedly new evaluation method in the management and training literature, no matter how sophisticated, can be placed somewhere here – or between – the four foundations. However, the order and depth in which they should be applied is still a matter of debate today (more on this in the last paragraph).
First to the details of the Four Levels: What exactly has to be done? How does measurement work?
- Measuring reaction – This essentially means assessing learner satisfaction: After a training session, ask the trainer and the trainee how it was – at best so that you can compare the results of the surveys afterwards. Yes, exactly, it’s like the smileys you can press at freeway service stations when leaving the toilet.
- Measuring learning – This entails assessing the extent to which participants changed their attitudes, or gained greater skills or knowledge. That’s already a little harder, but you can evaluate the levels of knowledge, skills or attitudes before training, and again afterwards to get a good picture. If practical, use a control group of people who have not yet received training so you can measure changes in the people who participated, compared to those who did not.
- Measuring behavior – Evaluate how participants’ behavior changes in the workplace. Same here: Assess the behavior before and after the training (but remember to let the learnings sink in first), and use a control group for comparison. You should also survey or interview trainees, their immediate supervisors, their subordinates, or anyone else who observes their behavior. Get a 100% response if you can, or use sampling. Tip: Repeat the evaluation at appropriate intervals.
- Measuring results – Evaluate organizational results, such as increases in production, improved quality, decreased costs, reduced frequency or severity of accidents, increased sales, reduced turnover, heightened regulatory compliance, or higher profits. Often these results are why your company wanted training programs in the first place, so you state training’s final objectives in these terms, based on corporate priorities. These results can be more difficult to measure, since many other internal (i.e., turnover, etc.) and external influences (i.e., market pressures, etc.) can affect them. However, link what happened in training to these broader organizational goals as much as you can.
The “Results” part has evolved the fastest over time, as all possible returns on investment (ROI) are now often easier to track in terms of data than in the past. The work of Jack Phillips started here and refined Kirkpatrick’s ideas. But read for yourself:
Related Summaries in getAbstract’s Library
Book Summary
Return on Investment in Training and Performance Improvement Programs
Butterworth-Heinemann Read Summary
Implementing the Four Levels
In many companies today, employees run from one training seminar to the next. That’s because executives hire high-powered consultants who recommend training programs to improve performance and bolster profits. However, without the proper philosophical approach and practical execution, training can have negligible bottom-line results, and the only one who really profits is the consultant. That’s why, in 2005, Kirkpatrick and his son broadened the view and addressed entire companies with the Four Levels, not just their training executives and their peers.
They wanted to show that the evaluation methods are not only limited to the training sector – and that isolated and solitary application does not bring what companies expect from the application. Back then, it became clear that excessive enthusiasm for measurement can also have negative effects, and is capable of destroying entire corporate cultures. Therefore, your take-aways are:
- Don’t believe everything consultants say. (In 2005, this seemed a revolutionary insight.)
- Give employees a framework, but allow for creativity.
- Bring high-level executives on board during planning and execution of trainings to change behavior.
- Create an environment that encourages learning.
- Put the right people in key slots.
- Be sure your training program isn’t too big or complex; know where it is heading and stay in touch with its basic objective: altering behavior.
- Provide up-to-date gear.
- Even with all that in place, do not expect behavioral changes without incentives, and don’t overemphasize supporting workers at the cost of measurable bottom-line results.
- Once a program works, monitor it constantly.
When it comes to the details, of course, Kirkpatrick has stepped it up too. In Implementing the Four Levels, he digs deeper and explains for each of the above take-aways how they can be integrated into processes, and also provides plenty of case studies:
Aligning Training Programs with Business Objectives
In 2010, Kirkpatrick’s son and daughter-in-law, James D. Kirkpatrick and Wendy Kayser Kirkpatrick deployed a seven-part program that training professionals can implement to align their training programs with their company’s business objectives. They did so for a specific reason: Research results proved over and over again that training does not improve the bottom line. Also, anecdotal evidence often lines up against the business value of corporate learning and training – everyone has their story to tell about bad training and the enormous waste of time associated with it.
So, the Kirkpatricks addressed the underlying question in Training on Trial: Why doesn’t training achieve strong business results? The answer: Often, training professionals fail to prepare participants for their upcoming training and what it will entail. Additionally, training professionals do little to reinforce the skills participants learned during training. Many training professionals do not demonstrate the inherent value of their programs. That’s why training professionals must develop a “chain of evidence” that demonstrates that training is valuable in developing necessary skills and helps the business grow. They have to:
- Present the value of training to key stakeholders as a defense attorney presents evidence to a jury.
- If you prove training’s merit to one business group, other groups will want it.
- Discover what stakeholders expect from training, then work to fulfill their expectations.
- Proactively demonstrate training’s worth to the bottom line; don’t wait to be asked to prove it.
- Solicit feedback from all training participants and use it as evidence.
The Misunderstanding of an “Industry Standard”
In the meantime, the Four Levels became the global training industry’s standard. However, those using Kirkpatrick’s work have misunderstood and misapplied it for decades according to James D. and Wendy in her most recent work:
We have observed that many training professionals say they are ‘using Kirkpatrick,’ yet are following dated practices that are failing to create and demonstrate organizational value with their training.
James D. and Wendy Kirkpatrick
The prevailing inability of most organizations to recognize the value of training and development programs appears to them as a result of a big misunderstanding.
It made them revisit the model – and they turned it on its head with the New World Kirkpatrick Model (NWKM). The NWKM urges organizations to start at Level 4 and work back (remember to apply each of the levels! Don’t, for example, attempt to evaluate at Level 4 if you have only collected data for Levels 1 and 2.)
This is how it works:
- Level 4: Results – The identified business benefit derives from the training and related follow-up activities and support. Identify a mission-level objective, such as providing an industry-leading level of customer service. Then, work backward to determine the metrics that drive that high-level outcome – for example, customer satisfaction survey results, customer retention and the like.
- Level 3: Behavior – Identify the crucial behaviors learners must adopt to achieve the measures that drive the outcomes in Level 4 (Results). Determine what learners need to exhibit those behaviors – such as supervisory support, incentives, coaching or equipment – and turn them into metrics.
- Level 2: Learning – This level covers the extent to which participants learn what training intends – including new knowledge, abilities, and the mind-set or determination to apply the new learning. Learners should finish the program knowing the subject matter and possessing the skills to apply it. They should believe in the program’s value and commit to using it at work.
- Level 1: Reaction – Participant satisfaction and engagement with the training and perception of its applicability aids in learning. Understandably, most organizations evaluate most of their programs at Level 1. Unfortunately, too many so focus on participant satisfaction they fail to focus on the more important measure of whether participants use the learning.
The detailed benefits of this (new) approach can be found in the summary. For more useful guidance on updating the Kirkpatrick model, see the abstracts below:
Related Summaries in getAbstract’s Library
How the Journal Saves You Time
Reading Time
8
min.
Reading time for this article is about 8 minutes.
Saved Time
66
h
This article saves you up to 66 hours of research and reading time.
Researched Abstracts
12
We have curated the most actionable insights from 12 summaries for this feature.
12
We read and summarized 12 books with 3262 pages for this article.
