Ask Why before Measuring

Posted by

Recently I had the following scenario put forward to me:

If someone asked you: “How do I get started thinking about and moving towards a measurement-based approach to training?” How would you answer them?


My first question to them would be “Why?”
“Why are you feeling the need to move towards a measurement-based approach to training?”
To be honest, it’s a trick question.  Seeking to have a measurement-based approach to training is wrongly focused.  Training (and all learning) should be focused on improving specific and measurable organizational and/or individual performance.  Measurement (and data collection, metrics, and learning analytics) is in service of a performance-based, data-driven approach to training and learning.
However, there is value in the question of why a measurement-based approach to training is desired.  The answers, some worse than others, will reveal how much work is to do be done.

Nice Try, But Try Again

The really bad answers will be along the lines of:

  • My supervisor asked me to look into it
  • Every other department has great looking graphs and charts
  • At DevLearn/ATD-ICE/HR Tech Conference everyone was talking about measurement and metrics
  • xAPI enables us to measure all kinds of

The problem with these answers should be obvious.  Taking on any initiative to “keep up with the Joneses” or to meet an unrationalized task given by a supervisor is a waste of time and money.  Without a purpose, you’re simply taking stabs in the dark hoping to land on something valuable.   My response to these answers is “don’t waste your time. ”
Go back to the drawing board and determine if there is a business need to drive your exploration of measurement or forget about the initiative completely.  With no real rationale, it’s doomed to failure.

That’s Better, but…

There is a set of “middle ground” answers that are headed in the right direction.  But yet they fall short of providing an answer to the question “Why?”

  • The stakeholders for our new sales training want to measure the success of our solution by sales metrics (percentage of deals closed in first three months, overall increase of sales per representative)
  • We know we aren’t capturing Fitzpatrick Level 3 and 4 data and have to figure out how to do that quickly.
  • Finance won’t accept our budget requests without an explanation of how we are going to determine if each program is meeting its operational and financial goals.
  • We keep reading industry reports that say senior leadership of most organizations feel they aren’t getting adequate data from learning and development.  We approached our senior management and found out they feel this way as well.  But we don’t know what data we need to gather to satisfy them.

At this level of understanding,  the push for measurement is coming from external (to L&D) agents – various stakeholders, industry thought leadership,  organizational gatekeepers (finance/IT), and others.  These answers still reflect a reactionary stance regarding how we report on our initiatives.  Input from these external agents is important.  But it should be input, not strategic direction.  We need to synthesize this input and build a coherent and achievable strategy for projects and learning as a whole.

This may seem a little overblown, but generally, it’s not.  We are just at the beginning of the transformation of L&D to being evidence-based and data-driven.  Most of us don’t yet understand the nuances of performance measurement tied to business objectives

Purposeful Strategy

Asking “why?”  to this point in answering the original question has been posed in order to identify 1) a lack of true business goals, 2) a scattered, unfocused approach to data collection and measurement, and 3) to unearth the potential resources and roadblocks to performance-based, data-driven reporting and decision making.

But the real “why?” (or why’s) gets at the heart of the purpose of each initiative and the desired change in organizational and/or individual performance.  Once this purpose is fully understood and a preliminary learning strategy and supporting measurement strategy can be developed.  Data collection, measurement, metrics, performance evaluation, and learning analytics are in support of the overall learning strategy. They are the means to an end, not an end in themselves.

It Comes Down to the Data

With all of this said, I still haven’t answered the original question, which re-written to address my initial concern would be:

If someone asked you: “How do I get started thinking about and moving towards a performance-based, data-driven decision and reporting system to support learning here at XYZ Corporation?” How would you answer them?

In general, I advocate choosing one or two new projects that are small-to-medium in scale to serve as a pilot and/or guinea pig.  One, if something goes haywire, it will have less of an impact. Two, you’ll be able to cycle through it faster – validating your new approach more quickly – so you can replicate your success rapidly.

On a more specific level, my response would be: Do you have a Data Strategy?

Do you know the answers to the following questions?

  1. Figure out who are the stakeholders in your project’s success. What role will they want to play in the project? Who is ultimately responsible for achieving the desired performance change?
  2. Make sure that the requested intervention has a clearly identifiable expected impact upon the business. If the stakeholders can’t define the benefit the change will have on the business, how will you create appropriate learning or performance management experiences?
  3. Understand how the stakeholders for this intervention expect to determine the project’s success or failure. What is the measure of success? Some may be qualitative.  Some may be quantitative.
  4. With your stakeholders, determine how each measure of success should be measured and set a SMART goal. If the measure of success is “increased sales” will it be measured by region? individual? company wide? Will it be recorded in units? currency? signed contracts?  Will the goal be an increase? a raw number?  a percentage over last period? 
  5. Determine the specific data that is needed to complete each measurement. What is it? Where in the process would it occur? Is it quantitative or qualitative? How might you measure it? Does the measure of success have component points of data that much be collected separately then calculated together? This should be done in an “ideal world” exercise.  Don’t worry about technologies, policies, collection methodologies, etc. at this point. What data would you need to provide the best information possible?

At this point, if not before, I’d stop and let them know that while there is much more beyond this, I’m guessing that their head is reeling.  My point is, it all has to begin with the business purpose behind the initiative which needs to be analyzed down to the data points needed.

The five steps I’ve outlined above, are required if you want to establish valid measurements that meet the business objective(s) of the initiative.  You will have powerful stakeholder buy-in and a foundation for valid and accepted reporting.  You’ll have the basis for ROI figures that are supported by senior management.  You also will be on your way to becoming a trusted business partner.

Determining the data you need to collect before you begin the design phase of your project is crucial, otherwise your design may leave out critical moments that are needed to generate the correct data.

SECRET: You Already Know How to Do This

You are doing stakeholder analysis already.  You’re already talking to your stakeholders doing a needs analysis, you’ll just add a few questions about their dreams and aspirations (and ask them to quantify them if they haven’t already).  You do task analysis of the process to be taught. You’ll add a few columns to your task analysis table for information regarding the related data.  You already know how to set SMART goals.

What do you think? Is this doable? Do you agree that sweating the data is worth it?  What would you change?  Why?  Please add to the conversation in the comment section below.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s