Technology · · 7 min read

Learning analytics: what the numbers from your LMS do and do not say

Practically every learning platform reports completion. That is the figure that says least. Four measurements that do mean something, and the reversal nobody makes.

M

Milan Stark

Business development, STARK Learning

Hand operating a large touchscreen full of settings and readings

What gets measured now

Open any learning platform and the first thing you see is a completion rate. Green above ninety, amber below. That is what gets steered on, that is what gets reported, and that figure travels to the board.

Completion measures whether somebody clicked through the material. It goes no further. Somebody who opened the module, went to work in another tab and guessed four multiple-choice questions correctly at the end shows green.

That does not make the figure worthless. It makes it an attendance record, and you should not confuse that with a measurement of competence.

Four figures that do say something

Error patterns per question, across the whole group. This is the most valuable number in your entire platform and it is practically never looked at. If eighty percent get the same question wrong, the problem is not eighty percent of your people.

Time per component, set against the expectation. A module everybody takes twice as long over as intended is too difficult or unclear. One everybody flies through is not being read.

Repeated failure on the same component. Once wrong is noise. Three times wrong on the same theme, spread over weeks, is a signal the knowledge has not landed.

Decay over time. Measure the same knowledge one month and six months after the training. That is the only measurement that says anything about whether something genuinely stuck, and with a short assessment it is cheap to do.

What xAPI adds

SCORM, the standard most e-learning runs on, reports a handful of things: started, completed, score, time. That is enough for an attendance record and too little for the four figures above.

xAPI captures individual events. Which choice somebody made in a scenario, where they jumped back, how long they hesitated, which step they skipped. That makes behaviour visible rather than only the outcome, and it works outside the learning platform as well: an operation in VR, a scanned instruction at a machine, an exercise on the work floor.

The flip side is that xAPI gives you data and not answers. Without a question thought out in advance, it produces a mountain of data nobody does anything with. So decide first what you want to know, and set up the measurement after that. The other way round does not work.

The reversal nobody makes

The return that almost always goes unclaimed sits here.

Learning data gets read as a verdict on participants: who passed, who still has to. The same data is a measurement of your own material, and in that direction it is considerably more useful.

A question that goes wrong en masse points at a module that explains the subject badly, or at a procedure that is itself unclear. A component everybody gets stuck on is usually a design fault. A module nobody finishes is too long.

Anyone who looks at it that way gains a maintenance instrument. Not: which people score low. But: which three modules cost us most in misunderstanding, and which do we repair first?

Privacy and the works council

As soon as you measure more than completion, this concerns personal data and the behaviour of employees. That is not a reason to avoid it, and it is a reason to arrange it in advance.

Three agreements usually suffice. Decide which data is visible at individual level and which only in aggregate, and keep the second as broad as possible. Record how long you retain it. And agree what the data will not be used for, particularly in appraisals, because the moment people suspect a practice assessment counts, their behaviour changes and the measurement is worthless.

That agreement is not only legally tidy. It is the condition under which the numbers stay accurate.

How measurement fits into a training programme as a whole is in the guide on setting up or renewing a corporate academy. What a platform has to be able to do for it is under learning management.

Do you know whether your training works, or only that it was completed?

Tell us what's going on, we'll help you think through an approach that fits your team and facility.

M

Milan

Business development, STARK Learning