Approach · · 7 min read

Assessing what somebody can actually do: from knowledge question to competence

An assessment that measures whether somebody clicked through the module measures nothing. Here is the difference between knowing and being able, and how to assess the second.

M

Milan Stark

Business development, STARK Learning

Operator working the panel of a CNC machine

The tick-box problem

The standard assessment at the end of an e-learning module has four multiple-choice questions whose correct answer jumps straight out. Everybody passes, the record goes green, and nobody knows more than they did before the module.

That is not only a waste of time. It is actively damaging, because the organisation now believes the matter is settled. At an audit there is a list showing a hundred percent passed, and that list gives false confidence about precisely the subject where things go wrong.

Three levels of assessment

It helps to separate what you are measuring.

Knowing. Does somebody know the fact, the standard, the limit value? Multiple-choice questions are fine for this, and this level is necessary. You cannot act without knowing. It is simply rarely sufficient.

Deciding. Does somebody choose the right action in a concrete situation? You assess this with scenarios: a situation with incomplete information, several defensible options and a consequence. This is the level most industrial training questions belong at, and the level most often skipped.

Doing. Does somebody carry the operation out correctly? This cannot be assessed digitally. It calls for observation at the workplace by somebody who knows what to watch for.

A mature assessment design uses all three, in that order, and limits the third to the operations that genuinely matter.

Good distractors make the assessment

With scenario questions, the quality sits entirely in the wrong answers.

A question with one plausible and three nonsensical options tests whether somebody can read. Put up four options an experienced colleague could each defend, with one of them just slightly better, and you are testing judgement.

You do not invent those distractors behind a desk. They come out of incident reports and out of conversations with specialists: which choice did the person who got it wrong make, and why did it look logical at that moment? That is the distractor that works.

What you record

For demonstrability you need more than pass or fail.

Record per person which version of the material they worked through; at an audit after a process change, that is the question that gets asked. Record which components somebody got wrong, not only the final mark. And look at patterns across the group: if eighty percent get the same question wrong, that is not a knowledge problem in eighty percent of your people but a problem with the procedure or with how it was explained.

That last point is the most underused return on assessment. The scores are not only a verdict on the participants; they are a measurement of your own material and sometimes of your own procedures. Anyone who looks at them that way gets more out of them than a certificate.

Assessment is the closing piece of a programme, and it only works when the rest in front of it holds. What that build-up looks like is in the guide on setting up or renewing a corporate academy. What you need to be able to get out of a learning platform for it is under learning management.

Do you want assessment that tells you something?

Tell us what's going on, we'll help you think through an approach that fits your team and facility.

M

Milan

Business development, STARK Learning