Liz Wager: Was it worth missing a bus for?

Liz WagerYesterday, I received some great feedback about a workshop I ran. Sorry if this sounds horribly self-congratulatory, but I’d like to share it with you. At the start of the workshop, one participant politely explained that she couldn’t stay for the whole session as she had a plane to catch. This struck me as a pretty good reason for an early departure and I was glad she had taken the trouble to tell me rather than just stalking out without an explanation making me wonder if I had offended her or simply bored her to despair. She also added, apologetically, that she was a nervous traveller and preferred to get to the airport several hours ahead of her flight rather than risk being late or arriving in a rush.

I was therefore surprised to see her at the end of the session. Having forgotten to bring formal evaluation forms, I did at least remember to ask the participants if they had found the workshop interesting and helpful. Smiling, the anxious traveller simply said ‘Definitely — I decided to catch a later bus’. Next time (if I remember to produce an evaluation form) I think I’ll replace boring Likert scales (please rate the course from 0-5) with some new end-points: Please rate the workshop on a scale of ‘a total waste of time, I wish I’d been somewhere else’ to ‘worth missing a bus for’.

On a slightly more serious and less promotional note, this made me think how unimaginative most rating scales are. How on earth do we know whether one patient’s score of 3 on a 0-5 scale of how they are feeling is the same as the next patient’s 3? Yet simple numerical scales are common, especially for measuring symptoms such as fatigue or pain. ‘Good enough to miss a bus for’ may not be particularly helpful for clinical ratings but maybe similar concrete examples would help.

About Liz Wager

Liz Wager is a freelance writer, trainer and publications consultant who works
for a number of pharmaceutical companies, communication agencies, publishers and
academic institutions. She is also the Secretary of COPE (the Committee On
Publication Ethics) and a member of the BMJ’s Ethics Committee.

  • Vivek Dhungana

    Dear Liz,

    The post you have come up with really brings new ideas and sounds interesting. If not uniformly, then also customising the scales to let the assessor better know its meaning will be helpful. After all, 3/5 in muscle power grading is NOT equal to 3/5 of pain; meaning that even then it was really developed to scale but in a customised manner for individual purpose, isn’t it?

    Good work!

    Vivek Dhungana
    KMC Teaching Hospital
    Asst Editor, KUMJ

  • Dear Liz,

    I am so glad you are wondering about measuring people’s responses. Of course, patient-reported outcomes, or PROs, have become a really important type of outcome measurement in modern clinical research. Fortunately, there is a whole field of scientific endeavour devoted to making PROs more objective, quantitative and valid. Much is known about types of rating scales, and you are right, categorical rating scales with descriptions generally function better than simply numerical rating scales. But that doesn’t really solve the main problems which are: how do you know how may categories people can actually use, and what value should you assign these categories, or if you are adding up multiple questions, what value should each question have in an overall score? Fortunately, item response theory (IRT) can answer these questions and turn simple categorical responses into true linear measurement. Questionnaires developed using IRT, particularly Rasch analysis are becoming increasingly used in clinical research leading to improved measurement of the patient’s perspective.

    A/Prof Konrad Pesudovs
    NHMRC Centre for Clinical Eye Research
    Flinders Medical Centre
    Bedford Park SA