Have you ever been involved with a debate with a partner or colleague, travelling from one place to another, and when the course they took has got you to the destination safely, they turn to you and say “So, [add endearment here], you see my way was right.”? If you have, I doubt that you took the opportunity to explain that they may be suffering from a methodological reasoning problem, of which the conterfactual argument and the possibility of differential verification bias may be important to consider. If you’re anything like me, you obediently reply “Yes, dear.” (Occasionally, this can be problematic at work, when it’s not your life-partner that has used the appropriate trigger phrase.)

The problems alluded to are really quite simple, but sound fancy. If you show that one course of action (such as undertaking surgery for appendiceal masses) has good outcomes, you have not shown that undertaking any other action (such as leaving them alone) would not have a good outcome. This is known as a ‘counterfactual’ argument, and is part of the reason why uncontrolled studies can encourage the continuation of unhelpful practices and do far more harm than good.

Differential verification bias is the problem when applied to the setting of diagnostics. If you only do radiographs on children who are tachynpnoic, and with this demonstrate on a few a lobar consolidation, you can’t reasonably turn around and say “So if you’ve not got a raised respiratory rate, you’ve not got pneumonia”. (Well, on the basis of this type of study you can’t.) Watch out for this sort of problem when you’re reading case series and case reports, where the illusion of medical hypotheses may mask a simpler truth: doing nothing is often quite effective and many tests are done for us, not the patient.

Acknowledgement: Photo from Andy Beez, CreativeCommons