This is what is meant when referring to ABSENCE OF EVIDENCE IS NOT EVIDENCE OF ABSENCE.
Rafael Perera
In science, it is difficult to show that something is so – to prove something. Instead, we deal with hypotheses and theories. We gather evidence and compare our hypotheses to the data collected to decide which is more likely.
The basic setup is to have two competing hypotheses, a null and an alternative, and, after a study is completed, decide which one is more likely. We can make mistakes when determining which one is more likely but statistics allow the probability of making a mistake to be quantified.
This setup is used in medical research to compare two treatments. The typical null hypothesis is: both treatments ARE the same; while the alternative is they are NOT the same. If the data collected is consistent with the null then the only thing we can say is that we cannot reject that the two treatments ARE the same. However, this does not mean we have collected evidence to support the statement: the two treatments ARE the same.
This seems paradoxical, but it is easily explained by a quick thought experiment. Imagine a very effective treatment that cures 50% of the cases compared to one that is not effective at all. A very small trial, 4 in each arm, identifies two recovered in the very effective arm and none in the not-effective arm. Although the difference in effect is large, the size of the study prevents us from finding that a difference exists and instead we would say that the results are consistent with the two interventions being the same – the null hypothesis.
This is what is meant when referring to ABSENCE OF EVIDENCE IS NOT EVIDENCE OF ABSENCE.
There are, however, situations where the primary objective is to gather evidence of no effect, that is, evidence of absence. This could be because we want to replace one treatment with another. We might not believe the new treatment to be better clinically, but instead, it might have other advantages. It could be cheaper, produce fewer side effects, require less monitoring, be less invasive, or be straightforward to deliver. In their pure form, these trials are called EQUIVALENCE TRIALS.
In an Equivalence trial, the null hypothesis is that there is a difference between the two interventions greater than a certain value (delta), while the alternative is that the difference is less than that value. The selection of delta is critical as it defines the tradeoff we are prepared to make in choosing between treatments. The size of this delta will determine how much evidence we need to collect to define if there truly is no difference; with more evidence needed as the chosen delta gets smaller.
Because of the size of the delta, equivalence trials sometimes need bigger sample sizes. But I’ll save that explanation for my next blog
Rafael Perera is Professor of Medical Statistics at the Nuffield Dept of Primary Care Health Sciences, University of Oxford, and Associate Editor of the BMJ EBM Evidence-Based Medicine journal.
Competing interests: none declared