The shrinking denominator enhances the effect size and misleads the reader into thinking the effect is better than it actually is.

Carl Heneghan

---

I am at the Ideal Conference, at a workshop talking about clinical trial reporting and recent surgical scandals.  I chose the 17 years follow-up of the Tension-free Vaginal Tape – often quoted – as one example of the significant problems with long-term outcome reporting.

Here is what the Oxford Clinical Commissioning Group report: ‘it confers a high prospect of long term cure (published 17 year data shows 70-80% long term efficacy) with a low risk of complication;’  the Cochrane review on mid-urethral review reports: ‘Observational studies of MUS show data confirming effectiveness in the long term with some data that cover 15 to 17 years’ and a literature review on De Novo Urgency reports: ‘the reported success rates for this procedure are high, results reported at seventeen year’s follow up indicate a 90% objective cure.’

Sound good, doesn’t it?

The original trial was done in three Nordic centres, and upon reading the trial publications closely (I found reports at 5,7,11 and 17 years of follow-up) what you find is a small study of 90 women and a denominator that shrinks over time. The shrinking denominator enhances the effect size and misleads the reader into thinking the effect is better than it actually is.

 

 

Here’s the summary of the four trial publications with 5, 7, 11 and 17 years of follow-up.

2005 publication (Five-year follow-up first published as a supplement in 2001)

The study population consisted of 90 consecutive patients who were enrolled in a prospective multicenter trial in three Nordic centres between 1^st January 1995 and 15^th  October 1996. Of these 90 women, 85 (%) could be evaluated after five years according to the protocol (five had to be interviewed by phone) 72 of the 85 patients were assessed (85%) as cured; 9 patients (10.6%) were significantly improved and 4 (4.7%) were regarded as failures.

2007 publication (Seven-year follow up):

The study population consisted of the same 90 women, but now only 80 (89%) were followed up: 3 had died, 6 lived in nursing homes, being disabled to the degree that they could not be evaluated, and 1 woman was completely lost to follow-up. You could have reported this results as 10% of women had died or been fatally disabled; but instead, objective and subjective cure rates were reported as 81%.

2008 publication (Eleven Year follow up):

One year later the eleven-year follow-up was published with the same 90 women. Now only 69 (77 %) women could be assessed: 53 women were seen at the clinics and 16 contacted outside the clinics, and 77% (53/69) regarded themselves as cured.

2013 (Seventeen years follow up):

The study consisted of the 90 women, but only 58 (64%) women could be invited to visit the clinics, and 46 women were assessed according to the clinical protocol.  But, look what happens. Suddenly cure rates get better: 42 out of 46 women (91.3 %) stated they were cured. This is backed up in the abstract, which reports: ‘Over 90 % of the women were objectively continent.’

The shrinking denominator

The shrinking denominator effect happens when you fail to include all of the patients in the analysis, an intention to treat (ITT) analysis would include all 90 women. In an ITT  analysis, the results are based on all those initially assigned to the intervention and it leads to smaller effects because the denominator is often much larger than the group followed up. With all the data less than half of the women (42/90; 47%) would be cured.

---

References:

Mid-urethral sling operations for stress urinary incontinence in women. Ford AA, Rogerson L, Cody JD, Aluko P, Ogah JA. Cochrane Database Syst Rev. 2017 Jul 31;7:CD006375. doi: 10.1002/14651858.CD006375.pub4. 

Seventeen years’ follow-up of the tension-free vaginal tape procedure for female stress urinary incontinence. Nilsson CG, Palva K, Aarnio R, Morcos E, Falconer C. Int Urogynecol J. 2013 Aug;24(8):1265-9. doi: 10.1007/s00192-013-2090-2.