The shrinking denominator enhances the effect size and misleads the reader into thinking the effect is better than it actually is.
I am at the Ideal Conference, at a workshop talking about clinical trial reporting and recent surgical scandals. I chose the 17 years follow-up of the Tension-free Vaginal Tape – often quoted – as one example of the significant problems with long-term outcome reporting.
Here is what the Oxford Clinical Commissioning Group report: ‘it confers a high prospect of long term cure (published 17 year data shows 70-80% long term efficacy) with a low risk of complication;’ the Cochrane review on mid-urethral review reports: ‘Observational studies of MUS show data confirming effectiveness in the long term with some data that cover 15 to 17 years’ and a literature review on De Novo Urgency reports: ‘the reported success rates for this procedure are high, results reported at seventeen year’s follow up indicate a 90% objective cure.’
Sound good, doesn’t it?
The original trial was done in three Nordic centres, and upon reading the trial publications closely (I found reports at 5,7,11 and 17 years of follow-up) what you find is a small study of 90 women and a denominator that shrinks over time. The shrinking denominator enhances the effect size and misleads the reader into thinking the effect is better than it actually is.
Here’s the summary of the four trial publications with 5, 7, 11 and 17 years of follow-up.
2005 publication (Five-year follow-up first published as a supplement in 2001)
The study population consisted of 90 consecutive patients who were enrolled in a prospective multicenter trial in three Nordic centres between 1st January 1995 and 15th October 1996. Of these 90 women, 85 (%) could be evaluated after five years according to the protocol (five had to be interviewed by phone) 72 of the 85 patients were assessed (85%) as cured; 9 patients (10.6%) were significantly improved and 4 (4.7%) were regarded as failures.
2007 publication (Seven-year follow up):
The study population consisted of the same 90 women, but now only 80 (89%) were followed up: 3 had died, 6 lived in nursing homes, being disabled to the degree that they could not be evaluated, and 1 woman was completely lost to follow-up. You could have reported this results as 10% of women had died or been fatally disabled; but instead, objective and subjective cure rates were reported as 81%.
2008 publication (Eleven Year follow up):
One year later the eleven-year follow-up was published with the same 90 women. Now only 69 (77 %) women could be assessed: 53 women were seen at the clinics and 16 contacted outside the clinics, and 77% (53/69) regarded themselves as cured.
2013 (Seventeen years follow up):
The study consisted of the 90 women, but only 58 (64%) women could be invited to visit the clinics, and 46 women were assessed according to the clinical protocol. But, look what happens. Suddenly cure rates get better: 42 out of 46 women (91.3 %) stated they were cured. This is backed up in the abstract, which reports: ‘Over 90 % of the women were objectively continent.’
The shrinking denominator
The shrinking denominator effect happens when you fail to include all of the patients in the analysis, an intention to treat (ITT) analysis would include all 90 women. In an ITT analysis, the results are based on all those initially assigned to the intervention and it leads to smaller effects because the denominator is often much larger than the group followed up. With all the data less than half of the women (42/90; 47%) would be cured.
Mid-urethral sling operations for stress urinary incontinence in women. Ford AA, Rogerson L, Cody JD, Aluko P, Ogah JA. Cochrane Database Syst Rev. 2017 Jul 31;7:CD006375. doi: 10.1002/14651858.CD006375.pub4.
Seventeen years’ follow-up of the tension-free vaginal tape procedure for female stress urinary incontinence. Nilsson CG, Palva K, Aarnio R, Morcos E, Falconer C. Int Urogynecol J. 2013 Aug;24(8):1265-9. doi: 10.1007/s00192-013-2090-2.
Eleven years prospective follow-up of the tension-free vaginal tape procedure for treatment of stress urinary incontinence. Nilsson CG, Palva K, Rezapour M, Falconer C. Int Urogynecol J Pelvic Floor Dysfunct. 2008 Aug;19(8):1043-7. doi: 10.1007/s00192-008-0666-z.
Seven-year follow-up of the tension-free vaginal tape procedure for treatment of urinary incontinence. Nilsson CG, Falconer C, Rezapour M. Obstet Gynecol. 2004 Dec;104(6):1259-62.
Long-term results of the tension-free vaginal tape (TVT) procedure for surgical treatment of female stress urinary incontinence. Nilsson CG, Kuuva N, Falconer C, Rezapour M, Ulmsten U. Int Urogynecol J Pelvic Floor Dysfunct. 2001;12 Suppl 2:S5-8.
BMJ Evidence-Based Medicine – original evidence-based research, insights and opinion
Read more in the Welcome to BMJ Evidence-Based Medicine Editorial.
Carl has received expenses and fees for his media work including BBC Inside Health. He holds grant funding from the NIHR, the NIHR School of Primary Care Research, The NIHR Oxford BRC and the WHO. He has also received income from the publication of a series of toolkit books. CEBM jointly runs the EvidenceLive Conference with the BMJ and the Overdiagnosis Conference with some international partners which are based on a non-profit model. he is a clinical advisor to the UK Gov’ts All Party Parliamentary Group (APPG) on surgical mesh.