Carl Heneghan<\/p>\n
\n
We found oral HPTs in pregnancy was associated with an increased chance of all congenital malformations; congenital heart, nervous system, and musculoskeletal malformations; and congenital <\/span>VACTERL syndrome<\/a><\/p>\n To obtain these results we performed a systematic review and meta-analysis of case-control and cohort studies that included pregnant women exposed to oral HPTs within the first three months of pregnancy.<\/p>\n We assessed quality\u00a0using the Newcastle-Ottawa Scale (NOS)<\/a> for non-randomized studies.\u00a0The NOS assesses quality and risk of bias in observational studies, and it has been validated for case-control <\/a>and longitudinal studies. Whether it was appropriate to use such a method is questionable, because scoring systems to assess quality have previously been criticized<\/a>.<\/p>\n We chose the NOS has it has been used widely. In our systematic review<\/a>, we cite five instances, covering a wide range of exposures and outcomes (s<\/a>econd malignancies<\/a> after radiotherapy for prostate cancer, pregnancy, thrombophilia, <\/a>the risk of a first venous thrombosis, and the relation between low cigarette consumption<\/a> and the risk of coronary heart disease and stroke).<\/p>\n The use of scales or scores in assessing bias fell out of favour, largely,\u00a0 because of the Cochrane handbook<\/a>. Cochrane state it is preferable to use simple approaches for assessing validity that can be fully reported and explicitly discourages scores as there is a strong emphasis on the reporting of the research rather than its conduct. Cochrane further dismisses scales<\/a>\u00a0because the approach is not supported by empirical evidence, they are unreliable (J\u00fcni 1999<\/a>) and because it is often difficult to justify the weights assigned to scores of different items.<\/p>\n Juni<\/a>, assessed three key domains (concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals) in a meta-analysis of 17 comparisons of low-molecular-weight heparin with standard heparin for prevention of postoperative thrombosis. They used 25 different scales to identify high-quality trials and found that the scales all differed in their assessments of quality.<\/p>\n Juni\u00a0state that relevant methodological biases should be assessed individually and their influence on effect sizes explored. However, this is problematic, as, at some point, individual weights will be subjectively assigned by an individual to determine the influence of the biases. They say this because many biases are not supported by empirical evidence, which is true –\u00a0 see The Catalogue of Bias<\/a>.<\/p>\n Part of the problem with early scales was they related to the reporting quality, ethics, or interpretation of the results, rather than the internal validity of the trial. Some scales, for example, asked whether the rationale for conducting the study was clearly stated, whether the trialists’ conclusions were compatible with the results, or whether participants had provided written informed consent. Juni also criticised one widely used scale at the time, the JADAD scale, because it gives more weight to the quality of reporting than to actual methodological quality.<\/p>\n Reporting bias, however, is a significant problem in assessing quality. In our assessment of the risk of bias of industry-funded oseltamivir trials,<\/a> the use of more detailed information, included in clinical study reports, showed that over half (55%, 34\/62) of the previous assessments of \u2018low\u2019 risk of bias were reclassified as \u2018high\u2019.<\/p>\n Strengths and Limitations of\u00a0 the Newcastle Ottawa Scale<\/strong><\/p>\n The NOS has been recommended by the Cochrane Collaboration. One weakness of the scale, though, is the possibility of a low agreement between assessors.<\/p>\n Hartling et al<\/a>\u00a0showed low agreement between two independent reviewers in scoring the NOS. This was particularly the case when authors had limited experience in doing systematic reviews. Training, even of novices<\/a>, can improve agreement, and test-retest reliability of the NOS <\/a>has been shown to be fair to excellent.\u00a0The developers have also examined the scale\u2019s face validity and criterion validity, inter-rater reliability, and evaluator burden.<\/p>\n In terms of the individual elements, the NOS evaluates the selection of study groups, their comparability, and ascertainment of either the exposure for case-control studies or the outcome of interest for cohort studies.<\/p>\n When the Journal of Evidence-Based Medicine<\/a> assessed methodological quality assessment tools for systematic review and meta\u2010analysis, it recommended using the NOS for cohort and case\u2010control studies. Also, the US Preventive Task Force uses the scale\u00a0 (Annals of Internal Medicine<\/a>), and the NOS has recently been used in IPD meta-analysis published in PLOS<\/a>.<\/p>\n Applying the NOS to our systematic review<\/strong><\/p>\n In our\u00a0 systematic review<\/a> of oral hormone pregnancy tests and the risks of congenital malformations, confounding factors are covered in detail in items 3, 4, 5a, and 5b of the NOS:<\/p>\n Item 5 of the NOS score is particularly important, as it addresses comparability of cases and controls based on design or analysis. Of the 16 case-control studies in our systematic review, 12 controlled for the most important factor (item 5a) and nine controlled for important additional factors (item 5b). Of the ten cohort studies, six controlled for item 5a and four controlled for item 5b. The mean NOS score was 6.1, indicating an overall moderate risk of bias. Table 2 in the systematic review also shows that seven studies did not report the confounding variables collected.<\/p>\n Ascertainment of exposure and outcomes identified by the reviewer are also captured in items 6, 7, and 8:<\/p>\n What scale should we use?<\/strong><\/p>\n Most other tools\u00a0 for assessing harms focus on reporting<\/p>\n The Cochrane Methods group<\/a> recommend the ROBINS-I is the preferred tool to be used in Cochrane Reviews for non-randomized studies of interventions. However it is not mandatory, and they further state the NOS is an alternative option.<\/p>\n In their full statement<\/a> they report the currently published ROBINS-I is designed for cohort-like designs, and although it may be applicable for case-control studies, further developments to signalling questions for this design are currently underway but have not yet been published. New guidance to specify the competence level is also in development along with separate software for integration with GRADE; this is not yet available. Until all of this is achieved and tested, we followed the Cochrane recommendation that it is \u2018appropriate to use another tool, such as the currently recommended Newcastle Ottawa Scale.\u2019<\/p>\n A Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies <\/a>reviewed 61 tools. Most tools were not designed to evaluate pharmacoepidemiological safety studies. and there was no specific tool found that ‘is adequately designed for the robust evaluation of pharmacoepidemiological studies of drug safety.\u2019<\/p>\n The Journal of Evidence-Based Medicine<\/a>\u00a0also recommended the Methodological Index for Non-Randomized Studies (MINORS)<\/a>\u00a0as an excellent tool for assessing non-randomized interventional studies in surgery.<\/p>\n What study design is appropriate for assessing harms?<\/strong><\/p>\n Our systematic review<\/a> included case-control and cohort studies that included data from pregnant women exposed to oral HPTs within the estimated first three months of pregnancy.\u00a0The CEBM levels of evidence place<\/a> systematic reviews of randomized trials, systematic reviews of nested case-control studies, n of-1 trial with the patient you are raising the question about, or observational study with dramatic effect as the highest level of evidence to determine an association of harms.<\/p>\n Establishing causal associations in the absence of randomization can be difficult. However, there are situations where randomization is not feasible or ethical. In the case of Primodos, there were already concerns over harms, making it unjustifiable to randomise individuals to such\u00a0a harmful treatment, particularly when there are also no expected benefits of the treatments – Primodas is a test.<\/p>\n Furthermore, as a test, Primodos does not meet the four concerns,\u00a0<\/a>set out by Dave Sackett,\u00a0<\/span>about observational evidence that requires randomised trials to negate them<\/a>. These concerns are:<\/span><\/p>\n It is important to recognise that observational studies can demonstrate associations<\/a> of harms. As an example, the association of maternal stilbestrol therapy<\/a> with vaginal adenocarcinoma was established from a small case-control study in which 7 of 8 mothers with carcinoma were treated with diethylstilbestrol during the first trimester. None were treated in the control group.<\/p>\n Concern about observational studies in assessing harms is due to the introduction of bias from uncontrolled confounding. Such confounding by indication, set out in the Catalogue of Bias<\/a>, is a distortion that modifies an association between an exposure and an outcome. It is caused by the presence of an indication for the exposure that is the real cause of the outcome.<\/p>\n Even after careful matching and adjustment for known risk factors, residual confounding may persist.\u00a0 An Assessment and control for confounding by indication in observational studies<\/a> highlight the impact of confounding depends on the prevalence of the confounder, the level of its association with disease and the exposure.\u00a0 A confounding factor with a prevalence of 20%<\/a> would have to increase the relative odds of both outcome and exposure by factors of 4 to 5 before the relative risk of 1.57 would be reduced to 1.00.\u201d<\/p>\n Conclusion <\/strong><\/p>\n To assign a clear association that use of oral HPTs in pregnancy is associated with increased risks of congenital malformations requires a number of factors beyond just a quality assessment.<\/p>\n Using a systematic review of case-control and cohort studies to answer the question of whether Primodos is associated with harms is appropriate because there were ethical issues over the exposure, there were no therapeutic benefits,\u00a0 no prognostic implications of the exposure, meaning randomization was not appropriate. The use of scoring systems has weaknesses, therefore, it is important to report the individual biases and consider their impact on the effect.\u00a0 Many pooled analyses had zero heterogeneity, and the direction of effect favoured the controls in 30 of the 32 analyses undertaken. This consistency further strengthens the association.<\/p>\n Carl Heneghan<\/strong> Heneghan C, Aronson JK, Spencer E\u00a0et al.<\/em>\u00a0Oral hormone pregnancy tests and the risks of congenital malformations: a systematic review and meta-analysis [version 2; referees: 3 approved]<\/span>.\u00a0F1000Research<\/i>\u00a02019,\u00a07<\/b>:1725 CH is an advisor\u00a0to\u00a0 All-Party Parliamentary Group on Hormone Pregnancy Tests<\/a>, has presented the results of the systematic review at the UK Houses of Parliament,\u00a0 and The Independent Medicines and Medical Devices Safety Review.\u00a0<\/a>CH has received expenses and fees for his media work including BBC Inside Health. He holds grant funding from the NIHR, the NIHR School of Primary Care Research, The NIHR Oxford BRC\u00a0 and the WHO.\u00a0 CEBM jointly runs the\u00a0EvidenceLive<\/a>\u00a0Conference with the BMJ and the\u00a0Overdiagnosis Conference<\/a>\u00a0with some international partners which are based on a\u00a0 non-profit model.<\/p>\n","protected":false},"excerpt":{"rendered":" In a recent systematic review, we assessed the use of Primodos, an oral hormone pregnancy test (HPT) marketed between 1958 and 1978, and the associated risk of congenital malformations. This post discusses the assessment of quality in assessing associations of harms.\u00a0 Carl Heneghan We found oral HPTs in pregnancy was associated with an increased […]<\/p>\n![\"\"](\"https:\/\/blogs.bmj.com\/bmjebmspotlight\/files\/2017\/06\/Carl-Heneghan-150x150.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/blogs.bmj.com\/bmjebmspotlight\/files\/2019\/03\/Screenshot-2019-03-15-at-11.36.21.png\")
<\/a><\/p>\n\n
\n
\n
\n
![\"\"](\"https:\/\/blogs.bmj.com\/bmjebmspotlight\/files\/2019\/03\/Screenshot-2019-03-15-at-11.38.01.png\")
<\/a><\/p>\n
\nEditor in Chief BMJ EBM,
\nProfessor of EBM, University of Oxford<\/p>\n
\n
\n(https:\/\/doi.org\/10.12688\/f1000research.16758.2<\/a>)<\/p>\n
\n<\/div>\n