Blog entry written on “A Framework For The Synthesis of Non-Randomized Studies and Randomized Controlled Trials: A Guidance On Conducting A Systematic Review and Meta-analysis For Healthcare Decision-Making” (bmjebm-2020-111493)

Authors: Grammati Sarri (Real World Evidence Strategy Lead, Visible Analytics Ltd, UK) and Thomas Debray (Assistant Professor, University Medical Center Utrecht, Utrecht, The Netherlands and Smart Data Analysis and Statistics, Utrecht, The Netherlands).

The collection of patient data has changed dramatically over the last 10 years and the landscape for evidence generation has changed with it.  Innovative study designs, digital technologies, and patient registries (or even linked databases across countries) are used increasingly often to investigate the effectiveness of certain interventions. As a result, new methodologies have emerged to address the limitations associated with these non-randomized studies (NRS), such as confounding and selection bias.

So far, these methodological advances have not been enough for NRS to build trust and to enter centre stage in healthcare policy making.  As a result, randomized controlled trials (RCTs) presently remain the primary source of evidence for evaluating the comparative effectiveness of medical interventions.  This is unfortunate, because NRS may offer important complementary evidence to RCTs and speed up the decision-making process.

The International Society for Pharmacoepidemiology (ISPE) Comparative Effectiveness Research Special Interest Group therefore wants to change the status quo. Drawing on recent guidance for standardising the principles of how to conduct and report results from NRS, the ISPE working group set out to develop a framework to guide the cross-design integration of evidence from RCTs and NRS through the different steps of evidence synthesis.

The group produced a seven-step guide, outlining how to source (search) and appraise the  evidence from NRS; how to select the best quality evidence for evidence synthesis; and when and how to produce a cross-design summary estimate that can be reliable and trustworthy at the level expected by healthcare decision-makers and rightly deserved by patients.

Recommendations outline the most appropriate statistical approaches based on three main analytical scenarios applicable to healthcare decision-making (“high-bar evidence”, when RCTs are the preferred source of evidence; “medium”, and “low”, when NRS is the main source of inference). The framework aims to increase transparency on data identification and selection, and to deepen understanding of NRS limitations while also highlighting potential implementation challenges.

So, why should readers utilise this framework, and when? Briefly, the framework is primarily designed for health policymakers, researchers, and members from the pharmaceutical industry. It can, for instance, be used when considering NRS to address limitations of RCTs for assessing the relative efficacy and safety of new technologies. A pressing need for NRS may arise when technologies are developed for rare diseases. NRS may be also be valuable when there is a need for identifying clinically relevant subgroups, or when evidence is needed for real-world product effectiveness in patients with multiple comorbidities or during longer timeframes.

Issues relevant for the trustworthy use of NRS such as study registration (particularly for hypothesis-evaluating treatment effectiveness studies), data collection (primary or secondary), source validation, and results reproducibility were topics beyond the scope of this framework as previous guidance has comprehensively covered these topics. In fact, readers should reference the framework alongside earlier reported guidance on this topic.

Ultimately, the framework can be used as a live guidance which is regularly updated as new research emerge, such as the role of machine learning in producing unbiased treatment effect estimates in NRS.

Authors

Grammati Sarri

Real World Evidence Strategy Lead, Visible Analytics Ltd, UK
Conflicts of interest: Employed by Visible Analytics, Ltd

Thomas Debray

Assistant Professor, University Medical Centre Utrecht, Utrecht
Conflict of interest: Performs consulting services to various stakeholders.

DISCLAIMER

The views and opinions expressed on this site are solely those of the original authors. They do not necessarily represent the views of the BMJ and should not be used to replace medical advice. All information on this blog is for general information, is not peer-reviewed, requires checking with original sources and should not be used to make any decisions about healthcare. No responsibility for its accuracy and correctness is assumed by us, and we disclaim all liability and responsibility arising from any reliance placed on such commentary or content by any user or visitor to the Website, or by anyone who may be informed of any of its content. Any reliance you place on the material posted on this site is therefore strictly at your own risk.