Clinical guidelines are often criticized for not paying enough attention to “real world evidence” (RWE) such as big data, patient generated data, data from registries, or electronic health records. Real world evidence is usually thought to more accurately reflect what happens outside a controlled research environment. In this context, randomised controlled trials (RCTs) can be dismissed as evidence which does not reflect the real world. The rationale for this is that evidence obtained outside of RCTs may more directly apply to the populations, interventions, comparators, and outcomes (PICOs) that practising clinicians, healthcare professionals, researchers, policymakers, and patients deal with, and that this evidence is better suited to support personalized medicine or individualized healthcare. This emphasis on applicability, generalizability, external validity, or transferability—concepts which one can simply label as indirectness—seems sensible; in particular, for trustworthy guideline recommendations which should inform the best interventions and decisions in daily practice. [1] However, undue emphasis on real world evidence is potentially risky for guidelines and beyond.

Firstly, the current concept of “real world” evidence suggests that RCTs come from an unreal world. But evidence from RCTs is part of the real world which researchers are studying by including real, although often selected, PICOs. Our language should reflect that they are still part of that real world.

Secondly, focusing only on real world evidence obscures concerns about risk of systematic errors (bias) by emphasizing directness. Even the most direct real world evidence will not be sufficient to provide certainty that an intervention has the intended effects if other bias is introduced when decision makers assign interventions and fail to achieve a fair comparison between an intervention and an alternative in producing real world evidence. Who is served if evidence is more applicable but the effects are spurious? Indeed, certainty in the effects of interventions arises only if data are obtained from research that has no limitations to study design and execution (risk of bias) and directly applies to the PICO of interest, but the data also are not a random result, are consistent across similarly conducted studies, and are fully disclosed and considered. [2]

Among these considerations, indirectness and imprecision are often the key area of concern with RCTs. Real world evidence, on the other hand, may reduce concerns about indirectness, but it does not eliminate them. For example, believing that real world evidence is superior because it more directly applies to the PICO of interest and using it to demonstrate that an intervention is effective ignores the fact that any evidence influences future decisions, not actual decisions. Regardless of how well the PICOs are represented in real world evidence, when an actual decision is made based on real world evidence, then decision-makers extrapolate from other situations. Thus, all evidence is indirect and labelling evidence as real world evidence does not overcome this problem. Simultaneously, guideline developers should apply approaches to integrate all types of evidence into their guidelines to reduce concerns about indirectness and then judge its strength.  [1, 3]

Real world evidence also often provides more precise evidence through large-scale routine data collection, but this does not reduce its disposition to confounding and selection bias. What may result from real world evidence are precise, but biased effects of interventions. GRADE domains for increasing certainty in evidence and mitigating these risks of bias, provide the few exceptions for when there is less trade-off between indirectness and risk of bias. When confounding or selection bias are unlikely to cause precise large effects, dose effect relations exist or plausible opposing confounding bias is present, our certainty in effects increases. [4] This must be kept in mind.

Thirdly, emphasis on real world evidence plays into the hands of those manipulating data for gain: bias in real world evidence is harder to assess by those evaluating it because the available tools for doing this are more demanding; the analyses are often less transparent and easier to manipulate; and registration of protocols of planned analyses is not done.

Glorifying real world data by (indirectly) undermining other research and evidence from RCTs tarnishes pragmatic trials or large international trials that mitigate concerns about indirectness without sacrificing principles to reduce risk of bias.    

Finally, the phrase “real world evidence” waters down what we should be focusing on. Trustworthy recommendations should be based on an evaluation of the evidence and an assessment of how reliably the evidence supports all factors that determine a recommendation or decision. These factors include the importance of a health problem, the balance between health benefits and harms, values that people attach to outcomes, resource use, equity, acceptability, and feasibility. Ideally this evidence should be transparently displayed in a GRADE evidence to decision (EtD) framework (5).  

However, not all criteria in an EtD framework require RCT evidence. Those emphasizing real world evidence often intentionally confuse this by stating that it is inappropriate to demand an RCT to determine the importance or burden of a health care problem, or to understand people’s values or integrate qualitative data. They are quite right. But emphasizing that real world evidence is sufficient for trustworthy recommendations all together confuses the issues. This is because real world evidence is sufficient for only certain aspects of an EtD framework and these nuances can be challenging to understand. EtD frameworks should explicitly mention the certainty of the evidence based on the appropriate and relevant study designs for all those factors, e.g. non-randomized studies for prognosis and baseline risk, without the need for real world evidence terminology and labelling.

In my opinion, we should stop using the phrase “real world evidence” in a way that suggests superiority to RCTs, or even excludes RCTs, and we should be cautious about using the term real world evidence to deflect conversations about bias. All evidence is real in our world. Guideline developers should judge if big data, patient generated data, or data from registries or electronic health records can mitigate  the key concerns raised about evidence from RCTs to inform decisions: primarily indirectness, imprecision, and use of appropriate evidence to inform factors that influence trustworthy recommendations. They should then assess the certainty of the evidence, regardless of its label. The GRADE approach already provides detailed guidance for this. [5]

 

 

Holger J. Schünemann is chair of the department of health research methods, evidence, and impact. He is the director of Cochrane Canada and co-chair of the GRADE working group but the opinion expresssed here are his own and may not represent that of the organizations he is affiliated with.

 

Competing interests 

I have read and understood the BMJ policy on declaration of interests and declare the following interests: I am co-chair of the GRADE working group. I have no direct financial conflicts of interest although GRADE’s acceptance as a tool contributes to my academic successes.

References

1. Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, et al. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods. 2013;4(1):49-62.
2. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995-8.
3. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence–indirectness. J Clin Epidemiol. 2011;64(12):1303-10.
4. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol. 2011;64(12):1311-6.
5. Alonso-Coello P, Schunemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.