Despite robust study designs, even double-blind randomized controlled trials can be subject to subtle forms of bias. This can be due to financial conflicts of interest of authors, intellectual or disciplinary based opinions, pressure on researchers from sponsors, or conflicting values. For example, some researchers may favour mortality over quality of life as a primary outcome, demonstrating a value conflict. The quality of evidence is often uneven and can include underappreciated sources of bias. This makes interpreting the evidence difficult or gives rise to uncertainty, which results in guideline developers turning to “experts” to translate the evidence into clinical practice recommendations.
However, can we be confident that guideline development “experts” are objective and free of bias? A 2011 National Academy of Medicine (formerly known as the Institute of Medicine) report challenged the assumption of objectivity and lack of bias among guideline development experts. [1] Members of that committee (Disclosure: I served as Committee Chair), who possessed firsthand experience of guideline development, were so concerned about conflict of interests that they recommended setting the bar high for several conflict of interest standards in guideline development. First, the committee recommended that the Chair (or co-Chair) should work in a discipline different from where the original research was performed. Second, they recommended that 50% of the committee members should be from another discipline, such as users or methodologists. The 50% (itself not “evidence based”) was intended to send a message about the importance of meaningful balance within the committee.
Subsequently, the National Guideline Clearinghouse (NGC) set standards for limiting bias in the generation of guidelines. In 2018, the ECRI Institute, a nonprofit healthcare research organization, created and launched the ECRI Guidelines Trust, whose purpose was to vet guidelines according to the standards set out by the Institute of Medicine. The NGC Extent of Adherence to Trustworthy Standards (NEATS) instrument directly addresses the issue of bias or conflict of interest by experts included in the guideline development process. [2] According to NEATS standards, two specific determinants of the quality of a guideline are the inclusion of a multidisciplinary team of “relevant clinical specialties and other professional groups,” and inclusion of “methodological experts.” In a recent conversation with the ECRI Guidelines Trust Leadership, they informed me that of the 1300 total guidelines in their inventory, more than 500 guidelines were scored to date. Of the 500 scored guidelines, 85% included a multidisciplinary team and 75% included a methodological expert.
They are optimistic that high quality guidelines, with decreased bias, can be produced by adhering to established standards.
I am optimistic too. Some of my optimism also stems from a recent series of controversial articles published in the Annals of Internal Medicine on the harms of red meat. [3] The extensive, even extraordinary, efforts to minimize bias were heartening. In addition to financial disclosures, which went back to three years prior to the study start date, other steps were taken. The chair of the systematic review was not involved in nutrition research, and the guidelines development team included both multiple disciplines and multiple methodologists who could interpret the study data. Further, members of the guideline development group were queried about their positions on the harms of red meat by asking how many servings of red meat were consumed on a weekly basis—an imperfect, but revealing way of assessing a position so strongly held that they would not change their minds in the face of evidence. Finally, the editors of Annals revealed their own disclosures, including the holdings of one of the editor’s spouses. Asking committee members about their positions on a topic and adjudicating them publicly, as advocated by Lisa Bero and others, is a major step further in minimizing conflicts of interest.
As noted in several articles since the publication of the original Annals article, concerns have arisen because the lead author did not disclose a tie with industry. This tie was not related to the issue at hand (the effects of red meat on health) and had occurred more than three years before this study began. The author probably should have disclosed his relationship to the sugar industry due to the sensitivity of the topic. However, as pointed out, this study would have been published anyway based on the multiple checks and balances that were transparent and outlined in detail, providing a seemingly robust bulwark against bias.
The science that supports clinical medicine is constantly evolving. The pace of that evolution is rapidly increasing. There is an urgent imperative to generate and update accurate, unbiased, clinical practice guidelines. So, what can we do now? I have two suggestions. First, the public, which may include physicians, payors, nurses and other healthcare providers dependent on guidelines, should advocate for organizations like the ECRI Institute and their international counterparts to be supported and looked to for setting standards. Second, we should continue to examine the details and principles of “shared decision-making” and other initiatives like it, so that doctors and patients can be as transparent as possible in the face of uncertain evidence about medical treatments and recommendations. It is an uphill battle, but one worth fighting.
Read the full collection: Commercial influence in health: from transparency to independence
Sheldon Greenfield is the executive co-director of the Health Policy Research Institute at the University of California, Irvine and Donald Bren Professor of Medicine.
Competing interests: None declared
References:
- Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. (Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, eds.). Washington (DC): National Academies Press (US); 2011. http://www.ncbi.nlm.nih.gov/books/NBK209539/. Accessed November 5, 2019.
- Jue JJ, Cunningham S, Lohr K, et al. Developing and Testing the Agency for Healthcare Research and Quality’s National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument. Ann Intern Med. 2019;170(7):480. doi:10.7326/M18-2950
- Johnston BC, Zeraatkar D, Han MA, et al. Unprocessed Red Meat and Processed Meat Consumption: Dietary Guideline Recommendations From the Nutritional Recommendations (NutriRECS) Consortium. Ann Intern Med. October 2019. doi:10.7326/M19-1621