By Sara Dahlen, Dean Connolly, Isra Arif, Muhammad Hyder Junejo, Susan Bewley and Catherine Meads
Doctors trying to help their patients are bombarded with too much and often conflicting information. Well-developed clinical practice guidelines (CPGs) can help busy practitioners ground their real-world frontline care in properly considered, best available knowledge gained from high quality research. Nevertheless, CPGs vary, and doctors need to discern the good from the bad, the helpful from the promotional. A CPG emanating from the World Health Organization, Britain’s National Institute for Health and Care Excellence, America’s National Academy of Medicine or Germany’s Agency for Quality in Medicine will likely have different and better methods, and thus authentic authority, compared to a consensus statement produced by, say, the International Society of Cosmetogynaecology, an organisation “dedicated to serving the needs of specialists” and “to enrich the member’s life both personally and professionally”. CPGs may be subject to various forms of bias, vested interests, uncertainties, claims based on experts’ eminence rather than scientific evidence, and they may lack methodological rigour. Thus, CPGs are worthy of quality assessment and scholarly scrutiny in-and-of themselves.
What CPGs do is link the primary research studies that form the bedrock of evidence-based medicine to clinical practice by making recommendations. They have a unique methodology. Thus, any appraisal of a CPG is not equivalent to the initial rigorous assessment of the scientific data that underpins the recommendations within said CPG; other frameworks such as GRADE exist for this. CPG appraisal does not itself independently examine the strength and risks of bias of the evidence in the field within which a CPG sits, but it asks whether these guideline developers properly demonstrated that they did so?
One validated CPG appraisal tool is the revised Appraisal of Guidelines for Research and Evaluation instrument (AGREE II). It was designed with the ordinary, interested clinician in mind. AGREE II allows structured evaluation of a CPG’s quality through examination of 23 key items divided into 6 domains, plus an overall assessment. AGREE II assesses quality by giving higher scores to guideline developers who, for example: performed systematic reviews, explicitly linked their recommendations to evidence, declared their interests and funding, included stakeholder views and considered resource implications. Guidelines are not ‘tramlines’ that tell doctors precisely how to practice. They are applicable most, but not all, of the time and must be applied judiciously. They are well developed in many countries as a means to drive up the quality of care and better outcomes, avoid overdiagnosis and overtreatment, promote financial and drug stewardship, as well as contain costs. There are many national and international producers of guidelines, whose methods continuously improve. In the UK, we even have a scheme whereby specialist societies can have their methods ‘accredited’ by NICE – all working transparently for the patient’s good.
The blog authors came together as a team following what had started as a King’s College London’s medical student BSc research training project. Working with a leading systematic reviewer, one of the UK government’s LGBT Health Taskforce members, we wanted to find and assess the quality of international CPGs that covered any aspect of trans and gender minority people’s healthcare and used AGREE II to do this. Gender transition related medical interventions are an area of clinical practice already acknowledged to lack high quality evidence to inform CPGs. We were interested in all aspects of healthcare, throughout the life-course and assessed the quality of these CPGs as a whole. It is normal good practice for a systematic review to start broadly by looking for everything (as that identifies gaps), and with a pre-specified protocol (as that minimises biases). We found a surprising variety of CPGs and sources, but a paucity of CPGs covering more general aspects of trans and gender minority people’s healthcare. Those CPGs that focused on transition-related interventions tended to be developed by special interest societies and scored lower on AGREE II. Those CPGs covering HIV and other blood-borne viral infections tended to be developed by the World Health Organisation and scored noticeably higher. This led to our calling for better quality CPGs and research in the field, in order to cover a broader range of issues and address trans and gender minority people’s holistic healthcare needs.
Readers should be very confident in our findings and their interpretation, given they were improved by seven rigorous BMJ reviewers followed by presenting the work at both World Professional Association of Transgender Health and Endocrine Society conferences. There have been thousands of reads and downloads and no objections. Although a blogger recently charged us with interpreting “the science of treating trans youths with puberty blockers as ‘experimental’ and ‘unethical’”, this was incorrect. Our paper did not address that question, nor make any such claims. The blog has since been amended. We realised there may be some confusion or unfamiliarity as to how CPGs and their appraisal work. We warmly encourage anyone interested in CPGs and evidence-based medicine to familiarise themselves with AGREE II. It is a helpful and easy-to-use instrument for anyone currently using CPGs in their practice and/or involved with developing CPGs. The website even provides free training tools. Aspiring medical students, academics and high-flyer clinicians will all find it useful as part of their toolkit for developing critical thinking skills. It could form the basis of presentations, audits, analysis and publications even at an early stage in your medical career.
Our paper could also form the basis of an educational journal club. If it’s held at a reasonable hour in the UK, we’d even try to be available to join you anywhere in the world and answer questions. Background work could be shared amongst a few students or doctors dividing, looking up, circulating and presenting; (i) the AGREE II tool and website, (ii) our protocol, (iii) the paper, (iv) the supplementary materials, and, finally (v) examining the ‘first evidence-based process’ methodology for a forthcoming decennial guideline which is out for consultation from WPATH until 16th December 2021. You might consider whether the identified methodological concerns have been allayed or remain. Scrutinize the composition of the CPG’s panel, the publicly available prespecified protocol and published systematic review results. How did the committee gather evidence, assess its quality (especially where there was no systematic review) and devise recommendations (called ‘statements’) linked to the supporting evidence? Did they obtain independent external peer review, and what are the plans regarding open publication of this current round of stakeholder comments and their replies? How will AGREE II judge this next version?
Finally, a note of caution. We have used our paper here as an educational ‘worked example’ of AGREE II, but trans and gender minority healthcare is by no means unique in terms of ending in a call for higher quality standards for CPGs. One of us wrote a commentary on a published systematic review of CPGs on uncomplicated births, noting similar themes and recent papers have severely criticised consensus cardiology guidelines as inappropriate or ‘just do it’, thus harming patients.
It would benefit the medical profession, particularly those parts using science and evidence, but most of all it would benefit patients, if we all agreed to use AGREE II more widely.
Authors:
Sara Dahlen, Department of Global Health & Social Medicine, King’s College London, London, UK http://orcid.org/0000-0002-1798-1816
Dean Connolly, Visiting Researcher, National Addiction Centre, Institute of Psychiatry, Psychology and Neuroscience, KCL King’s College London, London UK. Foundation Doctor, Newham Centre for Mental Health, East London Foundation Trust http://orcid.org/0000-0002-3139-4263
Isra Arif, Basingstoke and North Hampshire Hospitals, UK http://orcid.org/0000-0002-8623-2069
Muhammad Hyder Junejo, Homerton Anogenital Neoplasia Service, Homerton University Hospital, London, UK http://orcid.org/0000-0003-2458-9991
Susan Bewley Department of Women and Children’s Health, School of Life Course Sciences, King’s College London, London, UK http://orcid.org/0000-0001-8064-652X
Catherine Meads Faculty of Health, Medicine, Education and Social Care, Anglia Ruskin University – Cambridge Campus, Cambridge, UK http://orcid.org/0000-0002-2368-0665
Correspondence to Professor Susan Bewley susan.bewley@kcl.ac.uk