How do we find the evidence that might be of interest to individual health professionals?  

Jon Brassey

We’re all aware of the explosion of medical knowledge: “Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?” sums up the problem in the title alone, while the NEJM reports “In 1950, doctors in practice could expect the total amount of medical knowledge to double every 50 years. By 2020, it will take just 73 days.”

The majority of knowledge is unlikely to be of interest to a single health professional, particularly specialists.  But that still leaves stacks of evidence that might be of interest to individual health professionals.  The problem is not one of information overload but one of filter failure.

Information specialist’s role is to filter out the noise amongst all of this information, trying to arrive at the good stuff.  In my role as an associate editor at BMJ EBM, I have been tasked with finding practice-changing research – the good stuff – to the editors to allow them to select the research evidence that matters to clinical practice for the EBM Verdict.

So, what does this entail?  It’s early stages in the process, and we’ll be monitoring the performance of the filters that we introduce, but here are a few criteria we’ve developed:

• Definitive conclusions – should this be started (or stopped) in practice? No ‘needs more research’ or ‘these results might….’.
• Robust – are the conclusions based on high-quality research?
• Is the type of study the right one to answer the question?
• Is the research likely to be of broad interest?

The top two criteria are used to help create a long list of articles generated by two main methods.

• Harnessing of existing systems to support article selection. For instance, there are already systems, such as EvidenceAlerts and NIHR Signals, which have a role of filtering the mass of journal articles.  Evidence Alerts scans over a hundred journals and filters them based on quality, newsworthiness and relevance to clinical practice.  We look through these outputs and look for those with definitive conclusions and add these to our long list of articles.
• We scan the big five internal medical journals (BMJ, NEJM, Lancet, JAMA and Annals of Internal Medicine). Again, we use the criteria of definitive conclusions as inclusion criteria to the long-list.

For every included article we obtain the Altmetric score (https://www.altmetric.com/) which we use as a proxy for interest.  This is not an inclusion/exclusion criteria but more to guide the editors/writers in selecting articles that are likely to be of most interest to our readers.

In the first attempt at the long-list around 50 articles were highlighted – is that too many for one month (as we’re only likely to cover 8 per publication) or too few (as the editors don’t have enough to choose for their interest/speciality)?

Is Altmetric a reasonable proxy for interest?  If an article has a sizeable Altmetric score, has it already been seen and discussed by our readers. Altmetric might also introduce ‘hot stuff bias’, as fashionable (‘hot’)  research may be less critical in its approach and therefore less useful.

A further potential issue is understanding the role of BMJ EBM in unearthing useful, practice-changing research.  Is it the case that such research can be found using the methods outlined above?  Probably not, but how do we make sure we find them?  We don’t yet have the answers.

So, we’d like to hear from you to help guide the development of the methods in creating the long-list and refining it. Get in touch and help make BMJ EBM Verdicts better, more useful and practice changing.

BMJ Evidence-Based Medicine – original evidence-based research, insights and opinion

Read more in the Welcome to BMJ Evidence-Based Medicine Editorial.

Competing interests

I run the search engine the Trip Database (www.tripdatabase.com) and own 50% of the shares in the company, and I am an Associate Editor at BMJ EBM.