Cochrane systematic reviews provide perhaps the best evidence available on health, but do they improve the care of patients and shape health policy? This was the question addressed at one of the liveliest sessions at the Cochrane Colloquium in Auckland.
Most people are cared for in primary care, but the full Cochrane reviews are useless to GPs, said Bruce Arroll, professor of primary care in Auckland. Emphasising the point, he said he’d put on antipsychotics any GP he saw reading a Cochrane review. The reviews are too long, don’t give an effect size, and are unreadable. GPs need information that is quick, comprehensive, valid, relevant, and preferably pushed not pulled. Arroll and others have thus developed pearls that summarise reviews in 200 words, give the message in the title, include number needed to harm and number needed to treat, are put through a relevance filter, and pushed free to those who have signed up to receive them. They receive two every two weeks. So far 2000 people have signed up to receive the pearls, and you can receive them for free here.
Norman Swan, a doctor turned broadcaster, who spoke later in the session said “Put all your money consumers to effect change.” Cochrane from its beginning has given a strong emphasis to consumers being involved in every way, including reviewing systematic reviews, and it publishes “plain language summaries” of all reviews. Unfortunately, said Catherine McIlwain, who works with the Cochrane Consumer Network, about a quarter of the summaries have inconsistencies with the abstracts. The reading level of the summaries is university level, and many people don’t understand them. “We are not getting the message out,” said McIlwain. So work is underway to improve the summaries, even to the extent of sometimes removing the numbers, which seems heretical to some.
Although thousands of consumers have joined Cochrane, many feel disconnected and so drop off. McIlwain emphasised the need to feedback to consumers who offered reviews, and the Collaboration is providing a range of training material for consumers.
Policymakers are another audience for Cochrane reviews, and evidence based folk are traditionally frustrated that evidence seems to be ignored by policymakers. They are not ignoring evidence, said Ashley Bloomfield from the New Zealand Ministry of Health, but it’s one of many inputs to policymaking. He said that in his experience evidence had always been considered, but it was sometimes overridden by other concerns. The world is messier than evidence zealots would like it to be. Bloomfield urged Cochrane reviewers to make contact with their local policymakers and even include them as authors on reviews. Politely he asked for a “pious free zone.”
In one of the most powerful presentations at the colloquium Swan talked about reaching the media and summed up Cochrane’s predicament by saying that the Collaboration wants to remove emotion and insert evidence into health care, whereas the media are about the opposite. We communicate emotionally, he said, and you have to use emotion to reach people. The work of Daniel Kahneman, the Nobel prize winner, and other behavioural economists has shown clearly what irrational creatures we are.
We don’t, for example, notice the usual but we do respond to change. So if you usually speak quietly to children they will respond to a shout, whereas if you usually shout another shout will evoke little response. We respond more strongly to the negative than the positive, explaining why news in the media is always bad not good news. Loss has a much bigger impact on us than gain, so losing $50 creates much more of a reaction than finding $50. This probably explains why drug regulators give much greater weight to drugs killing than failing to save lives because they are not available.
Another defect in our thinking is that we are hopeless with big and small numbers: 20 000 killed by tobacco in Australia means nothing, said Swan. We identify strongly with one person, less so with two, and not at all with 100 000—hence the media’s attention on patient stories. But the media don’t have to worry about representativeness of the patient, while Cochraneites will. The tsunami that killed tens of thousands may have had such a big impact, speculated Swan, because stories about every kind of person were embedded in the disaster. Something else that always gets our attention is the possibility that a hidden hand, perhaps a big company, may be shaping events. I’ve discovered that a proposal for a television programme that includes a villain is much more attractive than one that says the reasons for something are “complicated and ill understood.”
All this, it seems to me, presents a problem for Cochrane, which is trying to promote rationality in a highly irrational world, and Swan understandably was stronger on the problem than the solution.
Indeed, the conclusion from the session was somewhat bleak. Cochrane is not doing well at reaching health practitioners. Its messages to consumers are inaccurate and poorly understood. Policymakers are interested in evidence, but it will often be trumped by other inputs. And to get its rational messages across Cochrane paradoxically will have to better understand our irrational natures. But Cochraneites love a challenge and will no doubt respond to these challenges.
Competing interest: RS spoke at the Cochrane Colloquium and had his expenses paid. He’s also chair of the Cochrane Library Oversight Committee, an unpaid position.
Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.