SMACCDUB is the conference of young critical care doctors, with a few mature ones thrown in for the mix, and the most energetic I’ve been to in a long time, perhaps ever. In a plenary session I sat on a white sofa drinking a 25 year old Irish whiskey next to Jeff Drazen, the editor of the New England Journal of Medicine. Our interaction was described by one Tweeter as a cage fight. There were six others on the sofas, them representing the future—and Jeff and I the past.
Judging by the applause, the audience of around 2000 sympathised with the idea that the age of journals was over. I’ve blogged before on the failures of journals. Sara Bassin, a training critical care doctor who was sat on the other side of me, described where she got her information: UpToDate, Life in the Fast Lane, the Bottom Line, ICU Trials, guidelines, and protocols. Sometimes, she said, she went to look at a trial in a journal. I wondered if she was saying that to be nice to Jeff.
So what might the post-journal world look like? We were asked that question, but I didn’t get a chance to answer—probably because I’d spoken too much already. But I promised in a Tweet that I would.
Sara’s answer shows that we are already in the post-journal world in terms of clinicians getting answers to their questions on how best to treat patients, and we have been for a long time.
Journals are still, however, the main conduit for publishing original science. Jeff “paternalistically” (his word) imagines himself to be doing the work that clinicians don’t have time to do by providing them with the best evidence available. In fact, I argued, he’s sorting among the perhaps 0.1% of new studies that happen to be submitted to the NEJM and publishing about 5%. I suggest that no clinicians should change practice on the basis of such a tiny fraction of the total evidence, and, of course, few do.
What the NEJM is primarily doing is contributing to an absurd system of judging academics by where they publish and making a ton of money for the Massachusetts Medical Society. I pick on Jeff because he was there, but other editors (and I was once one) are doing the same, albeit with less impact.
In the post journal world research will start with the publication on an open website of a funded grant proposal. Anybody will be able to comment, and to avoid repetition I should make clear that everything I mention will be posted online with anybody free to comment. This is open science.
The proposal should include a protocol for a systematic review relevant to the research question. It’s likely that the protocol will be revised in the light of comments. Changes to the protocol will be published with an exclamation of why. Ideally the systematic review will be based on raw data not published summary data. (This will eventually be the norm because all raw data will be published in this post-journal world.)
Once the systematic review is completed it will be posted, ideally with all the underlying data. Sometimes—indeed, probably often—the review will show that the new research isn’t necessary as the question will have been answered adequately. At the least, the review is likely to lead to refinement of the research protocol.
The research will be registered. At the moment only trials have to be registered, although many aren’t, but really all research ought to be registered, mainly as a mechanism to avoid researchers or others failing to publish research that disappoints them.
The research protocol will be posted including the analyses that will be conducted, and again any modifications will be declared. We know that many protocols are changed during the research process without either declaration or explanation.
Importantly when the study is published it will be accompanied by full data. I’ve argued elsewhere with a colleague that in many ways data are more important than “papers,” which are often spun to give the result the sponsors were hoping for. Researchers, we argued, should be incentivised to produce good data rather than “papers” and to reanalyse and replicate the studies of others. (Replication is at the heart of science, and we know that a large proportion of studies cannot be replicated—and most studies nobody ever tries to replicate.)
The new research should be absorbed into the systematic review. I see no need for peer review by a few people, a process that lacks evidence and is slow, inefficient, and expensive. The “real peer review” is the world deciding the value of the research in “the market of ideas.”
One key question is who will own the website or sites on which all of this is posted. One option would be that all research funders would have their own sites, but it would be best if there were one site. Such a site might be funded by one government, perhaps the US as with PubMed, or by a consortium of governments.
Although what I’m proposing might sound fanciful to some, it’s not far removed from what Britain’s National Institute for Health Research is already doing. The institute is also keen to continue the process by following the research with all attempts at dissemination and ultimately having an impact in the real world. The emphasis in evaluating research is already shifting from publishing papers to having measurable impacts.
And what about the many pieces that are not research but opinion and commentary? They might best be attached to the research if they are about the research, but they could continue to appear in journals. The journals would, however, need a new business model. Or there might be multiple websites for particular communities. Blogs, I suggest, will replace more formal articles. I still write articles for journals, but I find it a slow, tedious, unrewarding activity. I’m not alone in my displeasure, but—unlike academics—I have no need to publish in journals.
How should clinicians make sense of research? Exactly as Sara does now—by consulting sources that synthesise the data. And with the system I propose they will be able to synthesise all the evidence not just a biased half of it as is the case at the moment.
Richard Smith was the editor of The BMJ until 2004.
Competing interest: RS was editor of the BMJ and has a pension from the BMA, the owners of the BMJ. He was on the board of the Public Library of Science, and he does paid consultancy fir F1000Research. He had his expenses paid to attend SMACC and was given a tee shirt and, as mentioned in the blog, an expensive glass of whiskey.