Richard Smith: What will the post journal world look like?

richard_smith_2014SMACCDUB is the conference of young critical care doctors, with a few mature ones thrown in for the mix, and the most energetic I’ve been to in a long time, perhaps ever. In a plenary session I sat on a white sofa drinking a 25 year old Irish whiskey next to Jeff Drazen, the editor of the New England Journal of Medicine. Our interaction was described by one Tweeter as a cage fight. There were six others on the sofas, them representing the future—and Jeff and I the past.

Judging by the applause, the audience of around 2000 sympathised with the idea that the age of journals was over. I’ve blogged before on the failures of journals. Sara Bassin, a training critical care doctor who was sat on the other side of me, described where she got her information: UpToDate, Life in the Fast Lane, the Bottom Line, ICU Trials, guidelines, and protocols. Sometimes, she said, she went to look at a trial in a journal. I wondered if she was saying that to be nice to Jeff.

So what might the post-journal world look like? We were asked that question, but I didn’t get a chance to answer—probably because I’d spoken too much already. But I promised in a Tweet that I would.

Sara’s answer shows that we are already in the post-journal world in terms of clinicians getting answers to their questions on how best to treat patients, and we have been for a long time.

Journals are still, however, the main conduit for publishing original science. Jeff “paternalistically” (his word) imagines himself to be doing the work that clinicians don’t have time to do by providing them with the best evidence available. In fact, I argued, he’s sorting among the perhaps 0.1% of new studies that happen to be submitted to the NEJM and publishing about 5%. I suggest that no clinicians should change practice on the basis of such a tiny fraction of the total evidence, and, of course, few do.

What the NEJM is primarily doing is contributing to an absurd system of judging academics by where they publish and making a ton of money for the Massachusetts Medical Society. I pick on Jeff because he was there, but other editors (and I was once one) are doing the same, albeit with less impact.

In the post journal world research will start with the publication on an open website of a funded grant proposal. Anybody will be able to comment, and to avoid repetition I should make clear that everything I mention will be posted online with anybody free to comment. This is open science.

The proposal should include a protocol for a systematic review relevant to the research question. It’s likely that the protocol will be revised in the light of comments. Changes to the protocol will be published with an exclamation of why. Ideally the systematic review will be based on raw data not published summary data. (This will eventually be the norm because all raw data will be published in this post-journal world.)

Once the systematic review is completed it will be posted, ideally with all the underlying data. Sometimes—indeed, probably often—the review will show that the new research isn’t necessary as the question will have been answered adequately. At the least, the review is likely to lead to refinement of the research protocol.

The research will be registered. At the moment only trials have to be registered, although many aren’t, but really all research ought to be registered, mainly as a mechanism to avoid researchers or others failing to publish research that disappoints them.

The research protocol will be posted including the analyses that will be conducted, and again any modifications will be declared. We know that many protocols are changed during the research process without either declaration or explanation.

Importantly when the study is published it will be accompanied by full data. I’ve argued elsewhere with a colleague that in many ways data are more important than “papers,” which are often spun to give the result the sponsors were hoping for. Researchers, we argued, should be incentivised to produce good data rather than “papers” and to reanalyse and replicate the studies of others. (Replication is at the heart of science, and we know that a large proportion of studies cannot be replicated—and most studies nobody ever tries to replicate.)

The new research should be absorbed into the systematic review. I see no need for peer review by a few people, a process that lacks evidence and is slow, inefficient, and expensive. The “real peer review” is the world deciding the value of the research in “the market of ideas.”

One key question is who will own the website or sites on which all of this is posted. One option would be that all research funders would have their own sites, but it would be best if there were one site. Such a site might be funded by one government, perhaps the US as with PubMed, or by a consortium of governments.

Although what I’m proposing might sound fanciful to some, it’s not far removed from what Britain’s National Institute for Health Research is already doing. The institute is also keen to continue the process by following the research with all attempts at dissemination and ultimately having an impact in the real world. The emphasis in evaluating research is already shifting from publishing papers to having measurable impacts.

And what about the many pieces that are not research but opinion and commentary? They might best be attached to the research if they are about the research, but they could continue to appear in journals. The journals would, however, need a new business model. Or there might be multiple websites for particular communities. Blogs, I suggest, will replace more formal articles. I still write articles for journals, but I find it a slow, tedious, unrewarding activity. I’m not alone in my displeasure, but—unlike academics—I have no need to publish in journals.

How should clinicians make sense of research? Exactly as Sara does now—by consulting sources that synthesise the data. And with the system I propose they will be able to synthesise all the evidence not just a biased half of it as is the case at the moment.

Richard Smith was the editor of The BMJ until 2004. 

Competing interest: RS was editor of the BMJ and has a pension from the BMA, the owners of the BMJ. He was on the board of the Public Library of Science, and he does paid consultancy fir F1000Research. He had his expenses paid to attend SMACC and was given a tee shirt and, as mentioned in the blog, an expensive glass of whiskey.

  • JustinReilly

    I’d like to see BMJ be the first to go out of business (followed by Lancet) for their efforts to psychologize the organic disease ME (‘cfs’)

  • @openmylab

    The dilemma I find is that data can come from everywhere ie. from live biosensors (IoT) to population registries, and depending on the analysis and experts involved, data can be a source of truth, untruths, expedience, untraceable, editable, deletable etc. Storage, maintenance, protecting data is an ongoing challenge synonymous with AI developments, and I don’t think this issue can be resolved without the consent, input, commitment etc. from participants/patients.

    A collective systematic review has limits. Cochrane collaborations demonstrate this. A systematic review can differ markedly from another systematic review even though both examine the same data because it depends on the experts involved in the review who then decide which types of data should be included or excluded in the analysis. A global systematic review still requires a panel of experts. Automating this process still requires experts even with the best AI algorithms because at present I don’t think there’s enough pure data and consensus among experts to code such foolproof algorithms.

    I find clinical guidelines and protocols are subservient to marketing, and aren’t necessarily expert/evidence-based. It could be a blogger, ghost writer to who knows. For instance, whoever can SEO the loudest health info website on Google, whoever can run a monopoly with health info ministries are protocols typically translated to busy clinicians.

    Clinicians these days are bombarded with info from everywhere and it seems the human thing to do to select the cheapest medicine, fastest treatment, the closest treatment center, referral to friend, as opposed to the best medicine, best treatment choice, the best doctor, the best evidence, the best long term outcomes etc. for the patient. Clinicians these days seem to have thoroughly adapted to the consumer-supply-demand mentality.

    Patients often actively seek out the best treatment info. So it seems a more natural fit to involve the participants/patients directly in the data collection process and a govt mandate to protect that data. I also think such data would be more robust and more accurate for deriving automated whole body health profile/treatment algorithms.

  • Simon Carley

    Very interesting to share the panel with you at #smaccDUB. We did indeed run out of time to talk about a post journal world.

    We might have discussed the potential survival of journals as aggregators of information and studies from other publishing sources as you describe. Though, blogs may well serve that purpose and thus the paper versions may fall.

    We might also have discussed the mechanisms for the fall of the paywall journal. I’d have loved to get onto discussions around pirate bay sites, open data, research parasites, & creative commons.

    For me, as an advocate of a new way of sharing information and research, it was interesting to see the conflict of interest between researchers, funders and journals. The symbiotic, or perhaps parasitic relationship is ready to be broken.

    I am so glad that you enjoyed #smacc Dublin. It’s truly an amazing experience with an ethos of positivity and success; an window to the future of emergency and critical care.


    Simon Carley

  • braillon

    Brilliant. A simple and efficient solution. That’s why it is not implemented yet

  • Windriven

    The death of $35 reprints for 10 year old journal articles can’t come soon enough to suit me. But the open model including comments by ‘anyone’ suggests a burgeoning chaff-to wheat-ratio. Who has time to sort through dozens, maybe hundreds, of comments to find the nuggets of criticism among the dross. In truth, given the huge number of studies published, selecting the few to be carefully evaluated is going to require some sort of curation. Perhaps it is this curation that will replace traditional journals and their editors.

  • Chrisando

    Kath Maitland had the best point of the panel discussion. In the post journal world the pharmaceutical/equipment companies will be best placed to manipulate the system (versus busy working medical drs and scientists). The bias of the journal editors (freely admitted) and publication bias (well documented) is acknowledged but the risks of bias in a non peer reviewed system are surely much greater.

    Removing paywalls is something we should fight harder for however lets not throw the baby out with the bath-water.

  • Ashish Pandey

    Hey great discussion. I just loved it.