The Saturday before last I was rung up by a fellow of the Royal Society who was having trouble with the New England Journal of Medicine, and our conversation soon moved to the irrationality of “the REF” [research excellence framework].

We made the move because I asked why the results of a major trial undertaken in Britain were being submitted to an American journal.

“You know why,” he answered, “because of the REF.”

The REF is the latest version of the vast exercise that is undertaken every few years to rate the research performance of departments within British universities. It has huge consequences because money is allocated on the basis of how departments perform. Research universities and the academics within them are thus obsessed with the REF.

There is an underlying logic to the REF in that it used to be that funds were wasted on low quality research. I myself argued for something like the REF in a series of articles I wrote on research policy in the ‘80s, but I fear that the process may have become so elaborate and complex that it has become counterproductive. What I am sure about is that using where people publish as the main measure of the quality of their work is unscientific and a block to improvements in scientific publishing.

My colleague who rang on Saturday told me how publishing in the New England Journal of Medicine will give you a high rating, more so than publishing in the Lancet and much more so than publishing in the BMJ; and when it comes to many journals “forget it”—you might as well have published in the Daily Sport. (Mind you, it gave him some perverse pleasure when repeatedly congratulated for having published a paper in the New England Journal of Medicine to say that it had been rejected by both the Lancet and the BMJ.)

We have long known that it is wholly unscientific to use the impact factor of the journal as a surrogate for the impact of the article because there is little correlation between the number of citations of articles and the impact factor of the journals in which they are published—because the impact factor of journals is driven by a few highly cited articles. I and others pointed this out to the Higher Education Funding Council, which runs the REF, after the first round of assessing research some 20 years ago, and HEFC told assessors not to do it.

The problem is that the assessors are presented with hundreds of articles. It isn’t possible for them to read them all, and so they do pay a lot of attention to where articles are published.

What this means is that universities have effectively outsourced to journals and publishers the function of assessing academic quality, an activity that many would consider core to academia.

Then recent studies have underlined the illogicality of giving such emphasis to top journals because studies published in them are more likely to be wrong than those published in lesser journals. On reflection this is not surprising because big journals are attracted by the new and sexy, both markers of wrongness.

Another problem with big journals, particularly the New England Journal of Medicine, is that its high impact factor is driven by the many drug company sponsored studies they publish, studies that we know are full of bias rarely find results unfavourable to their sponsors.

My biggest objection to the craziness of the REF is, however, that it’s the main barrier to all studies being open access—because the big journals are mostly not open access.

I’d like to see a simplified REF that depends much less on where academics publish. So, I believe, would many academics.

Richard Smith was the editor of the BMJ until 2004 and is director of the United Health Group’s chronic disease initiative.

* This blog was edited post publication. It originally referred to the “research evaluation framework.” This has been corrected to the “research excellence framework.”