You don't need to be signed in to read BMJ Blogs, but you can register here to receive updates about other BMJ products and services via our site.

Open peer review

Guest blog: Improving peer review using peer-reviewed studies #PeerRevWk16

19 Sep, 16 | by aaldcroft

This week is the second ever “peer review week”. The theme for this year is “Recognition for Review”. Peer review week aims to highlight the importance of peer review, which is a crucial part of the research process. We asked Dr Adrian Barnett, from the Queensland University of Technology, and a member of our editorial board, to survey articles published in BMJ Open that present research on medical publishing and peer review

Avatar Image

It’s challenging to do peer review well and current models of peer review in health and medical research are regularly criticised by researchers who all have personal stories of when peer reviewers got things badly wrong. My own favourite recent example is a reviewer asking us to consider snow in our study of how rainfall impacts on salmonellosis in sub-tropical Queensland.

If we believe in peer review then we should believe in using peer review to improve peer review, and there are interesting studies that have highlighted problems with peer review. This introspective research is part of the growing field of meta-research or research on research, which uses research to examine and improve the entire research process. Such research is sorely needed considering that 85% of current health and medical research is wasted.

BMJ Open welcomes research on peer review and there are 54 papers in the category of “Medical publishing and peer review” including research on peer review as well other important meta-research issues, such as unpublished studies and how research is reported. The first paper in the category from 2011 examined reporting guidelines, and the most recent in 2016 looks at the reporting of conflicts of interest.

Can meta-research help when it comes to the difficult problem of recognition for review? To recognise good peer review we need to judge the quality of peer review, which means reviewing the reviewers.

An observational study compared the quality of reviews for reviewers suggested by authors with reviewers found by editors. The concern is that author-suggested reviewers may be too friendly, and in extreme cases be fake reviewers. The benefit of author-suggested reviewers is that it saves editors time in finding suitable experts. The study found no difference between the quality of reviews, but author-suggested reviewers were far more likely to recommend publication, with 64% of author-recommend reviewers recommending acceptance compared with just 35% of reviewers found by editors. It is possible that many authors suggest reviewers whose views agree with their own and whose work they have cited. Does this count as rigorous peer review, or would it be better if papers were critically analysed by researchers with a variety of views?

Another observational study examined peer reviewers comments for drug trials sponsored by industry compared with non-industry studies. The industry-sponsored studies had fewer comments on poor experimental design and inappropriate statistical analyses, and my guess is (based on personal experience) the industry trials employed more specialist staff because they have bigger budgets.

Both these studies had to spend time and effort reviewing the peer reviewers’ comments, and this extra effort is a key barrier to improving peer review.

Instead of reviewing every review a solution is to randomly check a sample of reviews. This would allow a reasonable number of reviews to be examined and graded in detail. If peer reviewers realise there’s a chance their work will be checked, then they should provide better reviews. The same idea is used by the tax office, who can’t afford to audit everyone but can increase compliance by random auditing.

Another benefit of regular random audits is that it would provide great data for tracking the quality of peer review over time, and allow a journal to ask whether things are getting better, or whether a policy change improved average review quality.

Of course the random tax audit works because there are severe penalties for those who are caught. A peer review audit would likely have to provide positive incentives, which could include a letter of commendation for the best reviews, promotion to the editorial board, or even the well-used incentive of money.

Dr Adrian Barnett is a statistician at the Queensland University of Technology, Brisbane. He works in meta-research which uses research to analyse how research works with the aim of making evidence-based recommendations to increase the value of research. @aidybarnett

The importance of reviewers – 2015

23 Dec, 15 | by Emma Gray

File:ScientificReview.jpg

The peer review process is central to scholarly research, a critical part of the publishing process and a method of quality control for the scientific community. While peer review can seem like a daunting, never-ending task at times, without it journals would not survive and continue to publish the important, accurate findings they do today.

At BMJ Open, we rely on our large bank of reviewers to help us ensure that the papers we publish are useful and of good quality. A good review can often require a great deal of work, and the continued success of the journal in the past year would not have been possible without the help of reviewers who provided us with their time, expertise and detailed comments. Our transparent, open peer review process is part of our aim to provide a home to sound medical research, making as much research available to the scientific community as possible while still upholding the BMJ’s core standards for research conduct.

It is important to recognise the contribution that reviewers make to the journal, and the essential nature of their work. BMJ Open would like to thank all those reviewers who have worked with us this year – we hope that you will continue to work with us in the year ahead! We are pleased to be able to continue with our reviewer discount of 25% for reviewers who submit manuscripts to us within 12 months of completing their review, and we would like to remind you that CME points are available for those who submit detailed comments within the given timeframe. Existing reviewers for the journal are encouraged to update their profiles on our submission system with their areas of expertise, so we can more easily allocate you appropriate papers – and, of course, we always encourage new reviewers to come forward and join our growing bank of referees.

We look forward to seeing what the next 12 months will hold for BMJ Open. With our fifth birthday on 23rd February 2016, we hope that the journal will continue to go from strength to strength, with the help of our reviewers and authors.

Peer Review Week: An analysis of peer review style and quality

30 Sep, 15 | by Fay Pearson

This week celebrates the first ever Peer Review Week; a collaborative concept from ORCID, Wiley, Sense About Science and ScienceOpen, to highlight and celebrate the invaluable role peer review plays in scientific and medical publishing.

Here at BMJ Open we are, of course, advocates of open peer review and as such are pleased to be publishing a timely research article by our friends at Biomed Central.

The paper, from Maria Kowalczuk et al., is a retrospective analysis of the quality of referee reports from author-suggested and non-author-suggested reviewers in open or single blind peer review journals.

Their objective was to elucidate whether reviews from peers suggested by authors would show bias in quality and decision recommendation compared to reviewers selected by other means. They also aimed to assess whether open review vs. single blind review had an impact on quality and recommendation. To achieve this, the study looked at 200 reviewer reports submitted in 2010-2011 to BMC Microbiology, 200 submitted to BMC Infectious Diseases, and 400 that were submitted to the Journal of Inflammation, these journals use single blind peer review, open peer review and a combination of the two, due to policy change, respectively. Comparisons were made by assessing the quality of report (using the Review Quality Instrument), by analysing the editorial recommendation made, and with author surveys. After statistical analysis of the data, they could conclude that the reports from reviewers suggested by the authors were of comparable quality but were more likely to suggest publication.

They also conclude that the open peer review reports were of a slightly higher quality than those using single blind review. These findings are in line with those from the randomised trial conducted by The BMJ, after they became one of the first journals to use open peer review in 1999, and similar to another study by the British Journal of Psychiatry.

As BMJ Open is open access with fully open peer review, we are always happy to see further research demonstrating the success of this model.  As we use a combination of both author suggested and non-author suggested reviewers (with an in-house filtering process), we couldn’t help but agree when we spoke to the paper’s authors and they said the following,  ‘It is reassuring that reviewers suggested by the authors provided reports of as good quality as reviewers found by editors using other means. Author-suggested reviewers tended to recommend acceptance of the manuscript more often than other reviewers, which highlights the important role of the editor in making the final decision on the manuscript’ .

The full text of the paper, Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models– Kowalczuk et al., can be found here: http://bmjopen.bmj.com/content/5/9/e008707.full

Thank you to our reviewers – 2014

18 Feb, 15 | by Fay Pearson

After a very busy year at BMJ Open, in which over 1100 papers were published, we would like to say a big thank you to all of our reviewers who contributed in 2014. All that we achieved last year would not have been possible without the help of the many referees who gave thorough and detailed reviews which are essential to our decision making process. The comments and evaluations provided by our reviewers allow us to maintain the quality and scientific validity necessary to the continued success of BMJ Open.

Since its conception BMJ Open has been tirelessly dedicated to provide a home for all properly conducted medical research from all disciplines and therapeutic areas. Our rigorous and transparent peer review process has been crucial to achieving this aim, because of this, we are pleased to be able to continue to offer the 25% reviewers discount to those who submit manuscripts within 12 months of completing their review and also give CME accreditation for the submission of timely and thorough comments.
The BMJ Open team is truly thankful for the continued hard work and support of our peer reviewers and we look forward to the year ahead.

Comparing the results from two surveys of BMJ Open authors

9 May, 14 | by Richard Sands, Managing Editor

 

BMJ Open authors were among those surveyed by Professor David J Solomon of Michigan State University for a study recently published in the journal PeerJ.

Needless to say we read this with great interest (we were unaware of the survey until the results were published). The survey reported a generally positive response to BMJ Open specifically and open access megajournals in general. The low response rates mean that many of the specific results should be interpreted with caution, though.

The response rate from BMJ Open authors was the lowest in the Solomon study (187/728 respondents; 26%). A possible explanation for this is that we were surveying the same people (authors published in 2013) at what seems to have been around the same time, asking some similar questions. BMJ journals regularly survey authors, readers and reviewers to help us stay in touch with the research community. There were some notable similarities in results and some major differences.

We achieved a 47% response rate (401/849) and thought it might be of interest to summarise our results which were roughly comparable with Professor Solomon’s.

Like Professor Solomon we surveyed our authors about the most important factors behind their decision to submit to BMJ Open. We offered 12 options from which authors could choose three. There was no ranking of these three choices.

The three most important reasons for submitting to BMJ Open in Solomon’s survey were

  • the quality of the journal (28%)
  • reputation of the publisher (18%)
  • the impact factor (IF) (13.5).

In our survey, impact factor was much less important. The three most selected options were

  • open access (59%)
  • BMJ Group branded journal (50%)
  • speed of review (37%).

Reputation of the journal (34%) was the fourth most selected in our survey (the most comparable option we had to Solomon’s ‘quality of the journal’). Impact factor was only the ninth most important reason given (13%). The most popular option in our survey without a rough equivalent in Solomon’s was ‘ease of transfer from another BMJ journal’, selected by 29% of respondents as one of their three most important reasons for submitting.

Although open access was the most-selected reason for submitting to BMJ Open in our survey, 84% of respondents believed publishing in an open access journal was not a requirement of their funder or their institution.

66% of our respondents said BMJ Open was not their first choice for submission, similar to the 68% in Solomon’s paper. The broad scope of the journal was a 35% said they used institutional funds to pay the publishing charge followed by 29% who said they used a direct grant. The number who received a waiver (9%) in our survey was roughly similar to those in Prof. Solomon’s survey (11.4%); the actual figure for 2013 was around 10%.

Further comments
BMJ Open’s IF was announced in July 2013. Many of the authors surveyed may have submitted to and/or published in the journal before it was announced. This may make the answers that relate to its importance when submitting less reliable (in both surveys). Alternatively there may be a balance between authors who didn’t care that no IF had been announced and those that would not have submitted if the journal didn’t have one.

Though omitted as an option from the Solomon survey BMJ also has institutional membership schemes that cover APCs and you can read more about them here.

With regard to publishing preliminary findings, BMJ Open publishes research protocols as well as results papers. So some of the authors surveyed would not have been publishing any research findings in BMJ Open.

It was nice to see that Professor Solomon opted to make the peer review comments open. We use open review and are glad to see more journals bringing transparency into the review process.

We’ll gloss over BMJ Open being referred to as BMC Open. Twice …

 

Thank you to our reviewers – 2013

16 Jan, 14 | by sjohar

Peer review is a fundamental part of publishing. Perhaps nowhere more so than in the open access field, which is often more scrutinised than other traditional publishing routes. Recognising this, the BMJ Open team would like to thank all 2725 peer reviewers who refereed for the journal in 2013 – your advice and considered remarks were essential in ensuring the quality and scientific validity of our articles.

At BMJ Open, we are pleased to have several policies in place that highlight the importance of peer review. For example, our open peer review policy allows complete transparency over the history of an article, and gives credit to specific referees through the disclosure of their names and the publication of their comments. We also provide reviewers with a 25% discount on the article-publishing charge of any manuscript they subsequently submit as an author and give CME accreditation for the submission of timely and thorough comments.

BMJ Open appreciates the support and hard work of all the peer reviewers who gave their valuable time in contributing to the journal throughout 2013. We hope to work with you again and look forward to forging new relationships in 2014. Given our continued success since our launch, we cannot wait to see what this forthcoming year will bring.

BMJ Open’s first birthday

23 Feb, 12 | by Richard Sands, Managing Editor

 

It is a year today that BMJ Open published its first papers: prompting donuts all round!

We have now published over 230 open access research articles, covering niche topics and major public health issues alike.

Several articles have received plenty of press coverage. Some have prompted considerable national debate, such as this paper suggesting that there would be major benefits to the rest of the UK if their diets improved to the level of England’s. Others, such as this paper on locked-in syndrome, have received widespread international coverage. Many articles now have thousands of downloads.

Wider exposure can bring closer scrutiny and some articles have received plenty of robust critique after publication. Everything we publish has received peer review, quite often including statistical review. We publish reviews of accepted articles so the justification for acceptance is clear. But research thrives on debate and we would be delighted to receive more comments – positive or (constructively) critical.

After just a year of publication we feel the journal is already making its mark. We have published papers from first-time authors and experienced researchers, offering a straightforward route to publication via fair, transparent peer review.

Many authors have also benefited from our willingness to waive article payment charges for those without the means to pay.

In our next year we look forward to publishing an even wider range of papers, hopefully with more authors taking advantage of our partnership with the Dryad repository to share their research data.

Here’s to the next 12 months …

BMJ joins the ‘open’ debate

31 Jan, 11 | by Richard Sands, Managing Editor

BMJ Open is the first BMJ Group journal to use fully open peer review

Revealing to authors who has peer reviewed their article has helped to make the process fairer and increase the credit for reviewers, according to a recent editorial published by the BMJ.

BMJ Deputy Editor, Trish Groves presents her perspectives on the advantages of transparency in peer review using the BMJ – with more than 10 years’ experience in open peer review – as an example of success.  The article explores alternate models of journal peer review and references the findings of a recently published randomised controlled trial which suggested that ‘telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website, had no important effect on review quality’.

Read the full article by Dr Trish Groves here:

In a counter-debate published in the BMJ, Karim Khan, Editor in Chief of the BJSM argues that open peer review may dissuade reviewers from offering a true judgement of a paper, and in some cases, enable a jealous rival to sabotage the review process. He takes a closer look at closed peer review, where reviewers can opt for anonymity.

Karim Khan’s article is available in full here:

Review for BMJ Open

2 Nov, 10 | by Richard Sands, Managing Editor

Keep your analytical skills sharp, receive public acknowledgement of your efforts and save money on publication charges!

BMJ Open is now reviewing articles for publication. Peer reviewing is a good way to hone your analytical skills and to get a ‘first look’ at forthcoming research. Our system of open peer review – publishing reviewers’ comments – provides public acknowledgement of your input. It is also helpful if you need to demonstrate your reviewing activity for continuing professional development purposes.

You can register as a reviewer at our submission site; once you have registered, drop us an email and let us know.

In recognition of your support, as a BMJ Open reviewer you will receive a 25% discount on article processing charges if you are the corresponding author of a paper submitted to the journal within a year of returning your review.

Our instructions for reviewers are here.