You don't need to be signed in to read BMJ Blogs, but you can register here to receive updates about other BMJ products and services via our site.

Censoring research

23 Jun, 16 | by Barry Pless

I am posting this for all Injury Prevention blog readers who are researchers or interested in research. I do so in part because John Langley is one of the pioneers in our field and was one of the Senior members of our editorial board from IP’s earliest days. But I also do so because the issue that prompted him to write this op-ed for his local Otago paper is by no means restricted to New Zealand. It is a widespread and important issue that has the potential to corrupt research in all manner of ways. IBP 

Read it and think carefully about the implications for your own work. Thanks to John for sharing it and for ODT for permission to reproduce it. Thankfully Injury Prevention, to the best of my knowledge, has never had to deal with this issue.

 Restrictive publication clauses in health research contracts

I was both pleased and disappointed to read the two ODT articles (20 February, 5 March) on this subject. Public airing of this important issue is long overdue. I was disappointed as I gained the impression that despite several scientists publicly expressing their concerns, the University of Otago Deputy Vice-Chancellor for Research appeared to have none. The Deputy Vice-Chancellor’s apparent lack of concern, however, is consistent with my experiences of research administration at the University.

For 20 years I was the Director of the University of Otago’s Injury Prevention Research Unit. This Unit was entirely dependent on funds from government agencies. Contrary to my expectations, the University of Otago did not pro-actively seek to protect me, or the public, from clauses in draft contracts that placed restrictions on publishing research findings. Protecting researchers increases the chances that they can serve the public interest, and meet the University’s legislated role of being the critic and conscience of society.

I spent a significant amount of time challenging clauses that would allow a government agency to censor a finding they disagreed with, or deny the right to publish work at all. On a couple of occasions during contract negotiations I was reminded by government agencies that they could purchase the outputs they wanted from other Universities or private organisations who would accept the restrictive clauses.

Why would Universities be accepting these clauses? I believe a key factor is the importance of, and competition associated with, generating research income. Collectively, NZ’s eight publicly funded universities derive approximately 15% of their income from research contracts. The success a University has, relative to others, in obtaining external research funds also has a significant role in determining the funding it receives from government through the Performance Based Research Funding scheme. Research income is critical to ensuring high quality research, thereby maintaining and enhancing a university’s reputation and thus attracting students, staff, and further funding. The purchasers of research can use the competition between universities, and researchers, to their advantage in getting restrictive publication clauses accepted.

Private research suppliers can accept restrictive publication clauses as they are typically uninterested in publishing in peer-reviewed journals and have no statutory or moral obligation to serve the public interest. While they would produce a research report for the purchaser, these reports are not frequently published, easy to access, or subject to rigorous quality control, e.g., independent peer review.

It was also my experience that many senior researchers did not care about the restrictive clauses. Why would this be so? Success in attracting research funding is, for most researchers, critical to pursuing their research, producing publications, and to promotion and public recognition.

This behaviour contrasts with Royal Society of New Zealand’s (RSNZ) Code of Professional Standards and Ethics in Science, Technology, and the Humanities. Section 8.1 of that code states that members of the Society must: “oppose any manipulation of results to meet the perceived needs or requirements of employers, funding agencies, the media or other clients and interested parties whether this be attempted before or after the relevant data have been obtained;”

I accept the right of a purchaser of research to see an advance copy of any paper for publication and to make any comments on it. But requiring modification beyond correcting factual errors is unacceptable. Even purchasers suggesting toning down a phrase here and there and putting in some qualifiers is problematic from a purchaser who is concerned to minimize bad publicity that might arise from the paper. It also places the researchers in a bind. Should they comply? If they don’t comply are they putting at risk future research funding from the purchaser? I suggest they might be. Why deal again with a ‘difficult’ group of researchers when you can purchase the work elsewhere?

The best approach to this issue is transparency. Make the deliberations between researchers and purchaser accessible to all as in the open review practiced by some scientific journals so readers can trace the discussion. This approach would not deal with contracts that explicitly prohibit the researchers publishing at all.

The Health Promotion Agency (HPA), a crown entity, charged with promoting healthy lifestyles recently put out a request for proposals (RFP) to assess whether the reduction in trading hours in Wellington has any impact on alcohol-related harm. The ‘indicative’ contract for this RFP stated: The Supplier will not publish the results of the Services undertaken pursuant to this Contract”. Only when challenged did HPA advise that it was a negotiable clause. Potential University researchers interested in bidding for such research should be able to take the ‘indicative’ contract as a reflection of the intent of the purchaser. Irrespective of this, preparing a high quality research proposal involves significant resources and many researchers would consider it was not worth the effort given the uncertainty of their right to publish.

In effect, some government purchasers are getting to decide what findings, if any, the public gets to see from research the public has paid for, either by pressuring some universities to accept restrictive clauses or by buying what they want from private suppliers.

Universities of New Zealand, the representative body of New Zealand’s eight universities, and the RSNZ need to enter in discussions with Government with a view to ensuring government research RFPs do not impose these restrictions. Interference with researchers ability to bid for, execute, and publish research compromises the role universities have as critics and conscience of society.

We need an independent audit of government research contracting to determine to what degree restrictive publication practices and the use of private suppliers is undermining the public’s right to be fully informed of the findings of research they have paid for.

Emeritus Prof John Langley

Concussion in sport: Changing the “Culture”

8 Jun, 16 | by Sheree Bekker

 

photo by Scott Beale / Laughing Squid This photo is licensed under a Creative Commons license. If you use this photo within the terms of the license or make special arrangements to use the photo, please list the photo credit as "Scott Beale / Laughing Squid" and link the credit to http://laughingsquid.com.

Photo by Scott Beale / Laughing Squid CC BY-NC-ND 2.0

[SB] Concussion remains the current hot topic in sports injury prevention. Injury Prevention has published many an article on the topic, including the recent An examination of concussion education programmes: a scoping review methodology. I have blogged about this here too.

I have invited Dr Johna Register-Mihalik (follow her on Twitter @johnamihalik), an assistant professor in the Department of Exercise and Sport Science at The University of North Carolina at Chapel Hill, to share her thoughts on concussion prevention with us. Dr Register-Mihalik serves as a research scientist at the Injury Prevention Research Centre at UNC-CH. She is on the Brain Injury Association of North Carolina Board and USA Baseball’s Medical and Safety Advisory Committee, and is also an active member of the National Athletic Trainers’ Association (NATA) and the American College of Sports Medicine (ACSM), amongst others.

[JRM] Few injuries receive the attention and the discussion that concussion does, especially those occurring in sports such as football, in both the mainstream and medical communities. Concussion is a complex injury that is the result of forces transmitted through the brain, resulting in a complex neurometabolic cascade leading to a wide array of signs and symptoms. The more we learn about concussion, as well as exposure to head impacts, the more we realize that we don’t know. It is an injury, that – perhaps because it is the brain that is affected – most in the sporting community are hyper aware of, regardless of level of participation.

However, due to this increasing attention and focus, one of the most common discussions and recommendations is to change the “culture” to improve safety concerning concussion and head trauma. However, when we say “change the culture”, what are we actually trying to change? Certainly, we can think of key things we want to see changed universally, such as: recognizing as many injuries as possible, student-athletes disclosing these injuries if they haven’t been identified (when possible), individuals and organizations consistently adhering to  no same day return to play, no student-athlete returning to play without clearance from a medical professional with the training to make the decision, and perhaps more general, a sporting environment that encourages safe practices, not playing through injury, and creates a positive environment for players, coaches, fans, and families alike. I am sure we could continue to add to this list of things we want to change or see as a consistent part of sport. However, most factors or behaviors around the culture of sport are complex and multifaceted.

For one, many aspects of the culture of sport that are at their core good, may progress to risky decisions down the road. Let’s take the concept of persistence and not quitting. While at the core those are good things, these may be the constructs that then drive playing through injuries, especially those like concussions that we cannot always see.  While no studies have directly addressed these relationships, data does highlight not wanting to let teammates or coaches down and not wanting to be pulled from play as primary reasons for not disclosing a concussion (McCrea, 2004, Register-Mihalik, 2013; Kerr, 2016). In addition, there is data to show that even some of the efforts that we direct to improve the culture, may have a negative effect (Kroshus et al, 2015) . This body of work highlights the importance of careful thoughtful messaging and imagery giving in our educational sessions, videos, and programs.

The type of change we talk about around concussion and head trauma is multifaceted. As such, the work to truly improve outcomes, improve behaviors, and create a “safe” environment (both social and physical) must also be multifaceted and affect multiple levels of the socio-ecological framework. It is also work that should be thoughtful, not sensationalized, and rooted in evidence – which can all be difficult things in the face of such heightened attention around a topic, where many have opinions. The work to truly insightful change will continue to require an interdiscplinary and community-based approach to not only develop the interventions and tools for change, but to see them be both successfully implemented and sustainable. I for one am excited about this work ahead and look forward to working with others to continue to see change happen for the better.

Fatality Free Friday | Road Safety | Australia

27 May, 16 | by Sheree Bekker

Fatality Free Friday is an initiative that started in Australia in 2007, and the campaign has continued to expand its operation and is now recognised as Australia’s only national community based road safety program.

Road safety is a complex issue but we believe that if drivers consciously think about road safety and safe driving for just one Friday in the year, that day’s toll – statistically about 5.3* deaths – could be reduced to zero.

That’s our aim. Not a single road death in Australia for just one day. Just one Fatality Free Friday.

We believe that if drivers are asked to actively concentrate on road safety and safe driving for just one day in the year, they’ll drive safer for the next few days too and, over time, change their outlook completely, consciously thinking about safety each and every day they get behind the wheel.

*DataSource: Australian Transport Safety Bureau

(From Fatality Free Friday)

Drivers can take the pledge to drive safely here.

On advocacy: championing young driver safety

2 May, 16 | by Sheree Bekker

9731139381_37fddfffa6_o

 

[SB] In this post, public health consultant Dr Sarah J Jones (follow her on Twitter @GDLSarahJones), an advocate for better safety for young drivers and all
who share the roads with them, shares her experiences on her efforts to advocate for Graduated Driver Licencing in the UK
. In her previous role, Sarah was an injury epidemiologist at Cardiff University, researching a range of injury prevention topics and completing a PhD on Child Pedestrian Injuries and Deprivation, a study that included analysis of the links between traffic calming distribution, deprivation and narrowing inequalities. 

[SJJ] It all began in 2008. I was in the final stages of Public Health Registrar Training when my supervisor told me to “go somewhere and do something”. My interests in road traffic crash prevention lead me first to Dot Begg at Otago, Dunedin and then on to Erin Cassell at Monash, Melbourne. The main objective, as well as a fascinating insight into how pandemic flu and other public health issues were being dealt with (I travelled the week after the 2009 swine flu pandemic first emerged), was an estimate of the effect that Graduated Driver Licensing (GDL) could have if implemented in the UK.

Back home, I presented what I had done, sat back, exceptionally pleased with myself, after all the case was now made, and awaited the “pat on the back” from my supervisor. “Nice” he said. “Now get it implemented”. “How” I lamely asked. “I don’t know. Work it out” was the response.

I’m still trying to work it out. Seven years on I have talked to a lot of people. I’ve given presentations to vaguely interested lay people in village halls, as well as to Members of Parliament. I have written articles for newspapers, magazines and peer reviewed journals. I have given newspaper, television and radio interviews, some live, and have participated in “phone-ins”.

Yet, we still do not have GDL in the UK. So, in seven years, I have achieved nothing.

I think people are more generally aware of GDL than when I started talking, but that may be completely unrelated to anything I have done. I am still looking for the guide on “How to bring about legislative change”, but there does not seem to be one. I have learned a lot about the policy process in the UK and about how reluctant people are to change their viewpoint, even in the face of overwhelming evidence to the contrary.

I always knew that the pace of Public Health change was painfully slow, but even with that knowledge continuing to advocate for change is difficult and often demoralising. It’s little wonder that we have become locked into a cycle of quick fixes (action that is just a proxy for effective action) to match political cycles that can then be quietly sidelined when they do not have the effect that was intended.

After 8 years in research / academia, my move into service based public health was my “translational research”. I am coming to believe that advocacy is the most important, but most overlooked area of both public health and injury prevention, partly because it is so difficult to measure the effect of what is being done. How we support people to become effective advocates is likely to be key to effective intervention prevention in the future.

[SB] I too have written on the vital importance of advocacy: here and here. As researchers, it is important to remember that we do not always need to disseminate purely our most up-to-date research results, or even the most innovative interventions; sometimes our work life’s work is bigger than that. Sometimes we need to advocate for the very heart that lies at our work: simple, credible information and resources that can make a difference in even one person’s life.

Book Review: Take Control of ICloud (Joe Kissell)

25 Apr, 16 | by Barry Pless

If you are anything like me and think you know enough about most aspects of the world of computers, think again. I recently was doing some work on old photos and had reason to consider whether I should use ICloud. I was uncertain how best to store the photos and, specifically, how to decide between a service like Flickr, my old friend, and ICloud. That was when I discovered that I really did not fully understand a great deal about ICloud – what it is , what it has to offer, or how to make it work well.

I started searching for advice with the help of Google but it seemed uneven. So I returned to see what the experts at Take Control had to offer. (Some while ago I reviewed two of their books and advised blog readers to make use of them.) As I expected, they had one on ICloud, written by a stalwart contributor, Joe Kissell.

This 167 page book – better still, think of it as a manual, guide, or handbook – is remarkably comprehensive and easy to follow. It is available as a pdf or mobi download or, I believe, on disc but not on paper. For $15 it’s an excellent investment unless you are 100% certain you will never want to use the cloud. But I assure you, not even folks older than me (an octogenarian), can safely assume that they will never need the cloud or be forced to use it. My prediction is that the Cloud will inevitably be part of your computing life, especially if you are a Mac user.

Of the 20 sections, there are three that alone justify the price of the purchase: Getting to Know ICloud, Set up ICloud, and any one or more of the sections devoted to Music, Photos, Documents, Mail, Contacts, or Calendars. The sections on finding your lost Iphone, perhaps Keychain, as well as the ICloud Web Site, complete the picture. The remaining sections will be useful to some readers but probably most will use one of those I have listed.

To give but one example of how clearly Kissell writes and how helpful the advice offered can be, this is an excerpt from the section on Photos.” iCloud Photo Library and the older My Photo Stream both sync photos across your devices via the cloud, but they differ in almost every other detail. You can use either or both, though most people will probably find that iCloud Photo Library alone is all that’s needed, as it essentially makes My Photo Stream superfluous (at least for those who have paid for enough iCloud storage to hold all their photos and videos).” Kissell unravels the confusion Apple’s changes introduced by explaining the other differences between My Photo Stream and ICloud Photo Library. He then discusses Photo Sharing and proceeds to untangle all three elements and compares them to help you choose which best serves your needs. My focus, and Kissell’s, is on the use of the Photo Library on ICloud. An important bonus for me, a Dropbox devotee, is the section where he compares the two noting where they overlap and where they do not.

To whatever extent this book may fail to answer all your Cloud questions, my guess is that the fault is not with Kissell but with Apple. The Cloud is confusing but it is now much less so having read this. I confess I am still not 100% comfortable but I am much more confident than I was a few weeks ago.

Contact for this book: https://www.takecontrolbooks.com

Barry Pless

On turning journal articles into blog posts

6 Apr, 16 | by Sheree Bekker

4604140980_c838e29396_z

Typing by Sebastien Wiertz CC BY 2.0

Blogging can be a divisive topic amongst academics. It has been called frivolous, and a distraction from ‘real’ work by some – whilst others wax lyrically that it is the real work.

Fact is:

Social media and blogs are not just add-ons to academic research, but a simple reflection of the passion underpinning it 

Recently, Injury Prevention Editor-in-Chief Brian Johnston shared how to write a blogpost from your journal article in eleven easy steps with the social media editorial team. I have decided to turn this into a pleasingly-meta blog post about turning your – yes yourInjury Prevention papers into posts for this very blog.

Blogging is a genre in and of itself. Today, blogs are as much a part of scholarly discourse as papers, presentations, and corridor conversations. This represents a new manner of sharing your life’s work with others, in a more relaxed and personal (if you like) way. This genre allows you to share more of your personal story behind a piece of research, or to highlight findings that are especially interesting, or merely to share your passion with a community of engaged scholars. Further:

Academically a blog post boosts citations for the core article itself. It advertises your journal article in ways that can get it far more widely read than just pushing the article out into the ether to sink or swim on its own. A post reaches other researchers in your discipline (those who are not digital hermits). And because it’s accessibly written, it travels well, goes overseas, gets re-tweeted and re-liked

On behalf of the social media team, I invite you to contact us if you would like to turn your Injury Prevention paper into a blog post. I will leave it up to you to read the “how to” detailed in the post shared above, and to peruse the tips and tricks given. My advice, however, is much simpler: write the blog post about your paper as if you are explaining it to your mother/a teenager/friends at a party (in other words: plain language!).

The most exciting part of blogging is that it has no rules, unlike academic writing. The most terrifying part of blogging is that it has no rules, unlike academic writing.

We promise we will make this process as painless as possible (we are all injury prevention researchers after all!). We can alleviate common fears such as not knowing where to start by doing an interview style blog post together. We can help you overcome the lack of formality by guiding the structure and content of the blog post. Or you can write freely as you please.

For examples of authors who have already done this, see: here, here, and here.

We are always looking to hear more about your thoughts, views, and experiences as injury prevention researchers.

Contact me via Twitter on @shereebekker, or email me at s [dot] bekker [at] federation [dot] edu [dot] au

Neuromuscular control program prevents lower limb injuries in men’s community Australian Football

23 Mar, 16 | by Angy El-Khatib

Injury researchers commonly study elite athletes because they participate in athletics year-round and thus have an increased chance of sustaining an injury. However, most athletes participate at the recreational or community level. (According to the NCAA, only 1.9% of American, high school, soccer players become professional players!)

Understanding that there is a difference between the physical profile of an elite player and a community player is imperative for making recommendations for injury risk factor management. The latest publication by Finch, et al. focuses on this matter.

In the current issue of Injury Prevention, Finch, et al. provide more evidence for targeted neuromuscular control exercise programs for decreasing knee injuries and lower limb injuries (LLI). The randomized-controlled trial (RCT) evaluated 18 male, non-elite, community Australian football clubs with data from more than 1,564 people. As profiled in the study, individuals who participated in the neuromuscular control intervention had a reduced rate of LLI as compared to control players.
The intervention was implemented as a “warm-up” prior to training. The program was based on the Preventing Australian Football Injuries through eXercise (PAFIX) study ; the control group participated in a “sham” program that included similar exercises. Although not in the published article, I was curious to know what PAFIX training fully entailed. The PAFIX training manuals include a detailed look at the neuromuscular exercises implemented, including a variety of plyometric training, stability and balance exercises, and change-of-direction drills.

Despite no statistically significant findings, this “analysis indicates that clinically relevant reduced knee injury and LLI rates can be achieved through targeted exercise training programmes in men’s community AF” (Australian Football).

This finding struck me as particularly important because of the vital role of community sport and recreation programs in providing nonelite athletes with the opportunity to gain the physical literacy skills needed to benefit from participation in sport and physical activity.

I look forward to more injury research which could potentially be generalized for nonelite, athletic communities.

I love a sunburnt country

10 Mar, 16 | by Bridie Scott-Parker

I received an email this week from a friend and colleague, alerting me to a report recently released by the Royal Flying Doctor Service:  The Royal Flying Doctor Service: Responding to injuries in remote and rural Australia.

The reports on falls, burns, poisonings, transport accidents, workplace injuries, drownings, self-harm and assault, with Australians living in remote and very remote areas:

  • Almost twice as likely as city residents to sustain an injury, and 2.2 times more likely to be hospitalised for an injury;
  • Four times more likely to die from a transport related injury than major city residents;
  • 3.8 times (remote) and 4.2 times (very remote) more likely to die from assault than major city residents; and
  • 1.7 times (remote) and 1.8 times (very remote) more likely to die from suicide than major city residents.

Injuries are a leading cause of death and hospitalisation among children—more children die from injuries (36%), than from cancer (19%) and diseases of the nervous system (11%) combined; Indigenous Australians; and agricultural workers.

While, as an injury prevention researcher, I encourage you all to become familiar with the report and the findings, the email sparked two memories for me. The first was a conversation with US colleagues after I invited them to come to visit Australia as we worked collaboratively. If you search the internet, you will find many animals might try to kill you. We have crocodiles, irukandji jellyfish, snakes, spiders, and my colleagues could share many more animals-of-death. Having lived in Australia my whole life, I reassured them that the likelihood of them meeting an untimely demise during their trip was pretty low, and the good news is they went home in one piece.

The second memory – sparked almost instantaneously – was a flashback to my childood. During primary school we learnt the most wonderful poem, My Country, by Dorothea Mackellar, by rote. This stanza in particular has always remained with me:

I love a sunburnt country,

A land of sweeping plains,

Of ragged mountain ranges,

Of droughts and flooding rains.

I love her far horizons,

I love her jewel sea,

Her beauty and her terror –

The wide brown land for me!

Despite the beauty of the poem, and that I love thinking about how this poem resonates with me, growing up in the country can be dangerous for many reasons, including the fact that medical assistance is not always close by.

Another editor hits the dust

8 Mar, 16 | by Barry Pless

We recently discovered that the Canadian Medical Association has fired the editor of its flagship journal, the CMAJ, for reasons that are entirely unclear and also dismissed the oversight committee put in place after the previous editor was fired. I am deeply disturbed. What follows is what I posted on WAME in response to comments by Lundberg and Kassirer about the dismissal. For the record, Injury Prevention, has never had any such conflict with its owners, the BMJ Publishing Group. I posted a similar note ‘to the editor’ in response to the editorial written by the board announcing this sad event. (PS I lay claim to coining the new adjective “Trumpist” to describe outrageous behaviour of this kind!)

Lundberg and Kassirer have responded with their customary perceptive and helpful suggestions. Undoubtedly this is about money not a ‘falling reputation’. As a Canadian I strongly favour Lundberg’s first option. The Journal is a source of pride and has been ever since John Hoey’s stewardship. In spite of what the CMA president or its executive may think, it has a sterling international reputation. To be guided by impact factors, (whatever they may be) is simply evidence of the CMA’s insufficient knowledge of the complex world of medical publishing. Hence I wonder why the onus for reform, etc. falls on the Journal rather than on the CMA leadership, especially given the history of its relationship with the Journal. I would be interested to learn whether the membership was consulted or informed about this decision before it was taken. if the membership was not involved it smells greatly of Trumpist behaviour. If the membership was not part of this vital decision perhaps the logical next step is not to focus on the Journal and its stalwart hanging-in editors, but rather on the CMA itself.

p values misused

8 Mar, 16 | by Barry Pless

Don’t ask me why but I follow Retraction Watch faithfully. Recently there was a posting about p values I thought would be of interest to our readers and contributors. Here it is verbatim.

“We’re using a common statistical test all wrong. Statisticians want to fix that.

After reading too many papers that either are not reproducible or contain statistical errors (or both), the American Statistical Association (ASA) has been roused to action. Today the group released six principles for the use and interpretation of p values. P-values are used to search for differences between groups or treatments, to evaluate relationships between variables of interest, and for many other purposes. But the ASA says they are widely misused. Here are the six principles from the ASA statement:

P-values can indicate how incompatible the data are with a specified statistical model.
P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
Proper inference requires full reporting and transparency.
A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
We spoke with Ron Wasserstein, ASA’s executive director, about the new principles.

Retraction Watch: Why release these “six principles” now? What about this moment in research history made this a particularly pertinent problem?

Ron Wasserstein: We were inspired to act because of the growing recognition of a reproducibility crisis in science (see, for example, the National Academy of Sciences recent report) and a tendency to blame statistical methods for the problem. The fact that editors of a scholarly journal – Basic and Applied Social Psychology — were so frustrated with research that misused and misinterpreted p-values that they decided to ban them in 2015 confirmed that a crisis of confidence was at hand, and we could no longer stand idly by.

Retraction Watch: Some of the principles seem straightforward, but I was curious about #2 – I often hear people describe the purpose of a p value as a way to estimate the probability the data were produced by random chance alone. Why is that a false belief?

Ron Wasserstein: Let’s think about what that statement would mean for a simplistic example. Suppose a new treatment for a serious disease is alleged to work better than the current treatment. We test the claim by matching 5 pairs of similarly ill patients and randomly assigning one to the current and one to the new treatment in each pair. The null hypothesis is that the new treatment and the old each have a 50-50 chance of producing the better outcome for any pair. If that’s true, the probability the new treatment will win for all five pairs is (½)5 = 1/32, or about 0.03. If the data show that the new treatment does produce a better outcome for all 5 pairs, the p-value is 0.03. It represents the probability of that result, under the assumption that the new and old treatments are equally likely to win. It is not the probability the new treatment and the old treatment are equally likely to win.

This is perhaps subtle, but it is not quibbling. It is a most basic logical fallacy to conclude something is true that you had to assume to be true in order to reach that conclusion. If you fall for that fallacy, then you will conclude there is only a 3% chance that the treatments are equally likely to produce the better outcome, and assign a 97% chance that the new treatment is better. You will have committed, as Vizzini says in “The Princess Bride,” a classic (and serious) blunder.

Retraction Watch: What are the biggest mistakes you see researchers make when using and interpreting p values?

Ron Wasserstein: There are several misinterpretations that are prevalent and problematic. The one I just mentioned is common. Another frequent misinterpretation is concluding that a null hypothesis is true because a computed p-value is large. There are other common misinterpretations as well. However, what concerns us even more are the misuses, particularly the misuse of statistical significance as an arbiter of scientific validity. Such misuse contributes to poor decision making and lack of reproducibility, and ultimately erodes not only the advance of science but also public confidence in science.

Retraction Watch: Do some fields publish more mistakes than others?

Ron Wasserstein: As far as I know, that question hasn’t been studied. My sense is that all scientific fields have glaring examples of mistakes, and all fields have beautiful examples of statistics done well. However, in general, the fields in which it is easiest to misuse p-values and statistical significance are those which have a lot of studies with multiple measurements on each participant or experimental unit. Such research presents the opportunity to p-hack your way to findings that likely have no scientific merit.

Retraction Watch: Can you elaborate on #4: “Proper inference requires full reporting and transparency”?

Ron Wasserstein: There is a lot to this, of course, but in short, from a statistical standpoint this means to keep track of and report all the decisions you made about your data, including the design and execution of the data collection and everything you did with that data during the data analysis process. Did you average across groups or combine groups in some way? Did you use the data to determine which variables to examine or control, or which data to include or exclude in the final analysis? How are missing observations handled? Did you add and drop variables until your regression models and coefficients passed a bright-line level of significance? Those decisions, and any other decisions you made about statistical analysis based on the data itself, need to be accounted for.

Retraction Watch: You note in a press release accompanying the ASA statement that you’re hoping research moves into a “post p<0.05” era – what do you mean by that? And if we don’t use p values, what do we use instead?

Ron Wasserstein: In the post p<0.05 era, scientific argumentation is not based on whether a p-value is small enough or not. Attention is paid to effect sizes and confidence intervals. Evidence is thought of as being continuous rather than some sort of dichotomy. (As a start to that thinking, if p-values are reported, we would see their numeric value rather than an inequality (p=.0168 rather than p<0.05)). All of the assumptions made that contribute information to inference should be examined, including the choices made regarding which data is analyzed and how. In the post p<0.05 era, sound statistical analysis will still be important, but no single numerical value, and certainly not the p-value, will substitute for thoughtful statistical and scientific reasoning.

Retraction Watch: Anything else you’d like to add?

Ron Wasserstein: If the statement succeeds in its purpose, we will know it because journals will stop using statistical significance to determine whether to accept an article. Instead, journals will be accepting papers based on clear and detailed description of the study design, execution, and analysis, having conclusions that are based on valid statistical interpretations and scientific arguments, and reported transparently and thoroughly enough to be rigorously scrutinized by others. I think this is what journal editors want to do, and some already do, but others are captivated by the seeming simplicity of statistical significance.

Pless note: I would be interested if any readers disagree. Please outline your views in 20 words or less. (Just kidding)

Latest from Injury Prevention

Latest from Injury Prevention