(This version of the blog was copyedited for clarity and to add an explanatory note below on May 22nd and June 1st by Editor Karim Khan. Previous versions available).

We respectfully ask Drs Bermon and Garnier (2017, hereafter BG17) to share the underlying athletes’ performance data summarised in their British Journal of Sports Medicine (BJSM) study.[1] Specifically, we request the underlying data used to generate the summary statistics reported in Column 1 “All athletes” as reported in Tables 6 through 13 in the supplementary information that accompanies paper.

We requested the data from the Editor of BJSM who explained that the journal does not hold the data and who suggested we contact the paper’s authors. We emailed Drs. Bermon and Garnier, as well as the study’s sponsor, the International Association of Athletics Federations (IAAF), but to our knowledge have received no response.

In 2018, it is generally expected that researchers make their data available to other researchers for the purposes of reanalysis, replication and further research. This year, in the BMJ, Milton Packer wrote “an investigator who performs studies in people has implicitly agreed to a social contract, which includes the responsibility to make the raw data available for examination.”[2]

More broadly, organizations such as The Wellcome Trust, the Health 8 (WHO, UNICEF, UNFPA, UNAIDS, GFATM, GAVI, Bill & Melinda Gates Foundation, and the World Bank) and the Nature Group of publishers (representing some 600 scientific journals) have adopted statements on the importance of data sharing in science.[3] Nature explains their policy clearly: “An inherent principle of publication is that others should be able to replicate and build upon the authors’ published claims… authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications.”

We only request BG17’s performance data.  We do not seek to know how these performances were linked to endocrine data (e.g., by their tertile classifications), nor are we requesting the associated endocrine data.  Thus, no link between the performance and endocrine data will be possible if all performance data are provided. Further, since all of IAAF performance data from the 2011 and 2013 World Championships are already public, there are no privacy concerns. To the extent that there may be such concerns, it is the responsibility of the authors and the journal to ensure that replication is made possible by peers (e.g., via confidentiality agreements).

We request these data because:

• It is appropriate scientific practice;
• The findings of BG17 are cited as an important part of the basis for the new IAAF regulations regarding the classification of athletes with differences of sex development;
• The IAAF claims that the regulations are supported by data used in BG17, cited as “peer‐reviewed data”[4] – while the research paper has been peer-reviewed, we are unaware if data were available for peers to review; [Editor’s note, added May 22nd, 2018 – the data were not [the word ‘not’ added on June 1st , my apologies] examined by peer-reviewers. It is not standard practice at BJSM for reviewers to review original data; i.e, the review process for this paper followed standard practice and reviewers did not ask for the data].
• June 1st clarification given the ambiguity above. (1) The paper was not commissioned and underwent external peer-review. This is important as some media have reported otherwise. (2) The reviewers followed routine journal practice which, to date, is that they do not request, or analyse, original data. 
• We are unable to reconcile the reported methods and results of BG17 with publicly available performance data from IAAF.org and have arrived at some questions that can only be addressed with access to the original data.
• One example is sufficient to illustrate why these data are necessary for independent replication of the reported results:
  • BG17 reports an average 800m (for n = 64) female time of 121.80 seconds with a standard deviation of 5.42 seconds (data from BG17 Table 6, column 1);
  • Given a normal distribution of times, this implies that there should be ~10 times < 116.38 (that is, <1 standard deviation below the mean, ~16% of 64)
  • However, in Daegu 2011 and Moscow 2013 there are only two such times and one of these has since been disqualified due to doping (data via IAAF.org).
  • Thus, it is impossible to reconcile the reported results in BG17 with official IAAF data under the methods reported in the paper.
  • This, and similar, apparent inconsistencies can only be addressed with access to the underlying performance data for all athletes.

In short, allowing findings to be replicated by sharing underlying data not only reinforces scientific results that will stand the test of time, but also builds public trust in the legitimacy and integrity of research. The IAAF should welcome this opportunity for independent scrutiny of such important science.

We call upon Drs. Bermon and Garnier, and the IAAF, to make these data available for replication without further delay.

This letter is open to additional signatories, who may add their names to it by indicating so in the comments below.

Sincerely,

[Note: Affiliations are for identification purposes only]

Roger Pielke, Jr., University of Colorado

Ross Tucker, University of Cape Town

Erik Boye, Oslo University Hospital

References

[1] Bermon and Garnier (2017), Serum androgen levels and their relation to performance in track and field:  mass spectrometry results from 2127 observations in male and female elite athletes, British Journal of Sports Medicine. http://dx.doi.org/10.1136/bjsports-2017-097792

[2] https://www-bmj-com.colorado.idm.oclc.org/content/360/bmj.k510

[3] https://wellcome.ac.uk/what-we-do/our-work/sharing-research-data-improve-public-health-full-joint-statement-funders-health; https://www.nature.com/authors/policies/availability.html

[4] https://www.iaaf.org/download/download?filename=c402eb5b-5e40-4075-8970-d66fccb10d41.pdf&urlslug=Explanatory%20Notes%3A%20IAAF%20Eligibility%20Regulations%20for%20the%20Female%20Classification