By David Hunter

Writing in his usual uncompromising style Ben Goldacre describes the latest carry-ons in the “trial” carried out in Durham by the Council on whether fish oils improve GCSE performance:

You’ll remember the Durham fish oil “trial” story, possibly the greatest example of scientific incompetence ever documented from a local authority.

Initially they said – to blanket media coverage – that they were running a trial on fish oils, giving pills to 3,000 children to see if it improved GCSE performance. I pointed out, along with several academics, that their experiment was incompetently designed, for no good reason, and so would only produce false positive results. They responded that this was okay, as they hadn’t called it a “trial”.

Now despite not being a “proper” trial, they bizarrely are reporting their results, although as Goldacre points out their analysis is woeful.

“Detailed analysis of the outcome of the initiative,” they say “shows that pupils who took the Omega-3 supplement did better than those who did not.” Hardly. Let’s try to disentangle what they think they’ve done.

“Initially, just over 3,000 Year 11 pupils began the study, taking the Omega-3 tablets at school and at home. By the time GCSE examinations came around, 832 pupils had 80 per cent or greater compliance.” This is appalling. 2,168 of their subjects dropped out of the trial: they must count these people in the results. They do not. This makes the rest of their claimed results even more meaningless.

“Mr Ford and his colleagues then sought to identify the same number of Year 11 pupils who had not taken the supplement and match them to those who had, according to school, gender, prior attainment and social background.” They originally said they were going to compare childrens’ predicted GCSE performance (whatever on earth that means) against actual performance. Of course, they refused to say how they would analyse this, despite hundreds of requests: and it is vitally important that people performing experiments are clear what they are measuring, and how they will analyse it, before they begin, otherwise they can move the goalposts and get a false positive result afterwards, by slicing the cake a dozen different ways.

“The GCSE results of 629 ‘matched pairs’ – fish oil takers and non-fish oil takers – were then analysed.” Who are these 629? I thought it was 832? But more importantly, by selectively only looking at the results from the pupils who were most highly adherent to the capsules regime, they have skewed their sample, entirely unnecessarily. They have, in fact, simply discovered that school performance is better in children who are more highly adherent to a school regime involving pills, and who are, in all probability, also more adherent to everything at school, harder working, better performing, from completely different families, with higher aspirations, and so on. If you wanted to design an experiment to produce a spurious false positive result, you could not do any better than this.

Anders Sandburg has already discussed over at Practical Ethics the unfortunate likely impact of this research: The price of ignorance: the Durham study and research ethics and there are other concerns to be raised about this poor quality science and indeed the risks it poised for no possible benefit for research participants.

However I instead want to focus on what might be done to avoid it occurring again largely by making an unpopular suggestion. This is of course that all research involving human subjects big or small, NHS or not, medical or social science, ought to get reviewed by an appropriate independent research ethics committee before being allowed to go ahead.

Here in the UK we have a rigorous system of ethical review for projects involving the staff, patients and resources of the National Health Service, and some (depending on which bit of the UK you are in) other research activities are legally required to go through that system as well. (Primarily research involving clinical trials, human tissue & those lacking mental capacity). However if you fall outside those categories then there is no requirement for review of the research by an appropriately constituted independent committee.

The result of this “system” is precisely that research like this can be carried out, the only time regulation may occur is legally if something goes seriously wrong (so for example if one of the children involved in this trial had a problem with bleeding that was arguably exacerbated by the fish oil pills).

It is difficult to see a justification for the present system. It might be thought that research related to the NHS is likely to be more risky, but if anything then that justifies a staggered system of ethical review, proportional to the likely risk (not that I am fond of that notion) rather than a two tiered system where some but not all of the highly risky research gets rigorous review and the rest gets no review at all. What the UK needs is a comprehensive and coherent system for the review of research.