Richard Smith on assessing health technology assessment
11 Nov, 09 | by BMJ Group
The budget of Britain’s Health Technology Assessment programme has grown from £13m in 2006 to £88m in 2010, and it has conducted a swathe of trials on new technologies, published dozens of papers, and supported a study that won the BMJ paper of the year. But could it do even better? This was the question addressed at its biannual meeting yesterday, which I chaired.
Liam Donaldson, England’s chief medical officer, kicked of the meeting, and gave a “traffic light analysis” of how Britain is doing with the kind of research that is variously labelled health technology assessment, public health research, or health services research. It was green for studies of the distribution, causes, natural history, treatment, and prevention of disease, amber for studies of process and outcome and of patients’ views and experiences, and red for research into models of care.
Then he revisited a list of the “hard nuts” to crack in public health research that he’d compiled years ago. Britain has made progress with increasing the number of world class researchers in these subjects — but is still behind the US. The research was largely disparate and descriptive-and still is, although better. Good models of care remain hard to spread, and, he said (choosing his words carefully), NHS leaders and managers are still not as evidence literate as they might be. Big system changes are still introduced without adequate attention being paid to evidence, but piloting big change is, he said, “tricky.” And the “hardest nut of all” is influencing mainstream clinical activity.
When I came to pick out the good and the bad of the day, I saw lots that was good. One of the most pleasing things was the increase in publicly funded trials, something that many have wanted for a long time—not least to counter the bias of industry funded trials. There were 30 publicly funded trials in 2004 but will be about 100 by 2011, 80 of them funded by the health technology assessment programme.
The last issue of the BMJ that I edited contained several articles on the nightmare of trying to get ethical approval for multicentre studies, and there has been a huge improvement in this problem—although difficulties persist. And sadly a new bureaucratic barrier to research has been developed: ironically, research governance seems to be one of the main barriers to research.
The programme has done an excellent job of including “service users” or patients in the planning, prioritising, designing, and carrying out of research studies, and their voices were prominent in the meeting.
The highlight of the meeting for me was the session on “failures,” where Marion Campbell gave a marvellous presentation on her long drawn out efforts to do a placebo controlled trial of knee arthroscopy for osteoarthritis. Placebo controlled trials of surgery seem to be a step too far for Britain, but the whole fascinating story of trying to do the trial is written up in the journal of the programme.
When it came to suggesting improvements for the programme I worried that their conceptions of both technology and evidence may be too narrow. Most of the trials are about drugs and devices not models of care, but this is perhaps because other parts of the National Institute for Health Research are responsible for such evaluations. But there seems to be room for more different forms of evidence, including more observational studies.
Although the programme has done well at including patients, it seems to have scope to improve its interactions with managers and commissioners, people who are most able to improve the overall system.
And with the dramatic cuts that are coming in the NHS the programme will have to do as well as it can at proving value for money, which has to mean improvements for patients and in health not just publication sin journals. Similarly there is probably more work to be done on dissemination, making sure that the results of studies reach the people who are most able to implement the results and make change happen.
But being an old dog, I can remember a time when the NHS had almost no research capacity. The National Institute for Health Research is a huge improvement.
Competing interest: Richard Smith chaired the meeting but wasn’t paid—and he had no expenses because he cycled to and from the Royal College of Physicians. He did get a free lunch and several cups of coffee, which may have influenced him as his price is low.
You challenged those present to reflect on the presentations. I am a service user representative on one of the panels, so I was there and have done what you suggested in a blog posting, which you might find of some interest. (my apologies in advance for lack of brevity
I do not have your or Ben Goldacre’s journalistic skills)
I concentrate on Sir Liam’s presentation - viewed from my own experiences as a former health service manager (another “old dog” not one of the modern storm troopers)who has battled against the innumeracy of colleagues in the past. These reflections relate to a couple of the tough nuts that Sir Liam identified and to which you refer.
I also reflect on how Gerd Giggerenzer’s ideas about the use of natural frequencies might assist in finding common ground between researchers, clinicians and managers and might also - if incorporated into more of what the HTA publishes - assist the worried well overcome the exploitation of their own innumeracy.
I believe that national leaders - like Sir Liam - could do more to reverse many of the trends (of the last 20 odd years) to micromanage the service which has destroyed Trust and unfortunately continues to crush what little motivation is still left for local innovation.
David Symes
November 11th, 2009 at 3:48 pm
[...] The budget of Britain’s Health Technology Assessment programme has grown from £13m in 2006 to £88m in 2010, and it has conducted a swathe of trials on new technologies , published dozens of papers, and supported a study that won the BMJ … See the original post here: BMJ Group blogs: BMJ » Blog Archive » Richard Smith on assessing … [...]
BMJ Group blogs: BMJ » Blog Archive » Richard Smith on assessing …
November 11th, 2009 at 8:01 pm
[...] See original here: BMJ Group blogs: BMJ » Blog Archive » Richard Smith on assessing … [...]
BMJ Group blogs: BMJ » Blog Archive » Richard Smith on assessing … - penawebid
November 11th, 2009 at 9:29 pm
How can I follow Richard Smith? He raised the conference to another level – not taking any nonsense from anyone (including me) and getting us all engaged in facing some tough challenges about how “we” (the HTA) could do better, as well as giving us some encouragement about what we do well. At the end of the meeting, Richard challenged us to jot down three thoughts as we were going home. Unlike Richard, I did not cycle the 120 miles to the venue from Newark, but I did have a quick jot then a quick beer then a quick nap on the train home.
These then, were my three thoughts on the day:
The first is that there has never been a better time to do independent clinical research that is important to patients in the UK. With the HTA prioritising research questions from a range of sources through their commissioning board, a response mode board, themed calls, infrastructure in the form of registered Clinical Trials Units, Research Design Services and the Clinical Research Networks to deliver research, a range of funding opportunities right from early innovation ideas through to assessments of mature technologies from the HTA, how can anyone in the UK complain that they cannot get important research questions funded thesdays? The HTA has transformed my own specialty of dermatology, because like Heineken beer, it has reached those parts that other beers (funders) have not reached. Dermatology is now no longer an evidence-free zone, and we shall soon find out whether water softeners are any good for eczema and whether oral tetracyclines are safer than long term oral prednisolone for pemphigoid thanks to HTA investment. I know – it sounds like a party political broadcast on behalf of the HTA, but that is why I joined them and put myself forward to work for them and the National Institute of Health Research in general.
My second thought is that the organisation of research in the UK is a complex adaptive system. There are enough structures now in place to make the NHS the best place to do clinical research in the world, thanks to the vision and courage of people like Sally Davies and all those who support her. Apart from minor tweaks here and there, and a bit of oil to lubricate the interlinking cogs, the most important thing is to leave it alone and let it settle down so that it produces lots of good things. Despite the frustration of many researchers that clinicians are slow to adopt new evidence, I have personally witnessed a sea change in the attitude of clinical dermatologists towards what constitutes good external evidence and the willingness to act on evidence now included in our British Association of Dermatologists and NICE guidelines. I have also witnessed a big change in attitude towards research in general in the NHS now that Trust colleagues are bringing in sessional payments for their endeavours in supporting delivery of projects that make it to the national portfolio, especially with dedicated research nurses to consent and recruit and record outcomes on those kind enough to participate in our studies.
Finally, I learnt a lot from Jon Nicholl’s talk about what makes a successful application to the HTA. Two things in particular; the first is an excellent paper by Thorpe et al about a simple method to help you decide if your trial design is predominantly pragmatic or explanatory (Thorpe KE, Zwarenstein M, Oxman AD et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009 May 12;180(10):E47-57). The second was Jon’s thoughts about the value of “add-on” studies within grant applications and the need to distinguish between studies that are simply additive as in more results, or multiplicative – in other words additional integrated work that illuminates the main questions addressed in the study.
I think that’s four things, and I am sure there were lots more…..Thanks for a great day.
Competing interests: Hywel is an enthusiast of the NIHR and will be taking over as the new chair of the HTA Commissioning Board in 2010. Like Dr. Smith, he too had no payment for taking part,had several cups of coffee and a rather nice plate of chicken paprika.
Hywel Williams
November 12th, 2009 at 10:11 am
I too liked John Nicholl’s talk very much.
The Thorpe paper that Hywel refers to and Jon mentioned provides an excellent tool for visualising quickly how pragmatic a trial might be. Stupidly, I’d thought of trials as pragmatic or not pragmatic, but of course there is a range. I will share the Thorpe paper with my global health network.
Jon threw out many other interesting thoughts in a short talk, but there were two others that left me thinking as I cycled home.
Firstly, should a trial that loses 20% or more of participants should not be called a randomised trial–because people are not lost randomly?
Secondly, should QALYs, which combine length and quality of life, be the main outcome measure of trials? Jon argued that it would be better than trials that combine, for example, deaths and acute events into one measure.
Richard Smith
November 13th, 2009 at 9:19 am