One approach to setting NICE’s cost per QALY threshold might be to survey the public.  In 2003 NICE and the Department of Health did just that, with a study  “assessing  the feasibility of estimating the value that the UK population might attach to a QALY.”

The report of that project was known as Social Value of a QALY (or SVQ) project, finally published in 2008. How this research was commissioned was unusual in that its design was put out to tender. Researchers, mainly those who eventually did the research, identified the key questions and designed the project which they then applied for and won. The research design focused on establishing which methods should be used and included leading advocates of each candidate method. Given the results of the project this is of some importance – not least since each method gave different results. The champions of each method predictably defended their position. This prompts the question: Was this research design feasible? Specifically, did it consider the possibility that each method would generate inconsistent results?

The report shows that while it was feasible to get people to answer the questions, the results varied widely according with the method employed. Two separate surveys of the public were carried out within the SVQ project. The first explored whether or not the public valued QALYs differently by age and severity. The two methods used were Discrete Choice Experiment and Grid (or Matching). The first  method suggested such variations did not matter, the second method suggested variations by age and severity of almost 3 to 1.

The second survey also used two methods, Standard Gamble and Willingness to Pay to estimate the value of a QALY. Again, very different cost per QALY results emerged from each method. Some were well above the current NICE thresholds, others well below.

Part of the fun  of reading the otherwise dense full SVQ  report has to do with identifying the attempts and failures of the different groups to reconcile their findings. The results are variously described as ‘challenging’, ‘discouraging’, and  “in need of reconciliation.” Ingenious possibilities are mooted in attempts to reconcile them. But the report more glumly also considers the option that someone other than the researchers may just have to choose the method. How is not explained.

The  report is coy about the fact that many respondents found the survey questions difficult. Worse, their answers were not consistent in the sense of being rational. Many apparently struggled to  understand probability. How questions were framed may have been important. The report acknowledges that respondents were often inconsistent and suggests that further work should include “cognitive testing,” a phrase which is not explained and which could mean many things, some of them slightly sinister. For instance, if only those capable of being consistent would be  surveyed.

I suggest two conclusions. First, the design of the study was flawed. That design was essentially a contest between methods but without rules for deciding the winner. Leaving aside the how long it took- over five  years from start to finish, or over half the lifetime of NICE!-  the authors failed to confront the answer to the basic question – was it  feasible to  establish the social value of a QALY. Instead they recommend more research. My reading of the report is that the answer is ‘no, not feasible’. What else  can be concluded given that  the public’s estimates depended on the method chosen, and no consensus exists as to the correct method? 

Second, the very question ‘what is the social value of a QALY? may be wrong. The briefing paper for the NICE methods review argued that the  appropriate £ per QALY threshold should be set by  the opportunity cost of NICE’s guidance.

If the NHS budget is given, the argument goes, then  NICE  should be a ‘threshold searcher’, searching that is for the cost effectiveness of those services displaced by its guidance. The key distinction is whether the budget sets the threshold or vice versa.

If the NHS works (broadly)  within budgets agreed by parliament, then the budget should set the threshold. The alternative of starting with a threshold, however determined, implies adjusting the  NHS budget to cover everything below the threshold. What a pity this was not recognised back in 2003 when the social value of a QALY project was conceived.

See also:

Head to Head: Should NICE’s threshold range for cost per QALY be raised?

James Raftery is a health economist with several decades’ experience of the NHS. He is professor of health technology assessment at Southampton University. A keen “NICE watcher,” he has provided economic input to technical assessment reports for NICE but has never been a member of any of its committees. The opinions expressed here are his personal views. He welcomes comments to his blog.