None of us knows how we will respond when we are given the diagnosis of a life-limiting illness, just as young men in the First World War did not know how they would respond when commanded to go “over the top.” Will we insist on every last iota of treatment, or will acceptance be a better option? Hope will enter the picture. We will be urged to “hope for the best.” Kind doctors will want to give us hope, keep hope alive. But will our search for hope delude us and increase the suffering of ourselves and those who love us? A study in Psycho-Oncology suggests so.

The authors, one of whom is my friend Eric Finkelstein who sent me a copy of the paper, studied 200 patients with advanced cancer or haematological malignancies whom their doctors thought might well not survive a year. The patients were asked how long they might live, and of the 111 who gave a prediction and died most (93%) died sooner than they expected and not one died later than they expected. On average these patients expected to survive over eight years, but actually survived less than nine months. The delusions went further in that 40% thought that their treatment would cure them.

This is the well-recognised phenonomen of “optimism bias.” Presumably at least some of the patients were by nature pessimists (I am, like Gramsci, a pessimist of the head, but an optimist of the heart), and it shows the power of optimism bias that it could overwhelm any inbuilt pessimism.

Another bias that operates in patients facing death is “the illusion of superiority”: half of the patients thought they would live longer than the average and a quarter much longer; less than 5% thought they would survive less than the average. Also at work were the biases of “motivated reasoning” and “self‐deception”: more than half of the patients believed that they were “very informed” about how their medical condition would change over time, while another third believed that they were “somewhat informed.” An important limitation of the study is that the researchers do not know exactly what the doctors had said to the patients, although all the doctors reported that they had informed all the patients of their prognoses. (Doctors, of course, are also prey to all the same biases as patients.)

How does hope influence these biases? It was known before this study that more hope means more unrealistic expectations of survival and cure, but does hope amplify our inbuilt biases?

The researchers measured hope in patients using the Herth Hope Index, which rates people’s level of agreement with 12 items—for example, “I have a positive outlook on life” and “I see possibilities in the midst of difficulties.” The summary score ranges from 12 (least hopeful) to 48 (most hopeful), and the patients reported high levels of hope: the mean score was 39.7 with 32 patients scoring the maximum and the lowest score reported by one patient was 15. The researchers found that the higher the levels of hope the greater all of the biases—in other words, hope is deceiving people.

Does this matter? Hope is said to have physical and psychological benefits, but having severely distorted judgements must cause harm. The most obvious harm is that patients continue with treatments that will bring them little or no benefit, but cause them side effects that will increase their distress and suffering. There will also be the “opportunity cost” that they will be sick and possibly in hospital when they could be enjoying their families, friends, and nature—or whatever brings joy to them. There may also be regret by patients and their families that the expected survival of seven years turns out to be nine months and that most of that nine months has been spent in treatment or in an ambulance travelling for treatment. Hope may also delay patients accepting palliative care, which we know can produce better outcomes.

Hope can also increase people’s vulnerability to quacks, of whom there are plenty. They will also go along with the suggested treatments of doctors whose instinct is for aggressive treatments. I think of Paul Kalanathi, the neurosurgeon who died young and whose oncologist was urging aggressive treatment the day he died. It is sadly much easier for doctors to respond to the delusions caused by hope by continuing treatments than launching into the “difficult conversation” and suggesting to patients that discontinuing “curative” treatment may be the best option. (We always tend to think of these decisions in relation to cancer, but they occur across all of medicine.)

My friend Eric is an economist by training, and the delusions caused by hope contribute to people having very expensive treatments with little or no benefit. For example, Bristol-Myers Squibb charged $80,352 for a course of Cetuximab, an epidermal growth factor receptor inhibitor, to treat non-small lung cancer. In a large European trial overall survival was increased by 1.2 months, meaning that the cost of an extra patient-year of life was $800 000. Martin Shkreli became infamous through buying the rights to Darapim (pyrethamine) and raising the price by 5000% from $13.50 to $750 per pill because of the possibility that it might prolong life in some patients with cancer.

Eric, a true economist, points out that it could be rational for individuals to spend every last penny on treatments that offer some “hope” as their money will be useless to them once they are dead. But in most high-income countries—and increasingly across the globe with the rise of universal health coverage—people are not paying for themselves. The deception of hope can contribute to excessive expenditure at the end of life, denying treatments to others, bankrupting health systems, and eroding financial support for education, housing, social care, and the environment, all of them important for health.

Were we to go back 70 years when there were few treatments that could prolong life and most deaths were rapid compared with now, then the delusions of hope would cause few problems. But as new treatments, many of them with high costs and limited benefits, are developed the range of options increases and more decisions must be made by patients, families, and their doctors. The deceptive capabilities of hope becoming steadily more important and relevant.

Perhaps writers and poets have a better understanding of hope than doctors and their patients. Elif Shafak writes in 10 Minutes 38 Seconds In This Strange World that “Hope is a hazardous chemical capable of triggering a chain reaction in the human soul.” T S Eliot wrote: “I said to my soul, be still, and wait without hope/For hope would be hope for the wrong thing.” And I found this poem, although I can’t find who wrote it but it’s possible that I wrote it myself:

Big and small hope

I hope you are well

Is small meaningless hope

A gesture, a verb to make a sentence.

I hope this cancer doesn’t kill me

Is a big hope, too big

Big enough to kill you.

Richard Smith was the editor of The BMJ until 2004.

Competing interest: RS and Eric Finkelstein work together unpaid on the Lancet Commission on the Value of Death. RS was also an unpaid member of the steering committee of a major trial that had Finkelstein as an author. https://www.nejm.org/doi/full/10.1056/NEJMoa1911965 RS did have his expenses paid to travel to a meeting of the steering committee in Sri Lanka.