Is US health reform a “monumental system transformation or a fatally flawed compromise?” This was the question addressed last night by Alan Gerber, doctor, economist, health policy expert, government adviser, and holder of five chairs at Stanford University (enough, pointed out by the chair of the meeting, for a one person dinner party) in the Office of Health Economics annual lecture. He gave his answer at the very start of his lecture: “Yes.”

And yes is clearly the right answer because the health reform bill is so complex and contains so much that it’s impossible to predict its effects. (It took the government printing office a month just to print the bill because it had so many amendments.) Although policy makers and analysts continue to treat healthcare systems as if they are machines where mechanical changes will have predictable effects, we know that health systems are “complex adaptive systems” where even small and simple changes may have unexpected results.

Gerber saw three factors that meant that reform was unavoidable and the status quo unsustainable. The first, the one emphasised by Obama, was that many millions had no health insurance. The brutal truth, however, is that this was not a big enough problem to move the electorate. A more important problem for policymakers is the “value deficit.” Fifty years ago the US had similar health expenditure and life expectancy to other developed countries; now it has much higher expenditure and lower life expectancy. Most of the American public is unaware of this problem, still believing that the US has the best health care in the world. But it was the third factor that convinced politicians in congress that something would have to be done – the ballooning costs of Medicare. Reform was driven ultimately by money.

Optimists within the White House see “game changers” that will mean that the outcome of the bill will be system transformation. These include emphasising better health rather than more health services, giving doctors and hospitals more financial responsibility, obliging health plans to compete on cost and quality, and constantly experimenting with new ways to pay doctors and hospitals.

One innovation that excites Gerber is the creation of an Independent Payment Advisory Board that will be able to make changes in federal health programmes with minimal oversight from Congress. One of the big drivers of rising costs has been the traditional pork barrelling whereby politicians insist on new technologies and benefits for their friends and supporters. But this new board mustn’t “ration” health care (the policy makers are looking for a definition of “ration”) and can’t rule on doctor and hospital costs before 2019 – and they are 70% of costs. Gerber also has hopes for “comparative effectiveness research.” We in Britain tend to think of this as the Americans discovering evidence based medicine 20 years after everybody else and needing to give it a different name. But it isn’t. Nor is it a version of the National Institute for Health and Clinical Excellence (NICE), not least because creating any cost effectiveness threshold is forbidden. Comparative effectiveness research is something much more subtle, a search for innovation in the processes of health care. (It is perhaps like poetry in being hard to define and being created in many forms, some of them, we hope, enchanting.)

Despite his bursts of optimism, Gerber saw many ways in which the reforms might make things worse. One is that implementation depends very much on the individual states – so don’t be surprised, he said, if reform works well in California and badly in Arkansas. Doctors may simply stop seeing Medicare and Medicaid patients because they are paid so poorly. Individual health insurance may unravel because of risk selection or the subsidies being too small. Providers may increase their monopoly power.

But things could also improve because doctors and hospitals integrate better, incentives to increase value work, prevention and primary care are both boosted, and new technologies reduce costs.

What happens will be determined by the effectiveness of implementation, which is highly complex, the behaviour of providers, employers, and individuals, and whether Congress feels the need to interfere.

At the end of an excellent talk I was reminded of a joke I heard years ago and have been telling ever since. Half way through a bout of US health reform one of the main architects of reform dies. He gets to heaven, meets God, and asks him whether the reforms will succeed. “Sure,” says God, “but not in my lifetime.”

Richard Smith was the editor of the BMJ until 2004.