After Michael Lewis’s 2003 book Moneyball revealed that statistical analysis seemed to offer an edge to baseball’s Oakland Athletics, reformers in many other fields tried to emulate the team’s success. There was ‘moneyball for government,’ ‘moneyball for education,’ ‘moneyball for policing’ – initiatives founded on the idea that the introduction of data analysis might upend tradition-bound domains and improve results.
Often forgotten was that Moneyball wasn’t solely about an effort to build the best baseball team: it was an effort to do as well as possible given extremely limited resources. Oakland, and its pioneering manager Billy Beane, knew that with the payroll of the New York Yankees, the investment in analytics to find undervalued players and make marginal improvements in game strategy might not have been needed. Oakland could simply have shelled out whatever it cost to hire proven stars. That wealthy teams also eventually adopted Oakland’s methods of talent evaluation and in-game strategy suggests their usefulness, but the origin of the statistical revolution in baseball was inseparable from the economics of the game.
This is a crucial distinction when thinking about analytics in healthcare. After all, in many healthcare contexts, higher costs do not guarantee better care.1 Of course, improving system performance is a worthy goal on its own, and it is tempting to think a baseball-style approach might work. Yet, as a recent Viewpoint in BMJ Quality & Safety suggests, the healthcare analytics revolution is yet to occur.
Perhaps the most convincing of the possible reasons the authors propose for the slow penetration of data analytics into healthcare is the simplest. Unlike in sports, where wins and losses provide an unambiguous and universal metric of success, healthcare organisations, health professionals, regulators, and patients often cannot agree about what constitutes ‘success’. Even apparently straightforward measures like morbidity and mortality are often context-dependent. Goals set from one perspective (reductions in complications or hospital-acquired infections) might in practice conflict with those of another (post-surgical quality-of-life; cost-effectiveness). If anything, the authors underplay the difficulties of integrating sports analytics into medicine; many readers will have their own list of reasons why it might not work.
Of course, data analysis is already a mainstay of modern healthcare, needed to address the multiple forms of quality measures already in place. That’s why the authors focus on the transition to what they refer to as “advanced analytics.” Though what counts as basic and advanced is in the eye of the beholder, one generally accepted distinction is that ‘basic’ analytics involve counting something already well-defined (clinical outcomes; infections; deaths) whereas ‘advanced’ analytics involve mining data-rich environments (video recordings; heat maps; self-tracking wearables) to discover novel strategies for improving performance.
The move to “advanced” analytics in sports was often justified by the claim that the basic measures failed to capture the essence of success: baseball’s traditional statistic of Runs Batted In (RBI) was eventually replaced by other measures, such as ‘exit velocity’ and ‘runs created’ because RBIs came to be seen as a poor proxy for the player’s hitting ability or overall offensive contribution. In healthcare, however, precisely because there is so little agreement on success, or about how to integrate competing measures of success, it is much less clear what advanced analytics might deliver. At times the authors elide the difference between the goal of turning every surgeon into the equivalent of Stephen Curry and the goal of making marginal improvements in settings of limited resources. Both might be worthy aims, but are likely to actually result in different and potentially opposing measures.
Moreover, the introduction of advanced analytics might still fall prey to the problem that the original Moneyball guru Bill James warned about in the early 1980s: if you shake any data set long enough and hard enough, some kind of correlation or inference will “fall out” of it. But that doesn’t mean such conclusions are useful or meaningful. Rather, analysts should begin not with the data but with the important questions, and then seek out the appropriate data to answer them. If every surgery were filmed using the same type of video- and radar-tracking that is now used to aid football and basketball coaches, it might be possible to discover some strategies that correspond with a specific outcome. But unless the outcome is a meaningful one, and the strategies widely implementable, the overall effect on the healthcare system will be nil.
This shouldn’t surprise us. Many fields have had difficulty moving from new data collection measures to meaningful improvements. Most metrics can be “gamed” and their use might distort from worthy goals or even potentially lead to worse outcomes if not implemented thoughtfully.2 Even in sports analytics, the idea that data enable a move away from bias and subjectivity is misleading. Sports statistics depend greatly on human judgment and expertise to be useful.3
The upshot is that there is nothing natural or inevitable about turning to advanced data analysis. Doing so requires support from a range of stakeholders, systematic structural change, and an immense amount of effort to establish widely-agreed upon standards. For as much as it is surprising that healthcare hasn’t fully embraced the analytical revolution, it is also perhaps predictable.
–Christopher Phillips
Dr. Christopher J. Phillips (@cjphillips100) is an Associate Professor of History at Carnegie Mellon University.
1Hussey, PS, Wertheimer, S, & Mehrotra, A. 2013. The association between health care quality and cost: a systematic review. Annals of internal medicine, 158(1), 27–34. https://doi.org/10.7326/0003-4819-158-1-201301010-00006.
2Muller, JS. The Tyranny of Metrics. 2018. Princeton University Press.
3Phillips, CJ. Scouting and Scoring: How We Know What We Know About Baseball. 2019. Princeton University Press.