When it comes to image interpretation, context can be all-important

A physician friend posted a photo on our group chat with the caption: “Where am I?” The photo showed a flower bed, backed by a stone wall. There were no clues. There must be thousands of similar scenes throughout the country, and for that matter in other countries. So I guessed, and I happened to guess correctly. My friend was impressed: “How on earth did you do that?” It certainly wasn’t because I recognized the flowers or the wall. I have little or no interest in gardening. My guess was based not on the elements within the photo, but largely on what I know about my friend. I won’t go too deeply into this for obvious reasons, but I know roughly where she lives, how she spends her time and a certain amount about her social and cultural heritage and preferences. From all that I could have a guess at where she might be visiting, and what destination she might choose to represent with this particular photo.

That’s a rather long-winded way of pointing out that when it comes to image interpretation, context can be all-important, and of course the same applies in radiology. Pattern recognition is important, but is usually not enough on its own. Accurate interpretation relies on a combination of pattern recognition and the application of contextual information. Radiology without context becomes a game of “Where’s Wally”, albeit a version of the game with rather perverse rules in which Wally is absent most of the time, often wears different outfits and also lends his stripy jumper to other individuals whose identification is of no interest or value.

Finding the right balance between the two elements of pattern recognition and context can be very challenging. There will be times when the two pull in different directions—it has all the features of “x”, but is that really likely in this particular patient? Or alternatively, patients presenting like this usually turn out to have “y”, but the images look nothing like that.

This has implications for the application of Artificial Intelligence to radiology. For a computer to interpret the image described above, it could either employ more sophisticated pattern recognition—the precise shade of the delphiniums or the arrangement of bricks in the wall might carry clues which the ordinary human observer would miss—or alternatively it could factor in contextual information. My strong suspicion is that it will need to do both.

Computers will certainly make use of better pattern recognition—quantification and textural analysis, for example, will extract information which is invisible to the human eye from standard Computed Tomography (CT) and Magnetic Resonance (MR) images. But it is in the application of contextual information, and crucially in balancing the relative importance of that information in a particular individual against the importance of features contained within the images themselves, that the real gains may be made.

Radiologists make these judgments all the time and, it has to be said, frequently come up with the wrong answer. An optimistic view of the future is that computers will help us to get it right more often.

This, by the way, may help to explain why radiologists can get so fractious when presented with less than the full and accurate clinical picture. Nobody wants to be a Wally-ologist.

Giles Maskell is a radiologist in Truro. He is past president of the Royal College of Radiologists.

Competing interests: None declared.