A new artificial intelligence system is proving more effective at detecting breast cancer in mammograms than trained radiologists. The software, developed by researchers from Google Health, is not designed to replace human radiologists but instead assist and speed up current diagnostic processes.
Although mammograms are the best diagnostic tool for uncovering breast cancer clinicians currently have at their disposal, they are not a perfect screening tool. One in five breast cancers are not picked up by trained radiologists looking at mammograms. At the other end of the spectrum around 50 percent of women receiving annual mammograms will have a false-positive finding at some point over a 10-year period.
The new AI system was trained on a dataset of nearly 100,000 mammograms. The newly published study, evaluating the performance of the predictive software, was tested on two large mammogram sets, one from the UK and the other from the US. Neither of the two new test datasets were used to train the AI system.
In the US dataset, the software performed significantly better than human experts, producing 5.7 percent fewer false positive diagnoses. Even more impressively, the system recorded 9.4 percent fewer false negatives, suggesting it picked up several breast cancers that human experts missed.
The results in the UK dataset were less impressive but still significant. In the UK mammograms are screened by two separate radiologists, generally reducing the volume of errors, yet the AI system still bested the human experts with 1.2 percent fewer false positives, and 2.7 percent fewer false negatives.
A second arm to this evaluation study was an independent “reader study” conducted by an external research organization. In this test six US radiologists were pitted against the AI system, evaluating 500 randomly sampled mammograms from the US dataset.
Again, the AI system significantly outperformed the human radiologists on average. However, the study does note that although there were cancers picked up by the AI that were missed by all six human experts, there was at least one case picked up by all six humans but completely missed by the AI system. No clear patterns were identified to explain why these particular cases resulted in significant differences between human and AI, but the Google researchers suggest the future of these tools lies in assisting human experts rather than entirely replacing them.
“This is a great demonstration of how these technologies can enable and augment the human expert,” explains Dominic King, one of the UK Google Health researchers working on the project. “The AI system is saying ‘I think there may be an issue here, do you want to check?’”
Daniel Tse, one of the US Google Health researchers working on the project, affirmed this idea to Stat News, suggesting the goal is not to replace human experts but instead find a way to deploy this software in clinical spaces to help reduce human error.
“We believe this is just the beginning,” says Tse. “There are things that these models and technology are really good at, and there are things that radiologists, who spend their whole lives doing this, are really good at.”
The new study was published in the journal Nature.
