Segmentation in biomedicine involves annotating pixels of important structures in medical images, such as organs and cells. Artificial intelligence models assist clinicians by highlighting pixels that may indicate signs of specific diseases or abnormalities.
However, while these models typically provide only one answer, medical image segmentation problems are often not black and white. Five expert human annotators may provide five different segmentations, but they may disagree about the presence or extent of nodule boundaries within a lung CT image.
“Having options helps with decision-making. Even knowing there is uncertainty in a medical image can influence someone's decision, so it's important to take this uncertainty into account.” ,” says Marianne Rakic, a PhD candidate in MIT computer science.
Rakic is lead author of a paper introducing a new AI tool to capture uncertainty in medical images, in collaboration with officials from MIT, the Broad Institute of Massachusetts Institute of Technology and Harvard University, and Massachusetts General Hospital.
The system, known as Tyche (named after the Greek god of chance), provides multiple plausible segmentations, each highlighting slightly different areas of a medical image. Users can specify the number of options that Tyche outputs and select the most appropriate option for their purpose.
Importantly, Tyche can tackle new segmentation tasks without retraining. Training is a data-intensive process that requires showing the model many examples and requires extensive machine learning experience.
Tyche does not require retraining and is therefore considered easier to use than other methods for clinicians and biomedical researchers. This could be used “out of the box” for a variety of tasks, from identifying lesions on X-rays of the lungs to identifying abnormalities on brain MRIs.
Ultimately, this system could improve diagnosis or aid biomedical research by drawing attention to important information that other AI tools might miss.
“Ambiguity is not well studied. If your model is completely missing a node that three experts say exists and two experts say doesn't exist, it's probably a warning.” ,” added lead author Adrian Dalca, assistant professor at Harvard Medical School and MGH. Scientist at MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).
Their co-authors include Hallee Wong, a graduate student in electrical engineering and computer science. Jose Javier Gonzalez Ortiz PhD '23; Beth Cimini, associate director of bioimage analysis at the Broad Institute. John Gutag, Dugald C. Jackson Professor of Computer Science and Electrical Engineering; Rakic will be introducing Tyche at his IEEE conference on computer vision and pattern recognition, and Tyche has been selected as a highlight.
Dealing with ambiguity
AI systems for medical image segmentation typically use neural networks. Loosely based on the human brain, a neural network is a machine learning model that consists of many interconnected layers of nodes, or neurons, that process data.
After speaking with collaborators at the Broad Institute and MGH who are using these systems, the researchers realized that two major issues limit their effectiveness. Since the model cannot capture uncertainty, it must be retrained for even slightly different segmentation tasks.
While some methods try to overcome one pitfall, tackling both problems with a single solution has proven particularly difficult, Rakic says.
“If you want to take ambiguity into account, you often have to use very complex models. ” she says.
The researchers built Tyche by modifying a simple neural network architecture.
The user first provides Tyche with some examples to illustrate the segmentation task. For example, the example can include multiple images of a cardiac MRI lesion segmented by different human experts so that the model can learn the task and see if there is any ambiguity.
The researchers found that 16 sample images, called the “context set,” were enough for the model to make good predictions, but there was no limit to the number of samples that could be used. Context sets allow Tyche to solve new tasks without retraining.
To help Tyche capture uncertainty, the researchers modified the neural network to output multiple predictions based on a single medical image input and context set. They arranged the layers of the network so that as the data moved from layer to layer, the candidate segmentations produced at each step could “talk” to each other and to the examples in the context set.
In this way, the model can solve the task while ensuring that the candidate segmentations are all slightly different.
“It's like rolling the dice. If the model rolls a 2, 3, or 4, but doesn't know that a 2 and a 4 have already been rolled, either one could come up again,” she says. Masu.
The training process has also been changed to reward maximizing the quality of the best predictions.
If a user requests five predictions, they will eventually see all five medical image segmentations created by Tyche, even if one is better than the others.
The researchers also developed a version of Tyche that can be used with existing pre-trained models for medical image segmentation. In this case, Tyche allows the model to output multiple candidates by making slight transformations to the image.
Better, faster predictions
Researchers tested Tyche on a dataset of annotated medical images and found that its predictions captured the diversity of human annotators, and its best predictions outperformed those of the baseline model. I understand that. Tyche ran faster than most models.
“Printing out multiple candidates and making sure they are different from each other can really give you an advantage,” Rakic says.
The researchers also found that Tyche was able to outperform more complex models trained using large, specialized datasets.
In future work, we will try to use a more flexible set of contexts, perhaps including text and multiple types of images. Additionally, we would like to explore ways to improve Tyche's worst-case predictions and enhance the system to recommend the best segmentation candidates.
The research was funded in part by the National Institutes of Health, the Eric and Wendy Schmidt Center at the Broad Institute of Massachusetts Institute of Technology and Harvard University, and Quanta Computer.