{"id":946080,"date":"2023-06-13T11:00:00","date_gmt":"2023-06-13T18:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=946080"},"modified":"2023-06-13T15:48:24","modified_gmt":"2023-06-13T22:48:24","slug":"accounting-for-past-imaging-studies-enhancing-radiology-ai-and-reporting","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/accounting-for-past-imaging-studies-enhancing-radiology-ai-and-reporting\/","title":{"rendered":"Accounting for past imaging studies: Enhancing radiology AI and reporting"},"content":{"rendered":"\n

The use of self-supervision from image-text pairs has been a key enabler in the development of scalable and flexible vision-language AI models in not only general domains but also in biomedical domains such as radiology. The goal in the radiology setting is to produce rich training signals without requiring manual labels so the models can learn to accurately recognize and locate findings in the images and relate them to content in radiology reports.<\/p>\n\n\n\n

Radiologists use radiology reports to describe imaging findings and offer a clinical diagnosis or a range of possible diagnoses, all of which can be influenced by considering the findings on previous imaging studies. In fact, comparisons with previous images are crucial for radiologists to make informed decisions. These comparisons can provide valuable context for determining whether a condition is a new concern or improving, deteriorating, or stable if an existing condition and can inform more appropriate treatment recommendations. Despite the importance of comparisons, current AI solutions for radiology often fall short in aligning images with report data because of the lack of access to prior scans. Current AI solutions also typically fail to account for the chronological progression of disease or imaging findings often present in biomedical datasets. This can lead to ambiguity in the model training process and can be risky in downstream applications such as automated report generation, where models may make up temporal content without access to past medical scans. In short, this limits the real-world applicability of such AI models to empower caregivers and augment existing workflows.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tPublication<\/span>\n\t\t\tLearning to Exploit Temporal Structure for Biomedical Vision-Language Processing<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

In our previous work<\/a>, we demonstrated that multimodal self-supervised learning of radiology images and reports can yield significant performance improvement in downstream applications of machine learning models, such as detecting the presence of medical conditions and localizing these findings within the images. In our latest study, which is being presented at the 2023 IEEE\/CVF Computer Vision and Pattern Recognition Conference (CVPR) (opens in new tab)<\/span><\/a>, we propose BioViL-T<\/a><\/em>, a self-supervised training framework that further increases the data efficiency of this learning paradigm by leveraging the temporal structure present in biomedical datasets. This approach enables the incorporation of temporal information and has the potential to perform complementary self-supervision without the need for additional data, resulting in improved predictive performance.<\/p>\n\n\n\n

Our proposed approach can handle missing or spatially misaligned images and can potentially scale to process a large number of prior images. By leveraging the existing temporal structure available in datasets, BioViL-T <\/em>achieves state-of-the-art results on several downstream benchmarks. We’ve made both our models (opens in new tab)<\/span><\/a> and source code (opens in new tab)<\/span><\/a> open source, allowing for a comprehensive exploration and validation of the results discussed in our study. We\u2019ve also released a new multimodal temporal benchmark dataset, MS-CXR-T (opens in new tab)<\/span><\/a>, to support further research into longitudinal modeling of medical images and text data.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\n\t
\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Azure AI Foundry Labs<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Get a glimpse of potential future directions for AI, with these experimental technologies from Microsoft Research.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tAzure AI Foundry\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Connecting the data points<\/h2>\n\n\n\n

Solving for the static case in vision-language processing\u2014that is, learning with pairs of single<\/em> images and captions\u2014is a natural first step in advancing the field. So it\u2019s not surprising that current biomedical vision-language processing work has largely focused on tasks that are dependent on features or abnormalities present at a single point in time\u2014what is a patient\u2019s current condition, and what is a likely diagnosis?\u2014treating image-text pairs such as x-rays and corresponding reports in today\u2019s datasets as independent data points. When prior imaging findings are referenced in reports, that information is often ignored or removed in the training process. Further, a lack of publicly available datasets containing longitudinal series of imaging examinations and reports has further challenged the incorporation of temporal information into medical imaging benchmarks.<\/p>\n\n\n\n

Thanks to our early and close collaboration with practicing radiologists and our long-standing work with Nuance, a leading provider of AI solutions in the radiology space that was acquired by Microsoft in 2022 (opens in new tab)<\/span><\/a>, we\u2019ve been able to better understand clinician workflow in the radiological imaging setting. That includes how radiology data is created, what its different components are, and how routinely radiologists refer to prior studies in the context of interpreting medical images. With these insights, we were able to identify temporal alignment of text across multiple images as a clinically significant research problem. To ground, or associate, report information such as \u201cpleural effusion has improved compared to previous study\u201d with the imaging modality requires access to the prior imaging study. We were able to tackle this challenge without <\/em>gathering additional data or annotations.<\/p>\n\n\n\n

As an innovative solution, we leveraged the metadata from de-identified public datasets like MIMIC-CXR (opens in new tab)<\/span><\/a>. This metadata preserves the original order and intervals of studies, allowing us to connect various images over time and observe disease progression. Developing more data efficient and smart solutions in the healthcare space, where data sources are scarce, is important if we want to develop meaningful AI solutions.<\/p>\n\n\n\n

\"An
Figure 1: The proposed self-supervised training framework BioViL-T leverages pairs of radiology reports and sequences of medical images. The training scheme does not require manual expert labels and can scale to a large amount of radiology data to pretrain image and text models required for downstream clinical applications.<\/figcaption><\/figure>\n\n\n\n

Addressing the challenges of longitudinal analysis<\/h2>\n\n\n\n

With current and prior images now available for comparison, the question became, how can a model reason about images coming from different time points<\/em>? Radiological imaging, especially with planar techniques like radiographs, may show noticeable variation. This can be influenced by factors such as the patient\u2019s posture during capture and the positioning of the device. Notably, these variations become more pronounced when images are taken with longer time gaps in between. To manage variations, current approaches to longitudinal analysis, largely used for fully supervised learning of image models only, require extensive preprocessing, such as image registration, a technique that attempts to align multiple images taken at different times from different viewpoints. In addition to better managing image variation, we wanted a framework that could be applied to cases in which prior images weren\u2019t relevant or available and the task involved only one image.<\/p>\n\n\n\n

We designed BioViL-T with these challenges in mind. Its main components are a multi-image encoder, consisting of both a vision transformer and a convolutional neural network (CNN), and a text encoder. As illustrated in Figure 1, in the multi-image encoder, each input image is first encoded with the CNN model to independently extract findings, such as opacities, present in each medical scan. Here, the CNN counteracts the large data demands of transformer-based architectures through its efficiency in extracting lower-level semantic features.<\/p>\n\n\n\n

At the next stage, the features across time points are matched and compared in the vision transformer block, then aggregated into a single joint representation incorporating both current and historical radiological information. It\u2019s important to note that the transformer architecture can adapt to either single- or multi-image scenarios, thereby better handling situations in which past images are unavailable, such as when there\u2019s no relevant image history. Additionally, a cross-attention mechanism across image regions reduces the need for extensive preprocessing, addressing potential variations across images.

In the final stage, the multi-image encoder is jointly trained with the text encoder to match the image representations with their text counterparts using masked modeling and contrastive supervision techniques. To improve text representations and model supervision, we utilize the domain-specific text encoder CXR-BERT-general (opens in new tab)<\/span><\/a>, which is pretrained on clinical text corpora and built on a clinical vocabulary.<\/p>\n\n\n\n

\"Two
Figure 2: Example of current (left) and prior (right) chest x-ray scans. The attention maps computed within the vision transformer show (in purple) how the model interprets disease progression by focusing on these image regions. In this example, the airspace disease seen in the left lung lobe has improved since the prior acquisition.<\/figcaption><\/figure>\n\n\n\n

Grounded model prediction<\/h2>\n\n\n\n

In our work, we found that linking multiple images during pretraining makes for both better language and vision representations, enabling the AI model to better associate information present in both the text and the images. This means that when given a radiology report of a chest x-ray, for example, with the description \u201cincreased opacities in the left lower lung compared with prior examination,\u201d a model can more accurately identify, locate, and compare findings, such as opacities. This improved alignment between data modalities is crucial because it allows the model to provide more accurate and relevant insights, such as identifying abnormalities in medical images, generating more accurate diagnostic reports, or tracking the progression of a disease over time.<\/em><\/p>\n\n\n\n

Two findings were particularly insightful for us during our experimentation with BioViL-T:<\/p>\n\n\n\n