Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model

Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, Pranav Rajpurkar

Proceedings of Machine Learning for Health, PMLR 158:209-219, 2021.

Abstract

We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.

Cite this Paper

BibTeX


@InProceedings{pmlr-v158-endo21a,
title = {Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model},
author = {Endo, Mark and Krishnan, Rayan and Krishna, Viswesh and Ng, Andrew Y. and Rajpurkar, Pranav},
booktitle = {Proceedings of Machine Learning for Health},
pages = {209--219},
year = {2021},
editor = {Roy, Subhrajit and Pfohl, Stephen and Rocheteau, Emma and Tadesse, Girmaw Abebe and Oala, Luis and Falck, Fabian and Zhou, Yuyin and Shen, Liyue and Zamzmi, Ghada and Mugambi, Purity and Zirikly, Ayah and McDermott, Matthew B. A. and Alsentzer, Emily},
volume = {158},
series = {Proceedings of Machine Learning Research},
month = {04 Dec},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v158/endo21a/endo21a.pdf},
url = {https://proceedings.mlr.press/v158/endo21a.html},
abstract = {We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.}
}


Endnote

%0 Conference Paper
%T Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model
%A Mark Endo
%A Rayan Krishnan
%A Viswesh Krishna
%A Andrew Y. Ng
%A Pranav Rajpurkar
%B Proceedings of Machine Learning for Health
%C Proceedings of Machine Learning Research
%D 2021
%E Subhrajit Roy
%E Stephen Pfohl
%E Emma Rocheteau
%E Girmaw Abebe Tadesse
%E Luis Oala
%E Fabian Falck
%E Yuyin Zhou
%E Liyue Shen
%E Ghada Zamzmi
%E Purity Mugambi
%E Ayah Zirikly
%E Matthew B. A. McDermott
%E Emily Alsentzer
%F pmlr-v158-endo21a
%I PMLR
%P 209--219
%U https://proceedings.mlr.press/v158/endo21a.html
%V 158
%X We propose CXR-RePaiR: a retrieval-based radiology report generation approach using a pre-trained contrastive language-image model. Our method generates clinically accurate reports on both in-distribution and out-of-distribution data. CXR-RePaiR outperforms or matches prior report generation methods on clinical metrics, achieving an average F$_1$ score of 0.352 ($\Delta$ + 7.98%) on an external radiology dataset (CheXpert). Further, we implement a compression approach used to reduce the size of the reference corpus and speed up the runtime of our retrieval method. With compression, our model maintains similar performance while producing reports 70% faster than the best generative model. Our approach can be broadly useful in improving the diagnostic performance and generalizability of report generation models and enabling their use in clinical workflows.


APA


Endo, M., Krishnan, R., Krishna, V., Ng, A.Y. & Rajpurkar, P.. (2021). Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model. Proceedings of Machine Learning for Health, in Proceedings of Machine Learning Research 158:209-219 Available from https://proceedings.mlr.press/v158/endo21a.html.


Related Material

Clinically Accurate Chest X-Ray Report Generation

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, Marzyeh Ghassemi

Proceedings of the 4th Machine Learning for Healthcare Conference, PMLR 106:249-269, 2019.

Abstract

The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMICCXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.

Cite this Paper

BibTeX


@InProceedings{pmlr-v106-liu19a,
title = {Clinically Accurate Chest X-Ray Report Generation},
author = {Liu, Guanxiong and Hsu, Tzu-Ming Harry and McDermott, Matthew and Boag, Willie and Weng, Wei-Hung and Szolovits, Peter and Ghassemi, Marzyeh},
booktitle = {Proceedings of the 4th Machine Learning for Healthcare Conference},
pages = {249--269},
year = {2019},
editor = {Doshi-Velez, Finale and Fackler, Jim and Jung, Ken and Kale, David and Ranganath, Rajesh and Wallace, Byron and Wiens, Jenna},
volume = {106},
series = {Proceedings of Machine Learning Research},
month = {09--10 Aug},
publisher = {PMLR},
pdf = {http://proceedings. mlr.press/v106/liu19a/liu19a.pdf},
url = {https://proceedings.mlr.press/v106/liu19a.html},
abstract = {The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMICCXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.}
}


Endnote

%0 Conference Paper
%T Clinically Accurate Chest X-Ray Report Generation
%A Guanxiong Liu
%A Tzu-Ming Harry Hsu
%A Matthew McDermott
%A Willie Boag
%A Wei-Hung Weng
%A Peter Szolovits
%A Marzyeh Ghassemi
%B Proceedings of the 4th Machine Learning for Healthcare Conference
%C Proceedings of Machine Learning Research
%D 2019
%E Finale Doshi-Velez
%E Jim Fackler
%E Ken Jung
%E David Kale
%E Rajesh Ranganath
%E Byron Wallace
%E Jenna Wiens
%F pmlr-v106-liu19a
%I PMLR
%P 249--269
%U https://proceedings.mlr.press/v106/liu19a.html
%V 106
%X The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMICCXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.


APA


Liu, G., Hsu, T.H., McDermott, M., Boag, W., Weng, W., Szolovits, P. & Ghassemi, M.. (2019). Clinically Accurate Chest X-Ray Report Generation. Proceedings of the 4th Machine Learning for Healthcare Conference, in Proceedings of Machine Learning Research 106:249-269 Available from https://proceedings.mlr.press/v106/liu19a.html.


Related Material

Chest x-ray and fluorography: what are the fundamental differences?

Chest fluorography is one of the screening methods of modern radiography. But specialists without experience in equipping clinics may face some difficulties in choosing the necessary medical equipment. Before you buy a device for diagnosing lung pathologies, you should find out what are the fundamental differences between fluorography and chest x-ray.

Chest X-ray

Fluorography is an x-ray based screening method that provides images of the chest on a fluorescent screen. Fluorography is performed with suspicion of tuberculosis, oncological diseases and pathological processes in the pulmonary system.

However, this method is considered superficial, as it does not give a clear idea of ​​the possible origin of the disease. As a rule, if the doctor suspects after fluorography, a clarifying chest x-ray may be required.

The technique is effective only in the primary study of lung pathologies. In this case, the pathological process can be detected only in direct projection.

Chest X-ray

X-ray is a diagnostic method that is also based on ionizing X-rays. The specialist sees the picture in full size on the screen.

X-ray machines examine the internal structures of organs. Through radiography, specialists can identify a wide range of pathological processes and evaluate the dynamics of treatment.

Currently, analog (film) and digital X-ray machines are used.

• The analog x-ray equipment displays images on film.
• Digital Equipment The allows you to take pictures in digital format.

Digital X-rays are used to examine the cardiovascular system, the musculoskeletal system, the respiratory system and diseases of the chest.

Which method is best?

Fluorography of the lungs is a method that is gradually becoming obsolete, giving way to more accurate radiography technologies. However, fluorography should not be abandoned, as this method has its advantages.

The most complete clinical picture

With the help of lung radiography, it is possible to obtain the most accurate clinical picture, accurately and reliably determine the state of deep tissues. Thus, radiography helps to detect the first signs of the disease already at an early stage of the development of the disease.

Fluorography is inferior to radiography in this regard, as it refers to more superficial methods of examining the chest. In the early stages, the method will not determine the signs of the disease.

Chest x-ray images are clearer and more reliable. When performing fluorography, images are displayed on the screen, and then “fixed” by photographing. The picture is not as clear as when taking x-rays.

Exam safety

Fluorography and radiography also differ in radiation exposure. With fluorography, the patient’s body is exposed to radiation, but the dose is not as significant as with radiography. However, modern digital X-ray machines are able to work effectively with minimal radiation doses.

Capabilities

A modern X-ray machine has a wider scope than a fluorograph, since it can be used not only to examine the lungs, but also other organs.

Chest X-ray allows you to evaluate the symmetry of the lung fields, determine the features of the structure of the lung roots, analyze the lung pattern, and see the transparency of the lung tissue.

Radiography is performed in two projections. This improves the quality of diagnostics, but because of this, the radiation dose increases.

Conclusions

• The method of chest fluorography is relevant for routine examinations and mass medical examinations, when it is necessary to assess the possible risk of developing tuberculosis and cancer in patients.
• Fluorographs due to lower radiation exposure can be used for patients (in the absence of contraindications) annually.
• Thus, fluorography is, first of all, a preventive method.
• However, the capabilities of the fluorograph are limited: the presence of a pathological process can be detected using the fluorography method.
• Chest X-ray is more often performed on special medical indications when certain pathological processes in the lungs are suspected.
• X-ray machines operate at a higher radiation dose and therefore are not recommended for routine examinations.
• At the same time, radiography is a more accurate method, as it allows you to establish a reliable diagnosis and, possibly, understand the cause of the pathological process.

Radiography – activities and specialists

Make an x-ray in Yekaterinburg

X-ray examinations are among the most common in modern medicine. X-rays are used for simple x-rays of bones and internal organs, fluorography, computed tomography, angiography, etc.

Based on the fact that X-ray radiation belongs to the group of radiation radiation, it (in a certain dose) can have a negative impact on human health. Conducting most modern methods of X-ray examination involves irradiating the subject with negligible doses of radiation, which are completely safe for human health.

X-ray methods of examination are used much less frequently in the case of pregnant women and children, however, even in these categories of patients, if necessary, an X-ray examination can be performed without significant risk to the development of pregnancy or the health of the child.

What are x-ray waves and what effect do they have on the human body?

X-rays are a form of electromagnetic radiation, other forms of which are light or radio waves. A characteristic feature of X-ray radiation is a very short wavelength, which allows this type of electromagnetic wave to carry a lot of energy, and gives it a high penetrating power. Unlike light, X-rays are able to penetrate through the human body (“shine through it”), which allows the radiologist to obtain images of the internal structures of the human body.

In essence, X-rays are “very strong light” that is not visible to the human eye, but can “see through” even dense objects such as metal plates.

Medical X-ray examinations (X-ray examinations) in many cases provide important information about the state of health of the examined person, and help the doctor to make an accurate diagnosis in the case of a number of complex diseases.

X-ray examination allows you to get images of the dense structures of the human body on photographic film (radiography) or on the screen (fluoroscopy).

The large penetrating power and energy of X-rays make them quite dangerous for the human body. X-ray radiation is one of the most common types of radiation. During the passage through the human body, X-rays interact with its molecules and ionize them. Simply put, X-rays are able to “break” the complex molecules and atoms of the human body into charged particles and active molecules. As in the case of other types of radiation, only X-rays of a certain intensity are considered dangerous, which affects the human body for a sufficiently long period of time. The vast majority of medical examinations that use X-rays use low-energy X-rays and irradiate the human body for very short periods of time, and therefore, even when they are repeated many times, they are considered practically harmless to humans.

The doses of x-rays that are used in a conventional x-ray of the chest or bones of the extremities cannot cause any immediate side effects and only very slightly (no more than 0.001%) increase the risk of developing cancer in the future.

Radiation dose measurement for X-ray examinations

As mentioned above, the effect of X-rays on the human body depends on their intensity and exposure time. The product of the radiation intensity and its duration represents the radiation dose.

The unit of measure for total human body dose is milliSievert (mSv). Also, other units of measurement are used to measure the dose of x-rays, including rad, rem, roentgen, and gray.

Different tissues and organs of the human body have different sensitivity to radiation, and therefore, the risk of radiation exposure of different parts of the body during x-ray examination varies significantly.
Term effective dose is used in relation to the risk of exposure to the whole body of a person. For example, during an x-ray examination of the head area, other parts of the body are practically not directly exposed to x-rays. However, in order to assess the risk presented to the health of the patient, it is not the dose of direct exposure to the examined area that is calculated, but the dose of total exposure to the body is determined – that is, the effective dose of exposure. The determination of the effective dose is carried out taking into account the relative sensitivity of different tissues exposed to radiation. Also, the effective dose makes it possible to compare the risk of X-ray studies with more common sources of exposure, such as, for example, background radiation, cosmic rays, etc.

Radiation dose calculation and radiological risk assessment

Below is a comparison of the effective dose of radiation received during the most commonly used diagnostic procedures using X-rays with natural exposure to which we are normally exposed throughout our lives. It should be noted that the doses indicated in the table are indicative and may vary depending on the devices used and methods of examination.

*1 rem = 10 mSv

Taking into account the latest data on the risk of radiation exposure to human health, a quantitative risk assessment is carried out only if a radiation dose is received above 5 rem (50 mSv) within one year (for adults in children), or if a radiation dose is received above 10 rem per year. throughout life, in addition to natural exposure.
There is clear medical evidence regarding the risks associated with high doses of radiation. In the event that the total exposure dose is below 10 rem (including natural exposure and occupational exposure), the risk of harm to health is either too low to be accurately assessed, or does not exist at all.

As a result of epidemiological studies among people exposed to relatively high doses of radiation (for example, people who survived the atomic bomb explosion in Japan in 1945) there were no side effects on the health of people who received low doses of radiation (less than 10 rem) for many years.

Natural exposure

X-ray examinations are far from the only source of radiation for humans. People are constantly exposed to radioactive radiation (including in the form of X-rays) from various sources, such as radioactive metals in the soil and cosmic radiation.

According to modern calculations, the exposure from one chest x-ray is approximately equal to the amount of radiation received in normal living conditions in 10 days.

X-ray safety level

Like many other medical procedures, x-ray diagnostics are not dangerous, if used carefully and rationally. Radiologists are trained to use the minimum dose of radiation necessary to obtain the desired result. The amount of radiation used in most medical examinations is very small, and the benefit of the examination almost always far outweighs the risk to the body.

X-rays affect the human body only at the moment the device switch is turned on. The duration of “transmission” of X-rays in the case of conventional radiography does not exceed a few milliseconds.

Lifetime collective exposure to x-rays

The decision to perform an X-ray examination must be medically justified and can only be made after comparing the likely benefit of the examination against the potential risk associated with exposure.

In the case of low-dose medical examinations, the decision to conduct an X-ray examination is usually a fairly straightforward task. In the case of studies using higher doses of radiation, such as computed tomography, and in the case of procedures involving contrast materials such as barium or iodine, the radiologist may take into account whether the patient has been previously exposed to x-rays, and if so, then in what quantity.
If you have had frequent x-rays and frequently change locations or physicians, record your entire history of medical examinations.

X-ray during pregnancy and lactation

Limitation of the use of X-ray examinations during pregnancy is associated with the potential risk of adverse effects of additional radiation on the development of the fetus.

Although the vast majority of medical procedures that use x-rays do not expose the developing child to critical radiation exposure and significant risk, in some cases there may be a small chance of negative effects of x-ray radiation on the fetus. The risk of having an X-ray examination depends on factors such as the length of pregnancy and the type of procedure performed.

X-ray examinations of the head, arms, legs or chest using special protective aprons for pregnant women, as a rule, the child is not directly exposed to X-rays and, therefore, the examination procedure is practically safe for him.

Only in rare cases, during pregnancy, it becomes necessary to perform an x-ray examination of the abdomen or pelvis, but even in this situation, the doctor may prescribe a special type of examination or, if possible, limit the number of examinations and the area of ​​radiation.

Routine abdominal x-rays are not considered to pose a significant risk to a child’s development. Procedures such as CT scans of the abdomen or pelvis expose the child to more radiation, but they also rarely result in developmental abnormalities.

Due to the fact that the vast majority of X-ray examinations in pregnant women are carried out for health reasons (for example, the need to exclude tuberculosis or pneumonia), the risk of these studies for the mother and unborn child is always incomparably lower than the possible harm that the examination can bring to them.

All x-ray procedures (plain x-ray, fluorography, computed tomography) are safe for breastfeeding mothers. X-rays do not affect the composition of breast milk. If x-ray examinations are necessary for a nursing mother, there is no need to interrupt breastfeeding or express milk.

In the case of breastfeeding mothers, only X-ray examinations that involve the introduction of radioactive substances (eg radioactive iodine) into the body are of some danger. Breastfeeding mothers should inform their doctors about lactation before such examinations, as some drugs used during the examination may pass into the milk. To avoid exposure of the baby to radioactive substances, doctors will likely advise the mother to interrupt feeding for a short time, depending on the type and amount of radioactive substance (radionuclide) used.

X-ray examinations of children

Although children are much more sensitive to the effects of radiation than adults, performing most types of radiographic examinations (even multiple sessions if necessary), but at a total dose below 50 mSv per year, does not pose a serious health hazard to the child.

As in the case of pregnant women, X-ray examination in childhood is carried out for health reasons and its risk is almost always much lower than the possible risk of the disease for which the examination is carried out.

How to remove radiation from the body?

In nature, there are a large number of sources of radiation, the carriers of which are various physical phenomena or chemicals.

In the case of X-rays, the carrier of radiation is electromagnetic waves, which disappear immediately after the X-ray machine is turned off, and are not able to accumulate in the human body, as is the case with various radioactive chemicals (for example, radioactive iodine). Due to the fact that the effect of X-ray radiation on the human body ends immediately after the completion of the examination, and the rays themselves do not accumulate in the human body and do not lead to the formation of radioactive substances, no procedures or therapeutic measures to “remove radiation from the body” after X-ray is not required.