Chest X-Ray Report Generation. Overview : | by Abhishek Devata

7 min read

·

Sep 27, 2020

Photo by Jonathan Borba on Unsplash

Chest radiography is the most common imaging examination globally, critical for screening, diagnosis, and management of many life-threatening diseases. Automated chest radiograph interpretation at the level of practicing radiologists could provide substantial benefit in many medical settings, from improved workflow prioritization and clinical decision support to large-scale screening and global population health initiatives.

Open-i has a collection of chest X-Ray Images from the Indiana University hospital network. Data contains two files, one for Images and the other one for the XML report of radiography. For each report, there could be multiple images. The image has mainly two views Frontal and Lateral view. XML report contains findings, indication, comparisons, and Impressions. There are 3955 reports and 7470 images in total.

Problem Statement: Our task at hand is to generate an impression given an image of radiography.

Data Overview:

• There are two sets of files, one contains an image of patients and the other contains a Report of that particular patient.
• The report is in XML format
• The report contains image_id, the caption of an image, indication of patient, findings, and impression

Find dataset:

Indiana University – Chest X-Rays (PNG Images)

@article{, title= {Indiana University – Chest X-Rays (PNG Images)}, keywords= {radiology, chest x-ray}, author=…

academictorrents.com

Indiana University – Chest X-Rays (XML Reports)

1000 radiology reports for the chest x-ray images from the Indiana University hospital network.

academictorrents.com

The BLEU score is a string-matching algorithm that provides basic quality metrics for Machine-Translation researchers and developers.

Sample report:

• Parsing XML to a data frame for better analysis and training of the model.

Sample XML format :

Extracting Abstract and parent Image nodes.

Dataframe contains sevencolumns: image_id,caption,comparison,indication,findings,Impression,height and width of image.

Missing Values and Imputation:

Imputing missing values of each column as “No column_name”.

Columns like caption, comparison, indication, findings, impression contain arbitrary texts, which has to be removed.

The number of images per patient :

There are 3227 patients who have two images. Both Frontal and Lateral view of Chest.

Height distribution of images:

The height of images is not constant and need to reshape.

But the width of the images is constant.

Sentence length of Impression data:

Most of the sentences have a length of four.

Word Cloud of Impression data:

“acute cardiopulmonary” occurred several times than any other vocab.

Note: There are a few images with no information on it. Either the image has high brightness where the chest can’t be seen or a totally black image.

As Each patient has different set of images, we first construct these images into structured format before diving into the model.

• patients with four images : creating four data points as shown below
  1. frontal1,lateral1 >> Impression
  2. frontal1,lateral2 >> Impression
  3. frontal2,lateral1 >> Impression
  4. frontal2,lateral2 >> Impression
• patients with three images : creating two data points as shown below
  1. frontal1,lateral1 >> Impression
  2. frontal1,lateral2 >> Impression
• patients with one image : creating one data point as shown below
  1. frontal1,frontal1 >> Impression (or)
  lateral1,lateral1 >> Impression

A Data frame is created. Let’s build a model!

Note: Split data frame into train and validation sets.

Present Image:

Generated Images:

The impression is of text data which needs to be converted to a numerical vector before feeding into our model.

The number of vocabs in impression train data is 1291. The maximum sentence length of the impression is 125.

Each impression is now converted to the vector of size (1,125).

Extract Features from Images:

EffiecientNet model which is trained on the Imagenet dataset can be used as a feature extractor.

why EfficientNet?

please refer this blog for more information :

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Since AlexNet won the 2012 ImageNet competition, CNNs (short for Convolutional Neural Networks) have become the de…

medium.com

TensorFlow Version 2.3.0 contains EfficientNetB7 model :

Now Each image is passed into this model for feature extraction, This returns feature vector of size [1,2560].

This feature vector is reshaped to [32,80]. So that we can get Attention weights of length 32.

Both Impression and Images are converted to numerical vector.

Create Dataset:

tf.data is used to fetch the data efficiently, shuffle the data, and create batches.

Both Images and impressions are now converted to the train dataset and validation dataset.

Model Architecture

Encoder :

In the encoder, two images are concatenated and applied a dense layer on top of this.

The encoder returns the output of size [batch_size,32,embedding_dim]

BahdanauAttention:

Normally we feed the last hidden state vector of the encoder to the decoder, but it may not have whole information, to get better information from the encoder, we use Bahadanau Attention, to understand more about this please refer :

Neural Machine Translation using Bahdanau Attention Mechanism

Table of Contents

medium.com

Using features from the encoder and hidden state of the decoder we get the context vector.

Decoder:

The context vector is then concatenated with the decoder input which is a numerical vector of an impression obtained after embedding.

This Merged vector is passed to LSTM which is a special type of RNN, which learns long-term dependencies.

Note: Pretrained Glove vectors are used for word embedding.

Define Custom Loss:

Training Subclass model:

TEACHER FORCING TECHNIQUE:

For each training step :

Initialize the hidden state with zeroes and <start> token as the first timestep input to the decoder.

Get the feature vector of two images by calling encoder.

For each time step of decoder:(Maximum length of sentences which is 125)

Pass decoder input, hidden state, and feature vector to the decoder. Update hidden state returned from decoder, append the prediction to decoder input.

Visualizing Loss:

orange-train loss | blue -Validation loss

Train loss is converging faster but validation loss is getting saturated after a few epochs.

For the Evaluation of this image captioning we use the BLEU score as a metric as mentioned above.

BLEU: Bilingual Evaluation Understudy

A Gentle Introduction to Calculating the BLEU Score for Text in Python – Machine Learning Mastery

BLEU, or the Bilingual Evaluation Understudy, is a score for comparing a candidate translation of text to one or more…

machinelearningmastery.com

In each timestep of decoder, we get vocab size output with probabilities.

Out of these probabilities, we can select which word has a high probability of occurring. For picking this top words, we have two techniques in common.

1. Greedy search

A simple approximation is to use a greedy search that selects the most likely word at each step in the output sequence.

2. Beam search

Instead of greedily choosing the most likely next step as the sequence is constructed, the beam search expands all possible next steps and keeps the k most likely, where k is a user-specified parameter and controls the number of beams or parallel searches through the sequence of probabilities.

‘k’ is known as beamwidth.

For more information on beam search:

#https://www.youtube.com/watch?v=RLWuzLLSIgw

Given two chest x_ray images of patients, our model returns the impression.

BLEU score obtained on the whole validation dataset is 0.39952.

Images with less bleu score :

Duplication of either frontal or lateral view of the image, high brightness, and more darkness leads to less performance of the model.

Images with high bleu score:

Better quality of frontal and lateral views, No duplication of images, and balance of brightness and black pixels lead to a good performance of our model.

The model performs very well on small sentences which occur significant times.

When Impressions which have rarer words and have low image quality or any noise in images, our model performs low.

To get a better performance larger dataset is required.

Image Augmentation technique improved performance but not significant improvement in bleu score.

1. Applying Bert model on word embeddings.
2. Creating Web API which takes images of the patient and returns the impression.
3. Working on larger data released by Stanford University(Chexpert competition).

1. https://www.appliedaicourse.com/
2. https://www.tensorflow.org/tutorials/text/image_captioning?hl=en
3. https://arxiv.org/pdf/1905.11946.pdf
4. https://arxiv.org/pdf/1911.06475.pdf
5. https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/
6. http://cs231n.stanford.edu/reports/2016/pdfs/362_Report.pdf

Github profile:

devathaabhishek/Chest-X-Ray-Report-Generation

Image Captioning. Contribute to devathaabhishek/Chest-X-Ray-Report-Generation development by creating an account on…

github.com

LinkedIn profile:

Abhishek Devata – Assistant Manager – Vedanta Limited – Aluminium Business | LinkedIn

View Abhishek Devata’s profile on LinkedIn, the world’s largest professional community.

Abhishek has 3 jobs listed on…

www.linkedin.com

Sample Diagnostic Radiology Reports | NationalRad

View sample reports from NationalRad’s musculoskeletal, neuro and body radiologists and nuclear medicine physicians.
 

MRI Abdomen With Contrast

History pancreatic cancer. Status post aortic chemotherapy and Whipple procedure on DATE. Chronic low back pain. Abdominal pain. Follow-up examination.

Download Report

MRI Arthrogram Right Hip With Cartigram Study

Assess right hip and groin pain for one year. This is associated with locking and the pain is sharp in character. History of playing soccer.

Download Report

MRI Brain With & Without Contrast

Right arm weakness. Difficulty expressing thoughts in writing beginning about 4-5 months ago.

Download Report

MRI Cervical Spine

Left neck and shoulder pain for 2 months.

Download Report

MRI Chest Without Contrast

69-year-old female with a history of smoking, asthma and bronchitis now with productive cough intermittently for several months.

Download Report

MRI Left Foot

Left medial foot and ankle pain and swelling. Plantar metatarsal pain for 5 weeks. No known trauma.

Download Report

MRI Left Hip

36-year-old male, assess chronic left hip pain.

Download Report

MRI Left Knee

Evaluate medial left knee pain. Injured playing football 3 weeks ago. Positive Lachman test. Evaluate ACL tear. Evaluate for meniscal tear.

Download Report

MRI Left Knee

Left medial knee pain and swelling for 2 weeks, injured during football, assess for medial meniscal tear, initial visit.

Download Report

MRI Left Shoulder

Work related injury. Assess for traumatic tear left rotator cuff with superior shoulder pain and weakness.

Download Report

MRI Lumbar Spine

Low back pain with bilateral lower extremity radiculopathy which is achy and intermittent.

Download Report

MRI Neck With & Without Contrast

Right parotid mass.

Download Report

MRI Pituitary With & Without Contrast

Macroadenoma with no priors available for comparison.

Download Report

MRI Prostate With & Without Contrast

Elevated PSA. Enlarged prostate. History of prostate cancer.

Download Report

MRI Right Elbow

Evaluate persistent right elbow pain. Status post fall.

Download Report

MRI Right Hand

Fall from a standing height onto an outstretched arm, assess for new versus old fracture, date of injury DATE, ulnar sided hand pain.

Download Report

MRI Right Wrist

Assess right wrist pain. Pain is diffuse. Status post fracture of the distal radius status post fall.

Download Report

MRI Thoracic Spine Without Contrast

Dorsal right medial upper back pain for 10 weeks. Evaluate for degenerative disc disease.

Download Report

US Guided Left Knee Injection

Left knee pain. Semimembranosus bursitis.

Download Report

Scheme for the description of radiographs and fluoroscopy of the organs of the chest cavity. — 24Radiology.ru

1) Name and age of the patient.
2) General assessment of radiographs
Method.
– fluoroscopy
– radiography (survey, sighting radiograph).
– superexposed radiograph.
– tomogram.
— bronchogram.
– CT scan.
– angiogram.
Indication of the examined organs (thoracic organs).
Research projection (direct, lateral, oblique, lateroposition).
Image quality (contrast, sharpness, hardness of rays, correct stacking).
3) Study of the lungs.
Determining the shape of the chest (normal, bell-shaped, barrel-shaped).
Assessment of lung volume (not changed, lung or part of it is enlarged, reduced).
Establishment of the state of the lung fields (transparent, darkening, enlightenment).
Analysis of the lung pattern (not changed, strengthened, weakened, deformed).
Analysis of the roots of the lungs (structurality, width, location, enlarged lymph nodes, vessel diameter).
Functional state during fluoroscopy (respiratory movements of the ribs, diaphragm, changes in the lung pattern during breathing)

Identification and description of pathological syndromes.
1) Shadow picture
Darkening.
Enlightenment.
2) Localization
By shares
By segments.
3) Dimensions in centimeters (at least two dimensions are indicated).
4) Shape
Round.
Oval.
Invalid.
Triangular.
5) Contours
Smooth or uneven
Clear or indistinct
6) Intensity
Weak
Medium
High
Lime density
Metallic density
7) Shadow structure.
Homogeneous
Inhomogeneous
8) Functional signs on fluoroscopy
9) Change in the shape of a round shadow during breathing – with liquid formations (cysts).
10) Shadow pulsation in vascular formations (aneurysms, angiomas), etc.
11) Correlation of pathological changes with surrounding tissues:
Enhancement of lung pattern in surrounding tissues
Enlightenment rim around a round shadow by pushing adjacent tissues
Pushing or pulling apart bronchi or vessels, etc.
Dropout centers.

4) Examination of the mediastinal organs
1) Location
Not displaced
Displaced (towards pathological changes in the lungs or in the opposite direction).
2) Dimensions:
Not enlarged
Dilated due to the left ventricle or other parts of the heart;
Widened to the right or left in the upper, middle or lower sections.
3) Configuration
Not changed
If changed, it may be due to volumetric formations of the heart, blood vessels, lymph nodes, etc.
4) Contours.
Smooth
Irregular
5) Functional state during fluoroscopy
Heart rate
Jerky displacement of the mediastinum during expiration towards atelectasis, etc.

5) Examination of the walls of the chest cavity.
1) Condition of the sinuses of the pleura
Free
Contains fluid
Has pleurodiaphragmatic adhesions.
2) Condition of soft tissues
Not changed
Enlarged
There is subcutaneous emphysema
Foreign bodies, etc.
3) Condition of the skeleton of the chest and shoulder girdle collapsed fractures.
4) Diaphragm condition
Ordinary location
Proximal displacement by one intercostal space, etc.
Domes have smooth contours or are deformed by pleurodiaphragmatic adhesions.
Diaphragm mobility under fluoroscopy.

CONCLUSION
RECOMMENDATIONS on the use of additional methods.
DESCRIPTION of additional techniques and methods confirmation or clarification of the previously described picture, description of newly identified pathological signs.
FINAL REPORT (e.g. pneumothorax, parenchymal pneumonia, central exobronchial carcinoma without metastasis, peripheral carcinoma, echinococcus in non-opened phase, etc.)

Chest X-ray – Systems Approach

Introduction

A systematic approach to the analysis of chest x-rays is used to ensure that important structures are not missed, and a flexible approach is needed for different clinical situations.

Although there is no single agreed upon order of image analysis, you can find many examples of chest x-ray descriptions.

Below is a short example.

Anatomical structures checklist

1. Trachea and major bronchi

2. Lung roots

3. Lung fields

4. Pleura

5. Lung lobes/interlobar fissures

6. Costophrenic sinuses

7. Diaphragm ragma

8. Heart

9. Mediastinum

10. Soft Tissue

11. Skeletal Framework

This guide will help you develop your own analysis system, from patient data, image data, and image quality. Next, you will study where and what pathological changes can be described. The manual also discusses an overview of blind spots where it is easy to miss a pathological process. Your results will be better if you are able to analyze and relate clinical data to radiological findings.

Patient and image data

Patient identifiers and date

Patient identification must be performed before X-ray image interpretation. The date of the examination, as well as, necessarily, the time, must be noted, as the patient may have more than one radiograph on the same day.

Image Projection

Note which view, AP or AP, the image was taken; standing, lying or sitting; stationary or mobile device.

Image annotations

Useful information is often displayed on an image. If the projection is not marked, it is likely that the image was taken in a standard anterior-posterior (PA) projection. If there are side markers, pay attention to the correct position of their position.

Image quality

Image quality should always be assessed because clinical questions cannot be answered if the image quality is inadequate.

Pay attention to the rotation of the chest, the depth of inspiration and the adequacy of the penetrating power of the X-ray radiation.

Image annotations

Artifacts

When you describe a chest x-ray, it is good practice to comment on the presence of any artifact.
An example is shown below.

Central catheter position ?

hover over image

A large number of radiographs are taken to assess the position of medical equipment such as a nasogastric tube or central catheter. If you are evaluating a chest x-ray for this purpose, remember to evaluate the entire image systemically.

Obvious pathology

It is advisable to start the analysis with the most pronounced pathology. However, once done, it is important to continue analyzing the rest of the image according to the checklist. Remember that a more prominent pathology may not be of clinical significance.

For example, don’t make the mistake of devoting most of your time to rigorously following a systems approach while ignoring obvious pathology.

The rule can be summarized as – don’t ignore the “elephant” in the picture – describe its long trunk, its large ears, tusks and rough, gray skin and you will be more likely to diagnose the “animal” you are dealing with, but then you must continue analysis using a systematic approach to watch the rest of the image.

Describing pathology

The art of radiology, not simply in stating and describing pathological features, but knowing how to relate the meaning of these pathological features and knowing which ones can be omitted. First, describing radiographic features can be difficult, and many medical students want clear terminological rules. However, in reality there are no clear rules. The main difficulties begin when describing the pathology of the lung parenchyma. What one radiologist describes as “darkening” may be referred to by others as “decrease in pneumatization” or “infiltration.” In fact, all of these terms are acceptable.

The description of the pathology on a chest x-ray can be compared with the description of a skin rash in a dermatological patient. Attention should be directed to such features as quantity, localization, size, shape, density and structure.

Special Findings

There are many specific X-ray findings that can guide you to the correct diagnosis. For example, occlusion of the costophrenic sinus, forming obtuse angles with the chest wall, should make you think of a pleural effusion. Obvious consolidations (infiltrations) with a sign of an air bronchogram should first of all suggest an infectious process. These signs must be indicated in the descriptive picture.

If you see one of these clear signs, try not to jump to conclusions. Continue the systematic description of the changes and perhaps you will see that the blunting of the angle of the costophrenic sinus is caused by emphysematous enlargement of the lung fields, and the consolidation of the lung tissue is combined with the destruction of the rib, making cancer a more likely diagnosis? than pneumonia.

Location of changes

In addition to determining the side of the identified changes, it is necessary to evaluate the localization in the anterior-posterior projection. A lateral view helps to localize changes in 3D space, but this is also possible from a direct view, with knowledge of x-ray anatomy and understanding of the contours of the shadows.

Contour sign

Contour sign is an erroneous name, it is more correct to call it a “lost contour” sign. Normal adjacent anatomical structures of varying density form clear “silhouettes” or contours. Violation of normal boundaries can help determine the position of the pathological process.

For example, the heart (soft tissue density, white) borders on the lung tissue (air density, dark color). A clear contour, or “silhouette” is formed at the junction of two fabrics of different density. The loss of a clear contour of the right heart (formed by the right atrium) suggests localization of the disease in the right middle lobe, which is adjacent to the right atrium. The loss of the density difference of the left heart contour indicates the pathology of the lingular regions (the part of the upper lobe of the left lung that surrounds the left ventricle).

Changes simulating contour sign

move cursor over image

Changes simulating contour sign

move cursor over image

05

After a systematic full inspection chest, it is worth re-checking areas that may hide an important pathology.

It is always worth double-checking that there is no pneumothorax or pneumoperitoneum. And indicating their absence in the descriptive part is a good practice.

Pneumothorax is easily seen at the apex on an anterior-posterior radiograph. Pneumoperitoneum (free gas under the diaphragm), only visible on x-ray taken while standing

Other areas to look at include soft tissue, bone, posterior mediastinum, and image margins.

Inspection areas – Apexes

move cursor over image

Inspection areas – Bones

move cursor over image

Area Inspection – Heart Shadow

Move cursor over image

Area Inspection – Aperture

Place cursor over image

Area Inspection – Edges of Image

Move cursor over image

Clinical Tasks 901 05

At first, most students think that radiography gives accurate answers without comparison with clinical data. Sometimes this may be the case, but ideally radiography should always be interpreted in full correlation with the clinical findings. Most radiological conclusions can only be given in the light of clinical data. Thus, you should always be provided with specific clinical data when requesting an x-ray.

Often the results will confirm the preliminary diagnosis, and the absence of changes will improve the prognosis, since an experienced clinician will often know the diagnosis before the X-ray examination, and use it to clarify the extent and localization of the pathological process.

Therefore, results should only be interpreted in relation to clinical data. Remember, the radiologist does not treat the patient. Occasionally there will be incidental findings that require careful consideration, especially if they can be interpreted in two ways or if they do not correspond to clinical data.

No clinical data provided

hover over image

Clinical data provided

hover over image

Conclusion missing important changes.

Patient data and image quality must always be evaluated.