Classification of mastoid air cells by CT scan images using deep learning method

Abstract

Purpose

Mastoid abnormalities show different types of ear illnesses, however inadequacy of experts and low accuracy of diagnostic demand a new approach to detect these abnormalities and reduce human mistakes. The manual analysis of mastoid CT scans is time-consuming and labor-intensive. In this paper the first and robust deep learning-based approaches is introduced to diagnose mastoid abnormalities using a large database of CT images obtained in the clinical center with remarkable accuracy.

Methods

In this paper, mastoid abnormalities are classified using the Xception based Convolutional Neural Network (CNN) model, with optimizer Adamax into five categories (Complete pneumatized, Opacification in pneumatization, Partial pneumatization, Opacification in partial pneumatization, None pneumatized). For this reason, a total of 24,800 slides of 152 patients were selected that include the mastoid from most upper to the lowest part of the middle ear cavity to complete the construction of the proposed deep neural network model.

Results

The proposed model had the best accuracy of 87.80% (based on grader 1) and 88.44% (based on grader 2) on the 20th epoch and 87.70% (based on grader 1) and 87.56% (based on grader 2) on average and also significantly faster than other types of implemented architectures in terms of the computer running time (in seconds). The 99% confidence interval of the average accuracy was 0.012 which means that the true accuracy is 87.80% and 87.56% ± 1.2% that indicates the power of the model.

Conclusions

The manual analysis of ear cavity CT scans is often time-consuming and prone to errors due to various inter- or intra operator variability studies. The proposed method can be used to automatically analyze the middle ear cavity to classify mastoid abnormalities, which is markedly faster than most types of models with the highest accuracy.

Introduction

The hearing system is responsible for collecting, conducting, and amplifying sounds and converting them into electrical energy, and transmitting to specific centers in the brain [1]. The auditory organ consists of three parts: the outer ear, the middle ear, and the inner ear and the temporal bone is comprised of four parts including the squamous, mastoid, petrous, and tympanic parts. The petrous part surrounds the inner ear and the squamous part forms the mastoid appendage in the middle ear [2]. The mastoid appendage is formed by the attachment of the squamous part of the temporal bone [2]. Cells that form in different parts of the temporal bone all originate in the middle ear [3]. In general, the mastoid air cells are either pneumatized or none pneumatized. in the case of none pneumatization, it could have either opacification or sclerosis [4]. Mastoid air cells illustrate a comprehensive system of interlinking air-filled cavities surrounded by walls of the mastoid antrum and middle ear [5]. The mastoid part of the temporal bone has a significant role in terms of absorbance and scattering of kinetic energy through lateral trauma to the temporal bone, decreasing the occurrence of the fraction in the settling of direct trauma [6]. The concept of the grade of pneumatization of the temporal bone is so momentous in terms of surgical contemplations and pathophysiological care of numerous temporal bone illnesses [7]. Some range of the inflammatory, neoplastic, vascular, fibro-osseous, and traumatic changes have been illustrated by opacification at the middle ear and mastoid which help specialists to diagnose ear diseases [8]. One of the most prevalent complications of acute otitis media after tympanic membrane perforation is otomastoiditis which has risen over recent decades [9, 10]. Mastoiditis is an inflammation of the mastoid bone that is caused by inflammation of the middle ear and acute otitis due to the connection between the mastoid cells and the middle ear [11]. Because the middle ear is connected to the Eustachian tube on one side and to the mastoid cells by the aditus and anter on the other, whenever an infection reaches the middle ear and the tympanic membrane, this infection and inflammation may spread to the mastoid cells [12]. Therefore, the presence of mastoid pneumatic cells and the conjunction between the cells and the middle ear and the Eustachian tube is one of the proper ways that the infection and inflammation spread not only to the mastoid appendage but also to different parts of the temporal mastoid bone [12, 13]. In addition, the patients who were given chemotherapy or underwent organ transplantation surgery are mostly immunocompromised which caused the enhancement of the of otomastoiditis [9, 14]. If treatment for acute or chronic ear infection fails, the infection can spread to other areas of the head and neck. Even mastoid infections of the ear can be life-threatening disorders such as meningitis, subdural infections, brain abscesses, petrosal infections located between the inner and middle ears, temporal bone infections, and paralysis of the face [8]. The concept of the degree of pneumatization of the temporal bone is very important in terms of surgery and pathophysiological care of many temporal bone diseases [8]. A wide range of inflammatory, neoplastic, vascular, fibrous, and traumatic changes with opacification in the middle ear and mastoid have been shown to assist specialists in diagnosing ear disease [8]. Sclerosis and opacification of the middle ear and mastoid air cells are key CT features of various ear diseases including acute otomastoiditis, necrotizing otitis externa, chronic otomastoiditis and cholesteatoma [8]. There are some other key CT features of mentioned ear diseases, for instance: CT features of acute otomastoiditis are middle ear and mastoid opacification with liquid levels and probably bone demolition [8]. Vast soft-tissue inflammation with middle ear and mastoid opacification and skull base osteomyelitis leading to bony demolition is the key CT features of necrotizing otitis externa [8]. For chronic otomastoiditis there are middle ear and mastoid opacification with mastoid trabeculae inspissating, sclerosis, and cell sabotage, probably ossicular chain abrasion [8]. Also, cholesteatoma causes middle ear and/or mastoid soft-tissue opacification with ossicular, caul tympani, or scutum abrasion, probably labyrinthine fistulas [8]. Mastoid process involved in some other diseases for instance, covid19 [15]. There is a poor correlation between mastoiditis on CT imaging with the clinical diagnosis which emphasizes the importance of CT images [13]. Clinical intervention for opacification in the mastoid process is very crucial after a diagnosis [16]. In some cases, we witness the discrepancies between CT reports and surgical findings regarding middle ear opacification which is mostly caused by misdetection of radiologists in imaging [17].

To find the solution for usual issues in clinical actions such as eruptively radiologists' workloads and innate challenges of explicating of medical images, the usage of deep learning has been explored by many studies in order to find the best model in terms of analyzing them.

There are so many achievements by using deep learning methods in diverse functions of computer vision for instance image classifications, object recognition, localization, and segmentation in natural images [18]. Convolutional Neural Networks (CNN) are used in medical images for detecting and evaluating illnesses [18]. Diagnosis of unique traits of medical images is customarily carried out by experts for detecting diseases. Neural networks or deep learning in various medical fields have shown great success. Automated diagnosis by artificial intelligence has recently been the focus of specialist physicians due to the significant decrement in error and high speed of diagnosis [18]. In some fields, the results were excellent compared to those of specialists [19, 20]. The main problem with generating CNN is its need for large number of training data which is not feasible in most cases [21]. Alternatively, pre-trained public CNN models for natural images could be utilized and fine-tuned to a particular usage which is named Transfer learning [21]. Transfer learning is the concept of dominating the cloistered learning template and using knowledge obtained for one task and solving related ones. In transfer learning, most of the network layers are transferred to the new model. But the difference is in the Fully-connected layers which are changed based on the new set of classes [21]. Previous studies have shown that the use of transfer learning in medical imaging has better results than building CNN from scratch [19, 21].

Some recent studies were performed for automatic diagnosis of ear diseases by using endoscopic images. A study was conducted in for diagnosing otitis media and they got 81.58% accuracy via decision and 86.84% via neural networks [22]. The other study performed by Cha et al. [23] in which the otoendoscopic images were used via public convolution-based deep neural networks to categorize common ear illnesses (Normal, Attic retraction, Tympanic perforation, Otitis externa ± myringitis, Tumor) [23]. Some ear disorders only can be diagnosed by computed tomography scans. The most relevant study conducted regarding mastoid abnormalities was introduced in radiographic images [24]. Since the mastoiditis incidence is mostly occurred in under two years old children who are very susceptible to radiation exposure [24]. With using multiple views, the area under the curve of their proposed algorithm was 0.971, 0.978, and 0.965 for the gold standard, temporal, and geographic external test sets, respectively [24]. And also the sensitivity and specificity of their method were 96.4% and 74.5% respectively [24]. However, the most detailed abnormalities in tiny parts of the middle ear such as petrosal and sigmoid sinus can't be detected in the radiographic images, and the use of radiography has obsoleted [25]. The most general technique utilized to elicit the details of images of the ear cavity is computed tomography scan (CT scan) [25]. The processing of mastoid air cells is only partially represented on a CT scan by Olivier Cros [26]. In [27] a two-class (normal and abnormal) classifier based on convolutional neural networks deep learning model was introduced. The proposed model has an accuracy of 98.10%, however, this study classifies only normal and abnormal mastoids.

In this paper the first and robust deep learning-based approaches is introduced to diagnose mastoid abnormalities in five groups (1. Complete pneumatization, 2. Opacification in pneumatization, 3. Partial pneumatization, 4. Opacification in partial pneumatization, 5. None pneumatized). The proposed method can reduce the analysis of the large and complex CT images which may be a tedious and complex task for clinicians.

This paper is organized as follows. In Section “Materials and methods”, we explain the used dataset and also our proposed deep learning-based method. The results and performance evaluation are presented in the “Results” section. Finally, the paper is concluded in the “Conclusion and discussion” Sections.

Material and methods

Materials

In this paper, 24,800 B-Scan images from 152 temporal CT scans (512px by 512px) in DICOM format of patients(84 female and 68 men) who have been referred to the Imam Reza hospital Center in Tabriz, Iran from the year 2017 to 2020 at the request of an ENT specialist have been obtained. The various types of abnormal mastoids were shown in Fig. 1. The mastoid air cells were classified by an ENT specialist and a radiologist physician into five classes.

• (1) Complete pneumatization: Normal pneumatization and there is no Sclerosis or opacification.

• (2) None pneumatized: Completely sclerotic, there is no air or opacification.

• (3) Opacification in Complete pneumatization: There is no sclerosis, only opacification in the mastoid.

• (4) Opacification in partial pneumatization: opacification in the partially pneumatized mastoid.

• (5) Partial pneumatization: There is no opacification but the mastoid is partially pneumatized.

Fig. 1

Various types of abnormal mastoids were presented. a Normal mastoid pneumatization on both right and left ear (class 1), b None pneumatization on both sides as yellow arrows indicate that there are no air cells (class 2), c Opacification in Complete pneumatization on the right mastoid which is shown by red arrows, left mastoid is normal, complete pneumatized, and there is no sclerosis on both sides (class 3), d Yellow arrows present none pneumatized parts of the mastoid on both side and the rest parts of the mastoid have opacified which is pointed by red arrows (class 4), e The right mastoid is partially pneumatized. The yellow arrow shows the sclerotic part of the right mastoid, while the left side is normal and completely pneumatized (class 5)

It should be noted that the age of all patients are more than 10 years and all images are scanned under the same conditions, with a specific device. The interval for each scan was 0.5 mm and depending on the patient's gender, age, and skull size, the number of selected images was between 60 and 90 scans per ear.

To segment the right and left mastoid with predefined coordination which is covered all parts of the mastoid on all scans, we initially pre-processed images and the region of interest (ROI) of images have been extracted. For these regions, we use Otsu's method (which chooses the threshold to minimize the intraclass variance of the black and white pixels) to compute a global threshold (level) that can be used to convert an intensity image to a binary image. Then opening and hole filling morphological operators [28] are used to generate a binary mask. The left and right binary masks to segment the mastoid region are shown in Fig. 2.

Fig. 2

A binary segmentation mask to extract mastoid region. a Original image, b Global thresholded image, c Use of the opening operation, d shows the result of applying the hole filling operation to the image (c). e, f Left and Right binary mask to segment L&R mastoid region

The frequency distribution for each category based on ENT and radiologist diagnosis, is illustrated in Fig. 3.

Fig. 3

Frequency distribution for each category based on ENT and radiologist diagnosis. a Statistics of the categories on a percentage based on ENT diagnosis, b based on radiologist diagnosis c Number of scans on each category based on ENT (Orange) and radiologist diagnosis (Blue)

Figure 3 shows the percentage of intergrader agreement and the proposed model has been trained and evaluated based on both Graders.

Methods

Proposed methodology

In this paper sixteen common CNN networks (Xception [29], VGG16 [30], VGG19 [30], ResNet50 [31], ResNet101 [31], ResNet152 [31], ResNet50V2 [32], ResNet101V2 [32], ResNet152V2 [32], InceptionV3 [33], InceptionResNetV2 [34], MobileNet [35], MobileNetV2 [36], DenseNet121 [37], DenseNet169 [37], and DenseNet201 [37]) with seven common optimizers (SGD [38], RMSprop [39], Adagrad [40], Adadelta [41], Adam [42], Adamax [42], and Nadam [43]) based on public CNN models which are pre-trained with the ImageNet database [44], are evaluated and are pre-trained with ImageNet database which learned with categorizing 1000 natural objects and is used for training to classify normal mastoid and its abnormalities. All 112 types of models have been trained and tested with a quarter of the dataset which was extracted from the entire dataset with the same ratio as shown in Fig. 4 (to reduce elapsed time) for five times on stage one in order to find a suitable network/optimizer with the highest accuracy. The batch size was 8 and the number of epochs for each model was selected to be 20. Eighty percent of the dataset is devoted to training and the others are considered for validation of data. In this study, the Keras library in python over a Graphics Processing Unit (GPU) with dual RTX 2080 was used. All the raw images were transferred to grayscale and normalized between 0 and 1 with Keras Image Data Generator library. At this library, the shear range and zoom range was 0.2, and also the horizontal flip was true for training data, but in order to evaluate the accuracy in original data, there was not any data generating for data validation. Figure 5 illustrates our proposed classification method based on the transfer learning method.

Fig. 4

Distribution of each category in the quarter of dataset

Fig. 5

A simple diagram of our proposed transfer learning-based classification method

System model

Xception architecture that involves depthwise Separable Convolutions is used and transferred into a new model. The last layer of the model was altered with a new Fully-connected layer which has five nodes based on five classes. This model is chosen due to outperform than other networks [29] and also the great results in comparison to other used architectures in terms of accuracy rate. The activation of the last layer for this model was Softmax and the optimizer used was Adamax (AdaMax has higher performance in comparison to other optimizers which are mentioned above). The schematic architecture of Xception model is shown in Fig. 6.

Fig. 6

The schematic architecture of Xception model

Experimentation and results

Experimentation

Performance of the proposed approach is assessed by comparing the classification results with ENT and radiologist diagnosis as ground-truth labeled images. For this purpose, two performance measurements, namely accuracy and confidence interval were calculated. One of the most common metrics in multi-class classification is accuracy. It is straightly calculated from the confusion matrix as follow [45]:

$$\mathrm{Accuracy }= \frac{TP+TN}{TP+TN+FP+FN}$$

(1)

where (TP: true positive, FP: false positive, FN: false negative, TN: true negative)

And the Confidence Interval is the probability that a parameter will fall between a pair of values around the mean [46]. It estimates the rating of uncertainty or certainty in a sampling method and is defined as follows:

$$\mathrm{Confidence interval}=\overline{X }\pm Z\frac{s}{\sqrt{n} }$$

(2)

where $:\overline{X }$ is the mean, Z is chosen from the Table 1, s is the standard deviation and n is the number of observations.

Table 1 Critical Z value in calculation of confidence interval

Results

After running the model for 20 epochs on the whole dataset, the results of our proposed method based on both graders (grader 1 was ENT specialist, and grader2 was Radiologist) in terms of accuracy and the average elapsed time has illustrated in Table 2. The confidence of the accuracy is also has been shown in this table which indicates the power of the model.

Table 2 The results of our proposed method include 20% of the whole dataset

The stability of this model with both graders is depicted in Fig. 7, which indicates that increasing the number of epochs does not rise the accuracy and the model reached the best performance of itself in terms of the number of epochs.

Fig. 7

The accuracy plot of Xception model with optimizer Adamax on the whole dataset. a Based on grader A (ENT). b Based on grader B (Radiologist)

Discussion and conclusion

Discussion

As we mentioned, 16 CNN networks with 7 optimizers which makes 112 types of different medels ran in this paper. All proposed models trained for 20 epochs on the quarter of dataset for five times each. The average accuracy for validation data indicated in Table 3.

Table 3 The average accuracy of the last epoch for each network with different optimizers after 20 epoch (%)

Based on Table 3, seventeen appropriate network/optimizer with greater accuracy were trained and tested on the whole dataset. Table 4 shows the results of these methods which have been sorted from high to low accuracy of the first stage.

Table 4 Results of selected seventeen types of models/optimizers that have been selected based on average accuracy

As shown in Table 4, although the InceptionV3 model, trained markedly faster than Xception using Adamax, the Xception model with optimizer Adamax has the highest accuracy (average accuracy of 20 epochs) 87.70% on the whole dataset and are selected as a suitable deep learning-based model for classification of mastoid abnormalities. This accuracy is 86.33% for model InceptionV3 with the same optimizer. Xception stands for “Extreme Inception” that takes the axiom of inception shows better performance than inception as expected. It emphasizes the importance of depthwise separable convolutions which Xception model is constructed based on it. In terms of optimizers, the AdaMax is outperformed almost with most of the other architectures. Therefore, the model Xception with AdaMAx optimizer have chosen as the best method which is used as the main and the best method. the results of this method have been shown in Table 2.

Performance comparison

In this paper the first and robust deep learning-based approaches is introduced to diagnose mastoid abnormalities from CT images in five groups. While there are some studies have focused on mastoid images processing [22,23,24], none of them deal with CT images.

Conclusion

Manual diagnose of mastoid abnormalities is time-consuming and could be labor-intensive and inaccurate diagnose could lead to inessential surgeries. This study presents the first machine-learning-based model with a high rate of accuracy to diagnose mastoid abnormalities from CT images in five groups.

In conclusion, Adamax shows better results in comparison to other optimizers, and in terms of selecting models, Xception is the best choice. The opacification of the mastoid air cells can occur in some circumstances and can include aspects of neoplastic, vascular, inflammatory, fibro-osseous, and traumatic changes. It would be better for ENT surgeons to have background knowledge of the mass of the opacification in the mastoid. After detection of mastoid abnormalities, these regions could be segmented by machine leaning approaches.

Future works

In this study, mastoid air cells were classified into 5 classes using CT scan images. Since mastoid air cell abnormalities represent a wide range of various middle ear disorders, for further studies, mastoid CT scans can be used to detect different types of ear diseases such as Acute otomastoiditis, Necrotizing otitis externa, Chronic otomastoiditis, Tympanosclerosis, Cholesterol granuloma, and Cholesteatoma. As a result, in addition to the classifying of abnormal classes, the type of disease is also could be diagnosed. Also, by increasing the number of data using data enhancement methods, it is possible to improve the performance of networks.