AI sees beyond humans: automated diagnosis of myopia based on peripheral refraction map using interpretable deep learning

Abstract

The question of whether artificial intelligence (AI) can surpass human capabilities is crucial in the application of AI in clinical medicine. To explore this, an interpretable deep learning (DL) model was developed to assess myopia status using retinal refraction maps obtained with a novel peripheral refractor. The DL model demonstrated promising performance, achieving an AUC of 0.9074 (95% CI 0.83–0.97), an accuracy of 0.8140 (95% CI 0.70–0.93), a sensitivity of 0.7500 (95% CI 0.51–0.90), and a specificity of 0.8519 (95% CI 0.68–0.94). Grad-CAM analysis provided interpretable visualization of the attention of DL model and revealed that the DL model utilized information from the central retina, similar to human readers. Additionally, the model considered information from vertical regions across the central retina, which human readers had overlooked. This finding suggests that AI can indeed surpass human capabilities, bolstering our confidence in the use of AI in clinical practice, especially in new scenarios where prior human knowledge is limited.

Highlights

• • Developing a deep learning algorithm to assess myopia status using retinal refraction maps obtained from a novel peripheral refractor.

• • The deep learning algorithm accurately determined myopia status with high accuracy.

• • Interpretable Grad-CAM analysis indicated that the deep learning algorithm utilized information from the central retina as well as vertical regions.

Introduction

Recent years have witnessed significant advances in artificial intelligence (AI), particularly in deep learning (DL) [1], which has been successfully applied in various domains [2]. Thanks to the advancement of the algorithms in combination with the tremendous computing capability of modern hardware, DL has demonstrated transforming capabilities and even superhuman performance in some specific tasks. For instance, with deep neural networks and tree search, AlphaGo, an AI program developed by Deepmind, has defeated the human European Go champion by five to zero. AlphaGo Zero, the successor version of AlphaGo, was completely self-taught without learning from humans and achieved a sweeping victory of 100 to zero against the previously strongest version of AlphaGo [3, 4]. This has demonstrated the possibility of training AI to a superhuman level even without human guidance or domain knowledge.

DL has also been widely applied in clinical medicine and achieved encouraging performance in medical image analysis of computed tomography [5], magnetic resonance imaging [6, 7], ultrasound [8], and ophthalmology images [9,10,11,12]. However, most of these programs still rely heavily on human expertise or basic rules presented in the training datasets. Recently, a novel instrument, Voptica Peripheral Refractor (VPR), was developed, which is able to measure the peripheral refraction and produce a high-resolution 2-D refraction map [13,14,15,16,17]. This has significantly expanded the pre-existing knowledge that ametropia is traditionally defined using solely the central point of refraction in the retina to that thousands points apart from the central point might also be considered. Using this novel instrument, it was revealed, for the first time, that the change of refraction in the vertical region (i.e., superior and inferior retina) synchronized with the center region much better than that between the horizontal (i.e., temporal and nasal retina) and the central region [14], which is again beyond our pre-existing knowledge. Since the proposed VPR is newly developed and the finding mentioned above is sharply new, the aim of the study was to utilize this new knowledge as an example to test whether DL could perform better than human experts. It was also of great interest to investigate how AI without pre-existing knowledge could help human experts to better understand the VPR data, if any.

Materials and methods

Subjects

The data presented in this study were collected from the subjects recruited for a cohort project implemented to investigate the natural history and risk factors of developing school myopia. The inclusion criteria of the project included: (i) astigmatism not greater than 1.50 D, (ii) best corrected visual acuity 20/20 or better, (iii) no ocular diseases, and (iv) no systemic diseases that might influence myopia development. The exclusion criteria included: (i) intraocular pressure > 21 mmHg, (ii) having problems with steady ocular fixation. The data of 214 children were included for analysis. The demographical characteristics of these subjects were summarized in Table 1.

Table 1 Demographical characteristic of the subjects in the study

All experimental protocols met the tenets of the Declaration of Helsinki and had been approved by the Ethical Committee of Aier Eye Hospital Groups (AIER2018IRB15). The children and at least one of the guardians were fully informed about the nature of the study, and written consent had been obtained before the commencement of the measurement. Part of the data from emmetropic children in baseline had been published previously [13].

Dataset of refraction across the retina measured with VPR

Peripheral refraction was measured with VPR, as demonstrated in Fig. 1. Details of the procedure were described in our previous studies [13]. In brief, VPR applied a rotatable L-shaped optical arm with a Hartmann Schack wavefront sensor to scan the refraction at a central horizontal 60° with an interval of 1°. To measure the peripheral refraction in other levels, cross-shaped lighting targets were placed in front of the right eyes at 2.5 m in horizontal distance. For each level, the corresponding light was switched on and remotely controlled by the examiner. The first target on the top and the last target on the button corresponded to the superior retina in 20° and the inferior retina in 16° with an interval of 4°. Thus, the examination covers the peripheral refraction with 60° x 36° visual field. The measuring process was under cycloplegic conditions induced by one drop of Alcaine (Alcon, Japan), followed by two drops of 1% Cyclopentolate (Alcon, Japan) administered 5 min apart. Meanwhile, the left eye was covered during the test. With the data, an individual retinal refraction map was produced for each subject, as previously reported [13].

Fig. 1

An example of retinal refraction map obtained by VPR from one subject with a central refraction of -3.47D. VPR: Voptica peripheral refraction, SER: spherical equivalent refraction

Myopia status labeled by human experts

The measured refraction from the central retina was used to define the categories of refractive error by two consultant ophthalmologists (Z.H. Lin and W.Z. Lan). Since subjects with hyperopia in this population were limited (n = 20), the subjects were classified into two groups, namely myopia (SER<-0.5D) and non-myopia (SER≧-0.5D), where SER stands for spherical equivalent refraction and equals to the spherical power + 1/2 cylindrical power.

Deep learning algorithm for myopia classification

Considering our task is a binary classification of input images (myopia vs. non-myopia), the DL architecture SqueezeNet was utilized in this study [18]. SqueezeNet starts with a convolution layer to input images, as shown in Fig. 2. Multiple modules are implemented using squeeze convolution filters and expanding convolution filters. Following the modules, the architecture ends with a larger convolution filter and a SoftMax layer to output the normalized distributions of classifications. By using smaller convolution filters instead of the usual larger filters, decreasing input channels, and late down-samplings, SqueezeNet could significantly shrink the parameters but still achieve equivalent accuracies to other larger convolutional neural networks (CNN). SqueezeNet has been successfully applied to various classification tasks in medical image analysis [19, 20]. The retinal refraction maps were randomly divided into one training dataset (80%, n = 171) and one test dataset (20%, n = 43), respectively. The training dataset was used to train the SqeezeNet, and the test dataset was used to evaluate the classification performance and later visualization.

Fig. 2

Schematic of the DL structure SqeezeNet (A). The SqeezeNet was utilized in this study to analyze retinal refraction maps and output binary classification predictions of myopia status. The SqeezeNet was implemented with a series of convolutional modules (B) and fire modules (C). Using smaller convolution filters, the number of parameters was significantly decreased, allowing smaller model size and faster computation. DL: deep learning

Interpretable deep learning visualization using Grad-CAM

Due to the staggering complexities of DL architectures, DL is facing critics for the lacking transparency and interpretability. To shed insights into the black box of DL, visualization techniques, such as class activation mapping (CAM), were proposed to identify the discriminative regions in the images using CNN [21]. Based on CAM, without a trade-off of accuracy and interpretability, gradient-weighted class activation mapping (Grad-CAM) was developed to highlight the discriminative details for broader CNN-based DL architectures allowing fully-connected layers [22]. Therefore, Grad-CAM could provide visual explanations for the interpretations of the DL architectures, in other words, the attention of DL. In this study, Grad-CAM was utilized to the SqueezeNet to visualize the critical regions of the retinal refraction maps, which significantly contribute to the successful classifications of the refractive error.

As shown in Fig. 3, the gradient of each pixel in one of the rectified convolution feature maps $\:{A}_{ij}^{k}$, with respect to the myopia classification score$\:{y}^{myopia}$. Was first obtained for each input map. The average weight $\:{\omega\:}_{k}^{myopia}$ was defined as,

$$\:{\omega}_{k}^{myopia}=\frac{1}{Z}\sum_{i}\sum_{j}\frac{\partial{y}^{myopia}}{\partial{A}_{ij}^{k}}$$

indicating the importance contribution of the $\:k$-th corresponding feature map for the myopia class. The overall Grad-CAM for myopia classification $\:{L}_{Grad-CAM}^{myopia}$ was computed as

$$\:{L}_{Grad-CAM}^{myopia}=ReLU\left(\sum_{k}{\omega}_{k}^{myopia}{A}^{k}\right)$$

by applying non-linear activation, rectified linear units (ReLU), to the weighted sum of all feature maps. Therefore, the Grad-CAM represents the pixel-level discriminative contributions to the myopia classification in the original input map. The weights were normalized and colored for better visualization. Lastly, the averaged Grad-CAM by averaging all Grad-CAMs was obtained for all maps in the test set. The Grad-CAMs were visualized as heatmaps. To further interpret the importance of regions in the maps from the perspective of ML, the classification algorithm random forest (RF) was applied and the importance scores were obtained for each pixel.

Fig. 3

The visualization of the DL algorithm using Grad-CAM. The weights for the rectified convolution feature maps were obtained in gradient manner respect to the predicted labels. The weights indicate the pixel-level contributions of the feature maps to the predictions. The Grad-CAM heatmaps were obtained for all test maps by multiplying the weights and the feature maps. Furthermore, the averaged Grad-CAM heatmap was calculated. DL: deep learning, Grad-CAM: gradient-weighted class activation mapping

All programs were developed in Python language (3.7.6). In addition, freely available Python libraries of NumPy (1.18.1) and Pandas (1.0.1) were used to manipulate data, cv2 (4.4.0) and matplotlib (3.1.3) were used to visualize, and scikit-learn (0.24.2) was used to implement RF. SqueezeNet and Grad-CAM were realized using the neural network library PyTorch (1.7.0). The DL network was trained and tested using a DL server mounted with an NVIDIA GeForce RTX 3090 GPU, 24 Intel Xeon CPUs, and 24 GB main memory. In the training of SqueezeNet, the Adam optimizer with a learning rate of 0.0001, a drop-off of 0.5 in the last layer, and epochs of 500 were adopted.

Statistical analysis

Statistical analysis was performed with SPSS (Version 20.00; IBM, Armonk, NY software). The receiver operating characteristic curve (ROC) was used to illustrate the performance of the model. The binary classification was evaluated by the area under the curve (AUC), accuracy (ACC), specificity, and sensitivity with 95% confidence intervals (CI). p < 0.05 was considered statistically significant.

Results

Initially, a total of 231 subjects completed all measurements. However, the data of 17 (17/231, 5.6%) subjects were excluded due to the poor quality of Hartmann-Shack images (e.g., strong corneal reflection or unsmooth transition of retinal refractive pattern). Figure 4 shows the overall workflow of the study.

Fig. 4

The overall workflow of this study. A group of 214 volunteers were recruited and examined using VPR. Retinal refraction maps were obtained and refractive status were determined by experts. The maps were randomly divided into one training set and one test set for DL model training and evaluation, respectively. Prediction performance of the DL model were measured by AUC as the main metric. The Grad-CAM were obtained to present interpretable visualization of the discriminative details of the maps and were found consistent to interpretations of human readers. ML algorithm RF was applied and further confirmed the consistence. VPR: Voptica peripheral refraction, DL: deep learning, Grad-CAM: gradient-weighted class activation mapping, AUC: area under the ROC curve, ML: machine learning, RF: random forest

Deep learning-based myopia status classification

First, the DL architecture SqueezeNet was trained in an end-to-end manner using the retinal refraction maps in the training dataset. During the training process, the weights of the SqeezeNet were updated iteratively to minimize the inference errors. After that, the trained SqeezeNet was used to passively discriminate the test maps in the test dataset. For each map, the SqeezeNet output the predicted binary classification of the myopia status, namely myopia or non-myopia. Based on the predictions, the confusion matrix and the ROC curve were obtained and illustrated in Fig. 5A and B, respectively. As shown, most test maps were correctly predicted with dominating true positive and true negative predictions. The SqueezeNet achieved a satisfying performance in the binary classification with AUC = 0.9074 (95% CI: 0.83–0.97), ACC = 0.8140 (95% CI: 0.70–0.93), sensitivity = 0.7500 (95% CI: 0.51–0.90), and specificity = 0.8519 (95% CI: 0.68–0.94).

Fig. 5

The prediction results obtained using DL. The normalized confusion matrix (A) and ROC curve (B) were illustrated. The confusion matrix showed that most test maps were correctly classified. In the ROC curve, the AUC was 0.9074, indicating a satisfying performance in classification. DL: deep learning, ROC: receiver operating characteristic, AUC: area under the ROC curve

Deep learning visualization

To interpret the results obtained by the DL algorithm, the Grad-CAM approach was utilized to visualize the essential regions that made significant contributions to the determination of the classification prediction. For each retinal refraction map, a corresponding Grad-CAM heatmap was generated. The heatmap highlighted the contributions to the classification result in pixel-level resolution. In the heatmap, the importance of each pixel was normalized and illustrated using a color bar. Red regions were more critical than blue regions. An averaged Grad-CAM heatmap for analysis was obtained by averaging all obtained Grad-CAM heatmaps. As visualized in Fig. 6A, the regions along the central retina were of more important of interest to the DL algorithm than were the peripheral regions, and the importance of the interest decreased with the distance from the center. However, very similar with human experts the ML algorithm RF only focused the very restricted round-shape area in the central retina (Fig. 6B). Obviously, DL algorithm utilized more clues than human experts and ML for a decision-making.

Fig. 6

Important regions obtained by algorithms of DL (A) and ML (B), respectively. A Based on the DL structure, Grad-CAM visualized the importance contribution of each pixel as a heatmap. The central region contributed more significantly to the discrimination of myopia status than the peripheral region. B Using ML, classifier of RF also showed that the most important pixels were located in the central region of the map. Both results of Grad-CAM and RF agreed with human readers who believed the central areas of retinal refraction maps were major regions in determining myopia status. DL: deep learning, ML: machine learning, Grad-CAM: gradient-weighted class activation mapping, RF: random forest

Discussion

In this study, a DL neural network using a set of retinal refraction maps obtained by a novel device VPR has been developed, trained and evaluated. The results showed that DL could accurately discriminate between myopia and non-myopia cases based on retinal refraction maps. Moreover, visualization techniques revealed that our DL algorithm adopted comparable logic as humans to accomplish the task. In addition, it utilized clues that were missed by humans and provided new insights into the domain relevant to the novel device.

Compared with the traditional refractor that only focuses on the central retina, VPR measures the refraction across a very wide region of the retina [13,14,15]. Though this is desired to provide much richer information of the refraction of the eyeball, this significantly increases the complexity of the interpretation of the data, bringing challenges for clinicians and consequently restraining the application of the technique in practice. Using a retinal refraction maps dataset to train the DL neural network, our algorithm achieved an AUC of 0.9074 and an ACC of 0.8140, with a sensitivity of 0.7500 and a specificity of 0.8519. This indicated the trained DL achieved a comparable prediction performance as human experts, remarkably reducing the complexity of reading the retinal refraction maps and significantly improving the efficiency. In addition, the present work applied interpretable visualization approaches to reveal that DL agreed with humans that central zones are the most important regions in the refraction maps to discriminate myopia status.

It is surprising to observe that, besides the central zone, DL also considered the information from other areas human experts ignored. Specifically, DL adopted those in the vertical region across the central zone, while paying significantly less attention to the horizontally far regions. This finding agreed exactly well with our very recent report that the change of refraction in the vertical region synchronized with the center region much better than that between the horizontal and the central region [14]. It should be pointed out that this very recent finding was actually not included in the stage of myopia status determination by human experts. In other words, this ‘knowledge’ was achieved by AI itself, rather than learning from prior expert knowledge. This ‘super-capability’ could be explained by the methodology of DL. DL can gradually learn abstract representations of inputs through multiple layers of neural networks [23]. The deep structures of DL algorithms enabled DL to learn higher abstractions, with which DL could achieve state-of-the-art performance in complicated tasks like image analysis [10, 24]. Unlike human experts who had cumulative experience or prior knowledge of the data, DL only had access to limited information in supervised learning. Therefore, DL investigated the whole distribution of the data at the initial stages before focusing on specific areas with relatively useful information. On the other hand, DL could utilize the tremendous computing capability of modern hardware, which is beyond human brains. Thus, this allows DL to process more data in multiple scales and granularities than human experts. As a result, DL is able to obtain more satisfying results in certain tasks superior to humans [25], as shown in this work that the DL could achieve skills that had not been learned from humans.

There are several limitations in the present study for improvements with future efforts. First, our DL approach relied on the quantity of available sample data. Since the newly developed VPR is a novel device, it takes time to accumulate samples. In the future, it is planned to collect more data from different cohorts and age groups and further evaluate the performance of our DL approach. It is believed with more high-quality data, the performance and robustness of our algorithms would be improved. Moreover, it is planned to expand the diversity of subjects by covering wider geographic areas, populations, and age groups. Second, due to the limited sample size in the present dataset, sub-classification of non-myopia into emmetropia and hyperopia has not been conducted. Whether the accuracy remains satisfactory needs to be further validated by including more data on emmetropia and hyperopia. Third, only one DL architecture was utilized in this study. SqeezeNet was considered in this study due to its apparent advantages, like the small size and good performance in image analysis. Considering the abundance of available and constantly emerging novel DL architectures, the framework proposed in this work could be easily further utilized to evaluate more new DL architectures. Optimizing the present DL architecture by integrating new DL advances could improve the interpretability and performance. Last but not least, the testing instrument (VPR) is a newly developed cutting-edge novel device, which is able to measure two-dimension of refraction. At present, VPR is not widely adopted in practice, there is a lack of available studies based on VPR. Therefore, we can conduct further comparisons with other studies in the future.

Conclusions

In conclusion, this study developed a DL method with satisfying performance to automatically determine myopia status for a novel refractor called VPR. The DL algorithm adopted similar observational logic as humans to accomplish the task, but it utilized more clues than humans, leading to a better understanding of the VPR data and provide new insights into the new domain.