A novel fusion method of 3D MRI and test results through deep learning for the early detection of Alzheimer’s disease

Related work

For the early diagnosis of AD, different methods, such as data diversification or model development, are preferred for the multi-classification of its stages. Demographic information, cognitive scores, genetic and neuroimaging data are commonly utilized in data diversification studies. These data are evaluated either as individual features or integrated with the image data.

In a study published in 2018 that focused on separate evaluations of various data, it was observed that the addition of the Mini-Mental State Examination (MMSE) score to MRI improved accuracy in the classification of healthy controls, MCI, and AD groups using different machine learning methods (10). Another study conducted in the same year combined MRI, positron emission tomography (PET), Rey Auditory Verbal Learning Test (RAVLT), Montreal Cognitive Assessment (MoCA), and Electrocochleography Testing (ECogT) standard neuropsychological test scores to classify stages such as AD, EMCI, LMCI, and CN (11).

Data fusion has found application in various neuroimaging studies, with some combining diverse image types while only a minority integrates demographic information with images. An example involves the combination of MRI and PET using a zero-masking strategy, aiming to extract complementary information from distinct data modalities (12). Additionally, Punjabi et al. (13) demonstrated the fusion of MRI and amyloid PET, two widely used imaging modalities. Preprocessing steps for the scans encompassed MRI bias field signal correction, affine recording, and skull stripping. Initially, the methods were individually compared through the utilization of a 3D CNN. In the fusion approach, both imaging datasets underwent parallel processing in separate branches, with features combined in the latter portion of the network through fully connected (FC) layers.

In 2021, a method based on embedded feature selection and fusion using multi-modal neuroimaging was proposed for AD diagnosis. By combining MRI, PET, and Cerebrospinal Fluid (CSF) Factor biomarkers, promising outcomes were obtained for the classification of Healthy Control, MCI, and AD (14). It has been suggested that the performance of the method for multiple classifications of AD can be improved using multi-modal data. In this regard, various modalities have been developed that combine MRI, PET, cerebrospinal fluid (CSF) biomarkers, and genetic features. It has been observed that more effective results were achieved regarding the multi-modal data scored with the linear discriminant analysis method (15).

In addition to multi-class studies, another study was conducted wherein different types of data were integrated into the image, aiming to ascertain the transition from the MCI stage to AD. Through the fusion of different data formats, Pelka et al. (16) employed a long short-term memory-based recurrent neural network (RNN) model for image classification. This data fusion study implies that superior outcomes can be achieved by integrating socio-demographic and medical data for disease diagnosis. Pelka et al. introduced a branding approach wherein socio-demographic and genetic data were encoded using markers on 2D MRI to attain a more advanced image representation and reduce computational load. By employing the ADNI Phase I and Heinz Nixdorf Recall study databases, the study generated five distinct markers denoting age, gender, education, marital status, and ApoE ԑ4 gene values. The outcomes of the study, wherein branded and unbranded images from each marker group underwent separate classification processes, underscored the general impact of branded data on specificity, F1-score, and accuracy performance metrics for both the Heinz Nixdorf Recall and ADNI Phase I databases.

Payan and Montana (17) proposed a two-step approach for feature extraction in a 3D CNN for the classification of sMRI scans. They built a 3D CNN in which the convolutional layers were pretrained using a sparse autoencoder, and directly used 3D sMRI as input data. However, pretraining with the autoencoder was not fine-tuned; therefore, it was suggested that the performance would have been improved by fine-tuning. The model achieved higher accuracy than those of previous studies for multiple classifications. Similarly, a deeply supervised and adaptable 3D CNN (DSA-3D CNN) built on a 3D convolutional autoencoder (3D-CAE) was proposed to capture variations in anatomical sMRI brain scans based on a previous study (18). As a threshold, the 3D-CAE was pretrained using the CAD-Dementia dataset. The proposed model can learn general and transferable features in different regions. The Hierarchical Attention-based Deep Neural Network (HadNet) architecture was suggested for the classification of MRI images that were normalized and skull-stripped (19).

It was concluded that this architecture achieved superior results in terms of sensitivity and specificity. In another study, demographic information and clinical brain activity test scores were marked on MRI images as landmarks for multi-classification. The preprocessed images and landmarks were trained in the CNN model as two separate inputs. During training, the positions of the landmarks were controlled, and the training of landmarks and images was conducted separately. Joint learning was implemented during training. The main deficiency is that landmark detection and landmark-based classification processes are independent (7). The brain images with the landmark skull in the study included colored data on disease classification. Thus, landmarks should be learned independently from CNN in unsupervised learning. In a previous study in which binary and multi-classifications were made, four classes were used: AD, EMCI, LMCI, and NC (20). In this study, 75 samples from each class label were used in the experiments.

It has been revealed that CNN models outperform machine learning methods in the domain of medical image classification. Due to their revolutionary capabilities in capturing multi-region features, CNN are widely used in the medical field to identify different types of diseases using different medical imaging as input data (21). While a majority of data is in 2D format, like chest X-rays or bone X-rays, and is well-suited for 2D CNNs. Advanced medical imaging (3D) produced by sophisticated equipment can be exceptionally well-identified through the utilization of 3D CNNs. Unlike 2D CNNs, 3D CNNs incorporate cross-layer contexts that are not easily discernible through visual inspection. A pivotal distinction between 2D and 3D CNNs lies in the fact that 3D CNNs can retain the spatial information inherent in images. Volumetric CNNs employ 3D filters and generate a 3D volume output by processing the input through a sequence of convolutional layers, including activation, pooling, dropout, and FC layers. Consequently, volumetric CNNs function on the voxels of 3D images while preserving their spatial attributes.

Some studies have analyzed health records in conjunction with 3D image data, and the methodologies they propose involve distinct stages, such as independent feature extraction prior to classification. The autonomy of these stages leads to increased computational complexity and time requirements. Consequently, to address the challenges associated with this approach and achieve early disease diagnosis, this study introduces a method that fuses and simultaneously analyzes textual clinical data and neuroimaging. Moreover, through classification using a deep learning model, this method aims to discern the disease-induced patterns within the brain and identify various stages of MCI and AD. In alignment with this objective, the study also investigates the influence of health records on early diagnosis, assesses the effectiveness of CNNs in analyzing sMRI data, and explores the application of deep learning within the medical domain. The key contributions of this study are outlined as follows.

• Utilizing 3D CNN while preserving the image’s spatial information.

• The 3DMRIwHR fusion method for combining neuroimages and textual data results in reduced algorithm complexity and time requirements.

• Make use of information between layers for early detection of AD using 3D sMRI processing

• Comparative analysis of classification performance between the 3D and 2D CNN models.

• Enhanced early and late MCI diagnosis through the inclusion of HR for both 3D and 2D sMRI classification.

Data collection

The data employed in this study were derived from the ADNI dataset. Initiated in October 2004 under the guidance of Dr. Michael W. Weiner, the ADNI dataset comprises clinical, biochemical, and genetic biomarkers, along with neuroimaging modalities such as MRI and PET. These resources can be used for early diagnosis, prevention, and treatment of AD. The study encompassed the initial six years of the ADNI project, denoted as ADNI 1. Subsequent phases include ADNI GO (2009-2011), ADNI 2 (2011-2016), and the ongoing ADNI 3 (2016 to the present). Throughout these stages, image protocols were refined, new participants were incorporated, and diverse data types were integrated into the project.

In this study, a cohort of 780 participants was examined, comprising 178 AD patients, 176 patients with EMCI, 161 patients with LMCI, and 265 CN patients. The 3D sMRI scans were acquired in T1-weighted format, with an in-plane spatial size of 1.25 × 1.25 mm² and a thickness of 1.2 mm in 3D MPRAGE format. In addition to neuroimaging data, the study encompassed demographics, cognitive metrics, and genetic information of the participants. Patient health records, encompassing demographic factors like age and gender, ApoE-ε4 genetic data, and MMSE cognitive measure data were fused with 3D sMRI data. Consequently, a 3D CNN model based on 3DMRIwHR images was deployed to classify the four distinct cognitive states associated with dementia: normal cognition, early mild cognitive impairment, late mild cognitive impairment, and AD. Table 1 provides details about the participant count and other pertinent information for each group. The majority of participants were over 65 years old, with MMSE scores exceeding 19, and a greater prevalence of negative ApoE-ε4 values compared to positive ones.

Preprocessing

Taking computational efficiency into consideration, the initial step involved preprocessing the images through the utilization of FMRIB Software Library v6.0.5.2. The 3D sMRI data, initially registered using FMRIB’s Linear Image Registration Tool module, underwent bias field correction using the FLIRT module. This procedure eliminated the density gradient that could impact the segmentation algorithm. MRI measurements pertaining to the skull, skin, fat, muscle, neck, and eyeballs were deemed insignificant for disease identification. A brain extraction tool module was applied to remove non-brain tissues, including the skull. Subsequently, intensity normalization was executed to rectify density variations potentially arising due to differing image sizes. The final preprocessing stage involved resizing the 3D images from 176 × 240 × 256 to 50 × 30 × 20, a resolution validated in [20]. The adoption of larger image sizes would result in heightened computational complexity, rendering them unfeasible for implementation on conventional computers.

To compare the impact of fusing health records with MRI scans for multi-class image classification, both 2D and 3D image data fusion methodologies were employed. From the preprocessed and unresized 3D images, 2D slices were extracted after skull-stripping. In the final phase of 2D MRI preprocessing, these slices were saved in PNG format with 256 × 256 pixels. Within the ADNI dataset, subjects with health records encompassing age, gender, ApoE-ε4, and MMSE are limited and are imbalanced across AD, LMCI, EMCI, and CN categories. Following a 20% allocation for testing, data augmentation was undertaken to increase the dataset’s size for both the 2D and 3D CNN models. The number of images per class, initially set at 1000, was doubled through this process. Consequently, the shuffled dataset, encompassing 8000 MRI scans, was subsequently partitioned into two sets: training and validation.

Fusing 3D sMRI with Health Records: 3DMRIwHR Method

In the diagnosis of Alzheimer’s disease, a specialist physician conducts a range of examinations after reviewing the patient’s medical history. These examinations encompass neuropsychological, neurophysiological, genetic, laboratory, neuroimaging, and nuclear medicine tests. Disease diagnosis isn’t solely reliant on neuroimaging; it also involves the analysis of diverse data types. Healthcare professionals face restricted opportunities to effectively utilize these varied data forms. Despite the growing body of research on this subject, MRI remains the preferred approach. In this study, we introduced a method named 3DMRIwHR, which facilitates the assessment of neuroimaging data by combining demographic, genetic, and cognitive test score outcomes. When selecting test results represented by 3D-HR markers, factors that heightened the risk of AD symptoms were considered.

3D sMRI contains additional information between layers, a feature not present in 2D sMRI. However, extracting this information through visual inspection of 3D sMRI images is not trivial. Machine learning techniques, however, can effectively harness this information. To underscore the interdimensional impact of the 3D method, a fusion procedure was applied to 2D slices derived from 3D images within the proposed deep learning model. This approach, utilizing the same data, is designated as 2DMRIwHR.

The approach used in the study encompassed the fusion of text-based information with sMRI scans through the utilization of 3D volumetric pipe-shaped markers. In marker design, preference was given to half and full rectangular prism shapes in black or white, with each color symbolizing specific attributes. These textual attributes included age, gender, ApoE-ԑ4 gene value, and MMSE score ranges. Figure 1 illustrates the attribute values associated with these colored volumetric pipes. The characteristics of the markers utilized in the fusion of 2D images, along with their corresponding value ranges and arrangement on the image, were applied similarly to the 3DMRIwHR method, with no changes except for spatial criteria. Upon extracting the 2D images as detailed in the data preprocessing section, they were combined with 2D markers that represented textual information. This methodology was labeled as 2DMRIwHR. Following this technique’s application, training was done on the proposed 2D model without including any additional image enhancements

• Download figure
• Open in new tab

Figure 1. 3D pipe shape coding for health records

In this study, four distinct health records — age, gender, MMSE, and ApoE-ԑ4 gene data — were utilized to construct 3D-HR pipes, as shown in Figure 1. Each of the 3D pipes created using the 3DMRIwHR method had a height of eight pixels, a width of 20 pixels, and a depth of 256 pixels.

The processing involved 3D images in NIfTI format, with dimensions of 176 × 240 × 256, carried out in the axial plane, spanning from pixel points (2, 10) to (10,49). Gaps of size (5 × 4) were deliberately left between the pipe markers to ensure that volumetric markers did not generate any erroneous correlations. 3D-HR pipes were located in the corners of the 3D sMRI scans for the purpose of data fusion. The corner’s selection was such that it didn’t impede the 3D brain image. In the case of the 2DMRIwHR method, 2D planar markers, with dimensions of eight pixels in height and 20 pixels in width, were employed. The positioning process on the image closely resembled that of the 3DMRIwHR method.

To visualize the Nifti scans, the 3D axial preprocessed and 3DMRIwHR images were transformed into PNG files, as shown in Figure 2. The 3D-HR pipes representing age, gender, ApoE-ԑ4, and MMSE scores on the image were placed in a specific order.

• Download figure
• Open in new tab

Figure 2. 3D MRI with and without 3D-HR pipes

2D MRI images were extracted from the 3D sMRI images by means of slicing. After the slicing phase, the 2D images in Nifti format were combined with health records using 2D encoding. The markers employed in the 2D images denoted the same values as those in the 3D-HR pipes. While the alignment of marker images remained consistent, due to the 2D and planar nature of the images, the 2D markers were depicted as rectangles, measuring eight pixels in height and 20 pixels in width.

Classification for Decision Support

CNNs are a specific type of neural network designed for processing data organized in matrix form. Convolutional networks have proven effective in real-world applications, yielding satisfactory outcomes not only with large datasets but also when adeptly modeled for smaller datasets. In the realm of image processing studies [19,20,22], the architecture of CNN best fits the concept of learning regional patterns by transforming pixel data into matrices. Given that dementia impacts specific brain regions, we employed 2D and 3D CNN architectures to classify the four classes, namely CN, EMCI, LMCI, and AD.

3D CNN method

The proposed model comprises three 3D convolutional layers and three 3D max pools. Following these layers, a dropout layer was introduced, succeeded by a flattened layer. At the model’s last layer, there existed an FC layer employing a softmax activation function. The 3D convolutional layers were all equipped with 3 × 3 × 3 kernels and 1 × 1 × 1 stride values, featuring 64 filters in the initial two convolutional layers and 128 filters in the final layer. The pooling layer kernels were all of size 3 × 3 × 3, with stride values set at 2 × 2 × 2. The number of training parameters was 1,710,660, whereas the number of non-trainable parameters was 0. For optimization during parameter training, the adaptive moment-estimation (ADAM) optimization algorithm was utilized with a learning rate of 0.001.

Since the rectified linear unit (RELU) is currently the most commonly applied function [21], it was used as the activation function in the three convolutional layers. A dropout value of 0.5 was selected to mitigate the risk of overfitting. Following the softmax activation function in the final FC layer, a 4-class disease classification was executed. The architectural layout of the 3D CNN model can be observed in Figure 3.

• Download figure
• Open in new tab

Figure 3. 3D CNN Model Architecture

2D CNN Method

In the 2D CNN model, there were three 2D convolutional layers accompanied by three 2D max pools. These were followed by a dropout layer, succeeded by a flattened layer. The model’s conclusion included an FC layer employing a softmax activation function. The 2D convolutional layers all consisted of 3 × 3 kernels with 1 × 1 stride values, with 64 filters present in the initial two convolutional layers and 128 in the last convolutional layer. The pooling layer kernels were all 3 × 3 in size, with stride values set at 2 × 2. The number of training parameters was 16,889,284, whereas the number of non-trainable parameters was 0. An optimization algorithm, specifically "ADAM" was utilized with a learning rate of 0.001 during parameter training.

As for the activation function, RELU was utilized for the three convolutional layers due to its widespread application [21]. A dropout value of 0.5 was selected to counteract overfitting. The classification of the four dementia groups was carried out based on the softmax activation function in the final FC layer.

Experimental Design

In our study, the proposed models considered the impact of health records on classifier performance. To illustrate the effect of health records on recognition rates, the comparison was made between 2D MR vs. 2DMRIwHR and 3D MR vs. 3DMRIwHR, utilizing 2D CNN and 3D CNN deep-learning techniques, respectively.

The 2D and 3D classification methods were implemented using the Keras 2.8.0 library in Python 3.9, built on TensorFlow-gpu 2.8.0. The execution occurred on a PC equipped with an NVIDIA RTX2060 GPU running the Windows operating system. In the network, the Adaptive Moment Estimation (ADAM) optimizer was employed, starting with an initial learning rate of 0.001. A dropout rate of 0.5 was set to mitigate overfitting. For 3D classification, the batch size was set to 64, while the 2D classification employed a batch size of 128.

The image count for each class was divided into two sets using a 0.8 training and 0.2 testing ratio. Subsequently, data augmentation was applied to the train set, effectively doubling the number of images in each class. As a result, the overall training dataset containing 8000 images was divided into a training set (comprising 64% of the images), a validation set (consisting of 16% of the images). During the classification phase, only images belonging to the class with the lowest image count were utilized to accurately assess the impact of the pipe-laying process. This approach ensured balanced representation among classes.

Data Augmentation

The image count for each class was divided into two sets using a 0.8 training and 0.2 testing ratio. Subsequently, data augmentation was applied to the train set, effectively doubling the number of images in each class. As a result, the overall training dataset containing 8000 images was divided into a training set (comprising 64% of the images), a validation set (consisting of 16% of the images).

The 2D and 3D classification methods were implemented using the Keras 2.8.0 library in Python 3.9, built on TensorFlow-gpu 2.8.0. The execution occurred on a PC equipped with an NVIDIA RTX2060 GPU running the Windows operating system. In the network, the Adaptive Moment Estimation (ADAM) optimizer was employed, starting with an initial learning rate of 0.001. A dropout rate of 0.5 was set to mitigate overfitting. For 3D classification, the batch size was set to 64, while the 2D classification employed a batch size of 128.