Abstract
Mild cognitive impairment (MCI) is clinical diagnosis that refers to individuals whose performance is below average on standardized cognitive tests, but who otherwise function independently in instrumental activities of daily living. Few prior studies have addressed the problem of selecting the optimal combination of behavioral instruments and cutoff scores for detecting MCI in an outpatient setting. The aim of this work is to provide insight into two related questions: (1) What is the relative sensitivity and specificity of a battery of standardized tests frequently used to assess for MCI, as defined using receiver operating characteristic (ROC)-based analysis? (2) What are the optimal “cut point” scores for distinguishing patients’ mildly impaired performance based on these instruments?
Two hundred forty outpatient behavioral neurology evaluations were retrospectively analyzed. All work was conducted with the formal approval of the Johns Hopkins University School of Medicine Institutional Review Board. All instruments that were evaluated performed very well in the detection of dementia (mean AUC = 0.8). However, fewer tasks performed acceptably in the detection of MCI (mean AUC = 0.7). Instruments that performed best in the detection of MCI tended to have higher total possible scores or not to reflect a score out of a total number possible.
Cognitive screening tools, like the MMSE, did not perform well in the detection of MCI, raising important considerations for their interpretation. No one task in isolation is sufficient to detect MCI, and behavioral performance is not the only relevant consideration in differential diagnosis. However, these findings highlight the relative weakness of many assessments when used to build a comprehensive profile of a very large portion of outpatients presenting at clinic, those whose deficits are more subtle.
Introduction
Mild cognitive impairment (MCI) is clinical diagnosis that refers to individuals whose performance on standardized cognitive tests has fallen outside of the average performance for individuals like them. People with MCI otherwise function independently in instrumental activities of daily living and do not meet criteria for a diagnosis of dementia.1 In practice, this frequently means task performance by people with MCI is poorer than one standard deviation below the normative performance available that permits the closest comparison to their baseline.2 Normative performance generally is parsed by decade of age, sometimes sex, and rarely education. The potential lack of specificity or congruency between individual characteristics influencing performance on cognitive and cognitive communication tasks in a given task or battery of tasks and available norms is well-known. It is a widely discussed problem in psychometrics that can lead to both over- and under-identification of clinical profiles, such as MCI.3-5
Concurrently, the identification and understanding of MCI faces another set of challenges related to its nature and the limitations of tools used to detect it. Because its definition is linked inextricably to performance, it is inherently heterogeneous in underlying etiology. People can have mild cognitive impairment in the absence of any observable clinical change on standard magnetic resonance neuroimaging. Due to the heterogeneous etiology, some individuals with MCI improve over time back to performance within normal limits (WNL) for their age, while some appear to remain statically impaired, and some progress further from normative performance to dementia. The ability to distinguish among these fates is a high-stakes, high-investment line of research in the clinical cognitive and neuropsychological sciences.6-8
In the meantime, as these predictive models continue to be refined, the identification of MCI faces another problem. That is, individuals who report cognitive change in an outpatient setting first are evaluated in most cases using tests designed for the detection of dementia, not the detection of MCI. While it may appear at first glance to be a minor point of concern, this can be conceptualized as another kind of normative performance mismatch to the individual, akin to poor normative comparators for an individual’s education or culture. Tests designed for the quantification of larger changes do not necessarily perform well when detecting small changes, especially if the individual is non-central to the assessment’s normative sample in other ways, for example, being of a younger age than most of the sample. In these circumstances, “normal” performance can equate to ceiling performance (see examples in Table 1), rendering it difficult if not impossible to verify whether an individual’s subjective impression of cognitive change is corroborated by changes in performance (either at the first evaluation or subsequent visits). Few prior studies have addressed the problem of selecting the optimal combination of behavioral instruments and cutoff scores for distinguishing typical and atypical performance associated with MCI across domains in a typical outpatient setting. This gap in the literature leads to the use of extended batteries that often are impractical to implement and challenging for patients, particularly those further in their disease progression, to complete. The aim of this work is to provide insight into two related questions:
What is the relative sensitivity and specificity of a battery of standardized tests frequently used to assess for MCI (comparable to the National Alzheimer’s Coordinating Center Uniform Dataset), as defined using receiver operating characteristic (ROC)-based analysis?
What are the optimal “cut point” scores for distinguishing patients’ mildly impaired performance based on these instruments?
Our hypothesis was that assessments designed to have no clear maximum (e.g., phonological fluency) or a very high maximum relative to common performance (e.g., delayed recall) would outperform more commonly utilized, often abbreviated assessments with relatively low maximum possible scores. This hypothesis was informed by our prior analyses of the National Alzheimer’s Coordinating Center frontotemporal lobar degeneration module when used to detect primary progressive aphasia,9, 10 in which a similar pattern of results was observed.
Materials and Methods
We report how we determined our sample size, all data exclusions, all inclusion/exclusion criteria, whether inclusion/exclusion criteria were established prior to data analysis, all manipulations, and all measures in the study.
Records reviewed
Diagnosis of mild cognitive impairment was determined using a battery of standardized tests (described below), in conjunction with comprehensive neurological evaluation, review of medical history, and consultation with family, if present. The findings of this evaluation were affirmed with magnetic resonance imaging for all patients to rule out the presence of a clear neurological etiology to explain the change observed.
Patients seen in the Johns Hopkins Outpatient Center between under investigation for cognitive impairment were examined retrospectively. Three hundred evaluations of 255 patients seen between December 2018 July 2023 were selected at random. All completed assessments were considered for analysis. No part of the study procedures or analysis plans was pre-registered prior to the research being undertaken. Sixty evaluations were removed due to a known etiology that accounted for cognitive impairments observed in testing: attention deficit disorder (2), cognitive impairment secondary to stroke (26), cerebral amyloid angiopathy (2), anoxic brain injury (4), traumatic brain injury (5), arteriovenous malformations treatments (3), other known conditions resulting in cognitive deficits (11), and other cognitive disorders (7). The remaining patients contributing 240 evaluations are summarized in Table 2. On Fisher-Freeman-Halton Exact Tests, groups were similarly dichotomized in gender distribution and psychiatric comorbidities. While individuals within normal limits were younger than those with clinical diagnoses of either MCI (t(77.7)=3.5, p < 0.001) or dementia (t(116)=2.6, p=0.009), those with clinical profiles did not significantly differ in age.
Assessment battery
A battery of widely used assessments of cognition was employed for each evaluation, the Hopkins Cognitive Battery. The battery contains both simple questions and those designed to probe complex of cognitive demands, even if at the expense of brevity (Table 3). The Mini Mental Status Examination (MMSE) was most commonly used until 2019, when it was replaced by the Wechsler Adult Intelligence Scale--Third Edition (WAIS-III) Wechsler Memory Scale (WMS) Orientation and Information questions.12
Statistical analysis
Aim 1
The first aim of this work was to examine the relative discriminatory performance of the assessments within the Hopkins Cognitive Battery with regard to sensitivity and specificity across the three contrasts. First, we examined WNL versus MCI+Dementia combined, as determining the presence of clinical change is the most consistent with the scenario when a patient presents in an outpatient clinic under investigation of cognitive change. Next, we examined the two constituent contrasts, WNL versus MCI and WNL versus Dementia, separately. Each visit was treated as an independent, not repeated, portrayal of clinical characteristics. This maximized the overall size of the sampling used for discriminatory analysis, but ignored the potential intra-patient influence; however, 29 patients contributed more than one visit to the analysis.
As in our prior analysis of detection using NACC battery assessments,9, 10 calculations were done in R using plotROC 2.2.21 to calculate empirical ROC curves and cutpointr for optimizing the cutoff score22 on the unsmoothed curve. Receiver operating characteristic (ROC) analysis illustrates the binary diagnostic ability (present or absent) of a given task as the critical threshold score is varied. In this analysis, 0.5 represents chance and 1.0 represents perfect performance. The x-axis is the false positive rate (FPR, also 1-specificity). The y-axis is the true positive rate (TPR, also called sensitivity). Area under the receiver operating characteristic curve (AUC) is a common way to define the overall performance of that measure in performing the diagnostic classification. AUROC 0.7/1 was used to vet tasks most useful to detection, as this is widely considered acceptable performance of a task.23 The tasks with acceptable performance detecting MCI versus healthy performance then were entered into a binomial logistic regression. This was done to estimate the overall diagnostic utility of the subset of selected tasks in predicting MCI status. The significance of the model was considered against α = 0.05.
Aim 2
The second aim of this work was to identify the optimal cut-point scores for detection of MCI based on these instruments. In order to examine Aim 2, “optimal” cut-points were established by maximizing the Youden-Index (sensitivity + specificity – 1)24 for any task with a AUC of ≥ 0.7 in the detection of MCI (WNL vs MCI). These values provide insights into scores where specificity and sensitivity interests are balanced, and can be further described in terms of accuracy, sensitivity, and specificity at that score.
Standard Protocol Approvals, Registrations, and Patient Consents
All work was conducted with the formal approval of the Johns Hopkins University School of Medicine Institutional Review Board (IRB-3; Federal Wide Assurance # FWA00005752-JHUSOM, FWA00006087-JHH & JHHS, and FWA00005719-KKI OHRP IRB Registration #IRB00400111). Participant consent was not required.
Data Availability
The conditions of our ethics approval do not permit public archiving of study data. Anonymized data not otherwise present in the appendix are available upon request to the authors, subject to review by the Johns Hopkins University School of Medicine Institutional Review Board resulting in a formal data sharing agreement.
Results
Aim 1: Receiver Operating Characteristic (ROC) curves
Results of the ROC analysis are presented in Table 4 (Aim 1). While clinical change (either MCI or dementia) was acceptably detected by most administered tasks, this was largely driven by very high AUCs representing the larger magnitude of difference between normal performance (WNL) and performance consistent with a diagnosis of dementia. This is consistent with the fact that these batteries and individual tasks are designed with magnitudes of change consistent with dementia in mind. However, relatively few tasks performed acceptably in the detection of MCI, as defined by the a priori cutoff of 0.7 (Table 4, bolded). Notably, those that did perform well tended to have higher total possible scores or not to reflect a score out of a total number possible. This was consistent with our hypothesis regarding the tasks that would be best for identifying MCI.
As planned, acceptably performing tasks were entered into a logistic regression to determine how they performed in concert with one another when predicting MCI status versus healthy performance. However, modifications were made based on the limitations of the data as observed a posteriori. To reduce the number of dimensions and multicollinearity among predictors, only the best performing score from a given test where multiple scores qualified was entered: digits backward, Rey delayed figure copy, RVLT delayed recall, phonological fluency in FAS, Trail-making test B, and Stroop Color-Words. Second, as very few patients completed the Stroop task, this was removed to maximize the available data with which to calculate the model (22 versus 103).
Overall, the model was significant (χ2(5) = 60.06, p < 0.001), accounting for approximately 61% of the variance in MCI status (Nagelkerke’s R2) and classifying 85.4% of cases correctly (Table 5). Holding each of the other tests constant, the odds of performing consistent with MCI decreased by 13% with each additional point on delayed figure copy and by 23% with each additional point on the RVLT delayed recall. No other predictors were independently significant.
Aim 2: Optimal cut-point scores for detecting MCI
Optimal scores were calculated for acceptable tasks based on detection of MCI versus healthy performance. While these calculations are not age-corrected, an important observation is how many fall within one standard deviation of normal performance for individuals 60-69 years old (Table 6 †) and all accept the optimal score for Stroop color-word performance fall within one standard deviation of normal performance for individuals 70-79 years old. This highlights the difficulty in identifying MCI on the basis of behavioral testing using common measures for the detection of dementia in older adults.
Discussion
When patients present for an outpatient neurological evaluation due to concerns regarding perceived cognitive change, they most commonly receive assessments designed with the detection of dementia in mind. However, based on our sample, nearly half (48%) of those who presented showed abnormal performance in at least one domain and otherwise met criteria for MCI, rather than dementia. To better understand task performance among this etiologically diverse and poorly understood population, we pursued two aims using a large retrospective sample of outpatient data. First, we examined the acceptability of individual commonly administered tasks in a receiver operating characteristic (ROC) analysis. Our hypothesis was that the tasks that performed best would be those in which the most common scores were well below ceiling performance (and well above floor) either due to high numbers of points possible or no maximum score (i.e., counts). Next, we used the acceptability threshold to calculate a binomial logistic regression determining how tasks worked together to detect MCI. Finally, we calculated optimal scores for distinguishing between performance of healthy individuals and performance of those with MCI. To assist in the interpretation of these values, we briefly contrasted them with available norms.
Based on our ROC analysis, the Hopkins Cognitive Battery tasks performed well in the detection of clinical change generally. However, this was driven by nearly perfect detection of dementia, rather than the milder deficits associated with MCI. Our hypothesis regarding which tasks would perform best in detecting MCI was supported; high performing tasks tended to be those like the Rey Complex Figure Copy and Rey Auditory Verbal Learning Test, which both are quite difficult, and thus, normal performance is well below ceiling performance. The number of words produced beginning with F, A, and S and amount of time needed to complete the Trail-making Test Part B time also performed well in detection of MCI. While Stroop Color-Word naming (that is, naming the color the word, such as BLUE, is printed in), also performed well, relatively few instances of performance were available for analysis of this assessment. Thus, it should be interpreted thoughtfully.
Particularly unfortunate was that the widely used cognitive screening tool, the MMSE, did not perform well in the detection of MCI. Perhaps less surprisingly, the Wechsler Memory Scale (WMS) Orientation and Information questions did not perform well either. This is an important observation for its practical clinical significance. If a patient comes in describing perceived cognitive change, and their performance on these abbreviated measures is within normal limits for their age, it could easily lead to missed identification of true MCI by well-meaning generalists, and those with MCI who do progress to dementia would have missed out on having an earlier understanding of their risk. A more conservative approach that is supported by the strong performance of tests more commonly used by behavioral neurologists and neuropsychologist would be to provide a referral for full cognitive evaluation for these patients. In that setting, an extended evaluation utilizing multiple high performing assessments is possible and may provide corroborative evidence of the patients’ experience or, conversely, provide a comprehensive personal baseline for a healthy aging individual that can be referenced in years to come.
Examining optimal cutoff scores establishes a useful guideline for critically examining a patient’s profile assessment by assessment. However, as previously noted, many of these scores would be considered typical of healthy individuals even as young as 60. This finding highlights two important points regarding interpretation. First, among those tested, no one task in isolation is sufficient to detect MCI, as expected because there are distinct types of MCI (amnestic, non-amnestic, single domain, multiple domain). Furthermore, behavioral performance is not the only relevant consideration in differential diagnosis. A more sensitive and specific approach to detection likely requires task performance to be considered in conjunction with a review of medical history, consultation with family, and, if possible, neuroimaging. These findings also reinforce that the currently available assessments and subsequent norms, most based only on age, have limited utility in detecting very mild impairments. As we become increasingly capable of treating the underlying etiologies that lead to cognitive impairments, the number of people whose impairments are mild and who are less frequently fully disabled by such impairments likely will increase. The lack of assessments in widespread use that can detect such impairments and the poverty of individualized norms available for well-established tests is an important area of future innovation.
One limitation of this work was that the number of instances of certain tasks available for analysis was quite low (e.g., Stroop). These tasks are administered relatively infrequently relative to the remainder of the battery. The Stroop task was not administered systematically during the coronavirus pandemic, when assessments were conducted almost exclusively using telemedicine and video conferencing. Even in person, the Stroop is not appropriate for many individuals, including those with color blindness or presbyopia. Future work should expand upon the analyses of these instruments in a larger sample.
Overall, these findings represent a positive impression of frequently used tests in accomplishing its key purpose: detecting dementia. However, they also highlight the relative weakness of our most administered assessments when used to build a comprehensive profile of a very large portion of outpatients presenting at clinic, those whose deficits are more subtle. It is our hope that this work will galvanize future clinicians and clinical researchers to develop a better understanding of MCI and the heterogeneous individuals who experience subtle cognitive changes beyond what is expected for their age.
Data Availability
The conditions of our ethics approval do not permit public archiving of study data. Anonymized data are available upon request to the authors, subject to review by the Johns Hopkins University School of Medicine Institutional Review Board resulting in a formal data sharing agreement.
Declaration of interest
The individuals who completed this work are supported in part by NIH/National Institute on Deafness and Other Communication Disorders (NIH/NIDCD): P50 DC014664, R01 DC05375, R01 DC015466, and R01 DC011739. Dr. Hillis receives compensation from the American Heart Association as Editor-in-Chief of Stroke and from Elsevier as Associate Editor of PracticeUpdate Neurology.
