Off-body Sleep Analysis for Predicting Adverse Behavior in Individuals with Autism Spectrum Disorder

2.1 Participants

The study took place at The Center for Discovery (TCFD) in New York State, which provides educational, medical, clinical, and residential care to individuals with severe and complex disabilities including ASD. The participants in this study required a residential level of care having been unsuccessful in a less restrictive environment due to the severity and complexity of their conditions. The center has an internal Institutional Review Board that reviewed and approved of this study protocol.

Fourteen individuals previously diagnosed with ASD ages 15 to 22 (Mean ± Standard Deviation (SD): 18.23 ± 1.84) participated in this study. The sample consisted of 11 white (accounting for 78.6%), two Asian (14.2%) and one African American (7.1%). In terms of gender, 12 were males (85.7%) and 2 were females (14.2%).

The Autism Spectrum Rating Scales (ASRS) [24] is a standardized, norm-referenced instrument that measures behaviors associated with ASD and aligns with criteria from the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) [51]. Because the ASRS is not normed for participants over the age of 18, the last administered ASRS was included for those over 18 during the data collection period. T-scores from 60-64 are in the Slightly Elevated classification; 65-69 Elevated classification; 70-85 Very Elevated classification. As referenced in Table 1, the number and pattern of symptoms associated with ASD as measured by the ASRS DSM-5 Scale was in the Elevated range for three participants and Very Elevated for 11. All participants met DSM-5 criteria for a Level 1 or Level 2 severity for ASD, requiring “substantial” to “very substantial” support based on clinical judgment by a licensed psychologist.

In most cases, cognitive testing was not able to be conducted due to language limitations and inability to respond to test items. As such, level of intellectual disability was determined based on scores on adaptive behavior measures. As part of standard practice, TCFD school psychologists administer either the Adaptive Behavior Assessment System (ABAS) [27] or Vineland Adaptive Behavior Scale-3rd Edition (VABS-3) [26]. The ABAS yields a “General Adaptive Composite” (GAC) score that represents a norm-referenced score for the individual. The VABS-3 provides an “Adaptive Behavior Composite” norm-referenced score based on five domains measured. As can be seen in Table 1, all participants functioned in the moderate or severe range of intellectual disability with adaptive scores in the lowest classification range designated for the particular measure (Low for the VABS-3 and Extremely Low for the ABAS).

Participants had behavior intervention plans to address target behaviors that interfered with functioning. For the purposes of this study, only aggression (e.g., hitting, biting, kicking, scratching), self-injurious behaviors (SIB) (e.g., hitting body parts on objects or with hands, biting self) and disruption (e.g., screaming, throwing objects, dropping) were included in the analysis given their frequency in the sample. The Aberrant Behavior Checklist [8] is a measurement tool that assesses maladaptive behaviors in children and adults with intellectual disabilities. It yields scores in five areas: Irritability/Agitation, Lethargy/Social Withdrawal, Stereotypic Behavior, Hyperactivity/Noncompliance, and Inappropriate Speech. As shown in Table 1, 11 of the participants had at least one subscale on the ABC that was above the 75th percentile, indicating more extreme behavioral presentations compared to a normative sample of peers with disabilities in community-based special education programs [7].

2.2 Data sources

Board Certified Behavior Analysts (BCBAs) at TCFD prescribe and implement BIPs, and behavior data for the challenging behaviors is collected continuously by trained direct care staff across three shifts in a 24-hour period: morning (7:00-3:00) afternoon (3:00-23:00), and overnight (23:00-7:00). Additionally, Raspberry Pi based sensor units with infrared cameras were installed in each participant’s bedroom to record movement across the evening period (19:00-8:00). To preserve privacy, no images were recorded; movement was captured by comparing and recording changes in light intensity, pixel by pixel, from one frame to the next across the recording period. Overnight sensor data was paired with the corresponding participant’s behavior data for this analysis.

The study was approved by the Emory Institutional Review Board and is registered under the project title “Utilizing Technology to Accurately Track Sleep and Predict Adverse Behaviors in Autism” (IRB00003823).

2.3 Sensing pipeline

We employed a Raspberry Pi 4 IR camera (RPi-IR) for data acquisition. The detailed process of movement data acquisition and analysis is illustrated in Figure 1. Our system also incorporates a 5MP OV5647 infrared camera, which has a native resolution of 2592 × 1944 pixels and a 60 Hz frame rate. For computational efficiency and privacy preservation, we downsampled the captured frames to 5 × 4 pixels and down sampled the frame rate to 1.5Hz.

• Download figure
• Open in new tab

Figure 1:

Diagram illustrating the workflow for collecting movement data using a Raspberry Pi 4 Model B equipped with a 5MP OV5647 infrared camera. The camera has a resolution of 2592 × 1944 and a frame rate of 60fps. Frames are downsampled to 5×4 superpixels (with 4 superpixels wide and 5 superpixels high), and the frame rate is resampled to 1.5Hz for analysis. Both Global Difference Sum(GDS) and Local Difference Sum(LDS) are calculated. These features are aggregated for each time frame to generate a multidimensional time series. The resulting data is then securely uploaded to a HIPAA-compliant cloud. In the days following data collection, the behaviorist logs resident labels during two sessions (morning and afternoon) for three distinct target adverse behaviors: aggression, disruptive behavior, and self-injurious behavior (SIB).

3.1 Preprocessing

Under certain assumptions, our algorithms calculate both the Global Difference Sum (GDS) and Local Difference Sum (LDS) [52] for each frame at a distinct time t with pixel resolution M × N. The assumptions were as follows:

1. We have only one person in the entire video.

2. The background is static; the only moving object is either the person or an object attached to them.

3. The video is free from corruption and noise.

Given a frame F_t and its preceding frame F_t−1, the difference frame D_t is D_t = F_t − F_t−1. The GDS is mathematically expressed for each difference frame D_t as:

Similarly, for LDS calculations, the difference frame D_t is partitioned into K × L equally sized local blocks DL_t,s, where we set K = 5 and L = 4. The LDS is then given by:

These GDS and LDS values are aggregated over each time frame to produce a multidimensional time series. This resulting data was securely uploaded to a HIPAA-compliant cloud for further analysis. In a parallel process, trained staff record labels for resident behaviors during two separate sessions (i.e., morning and afternoon), focusing on three adverse behaviors: aggression, disruptive behavior, and SIB.

3.2 Feature extraction

The methodology employed in this study is schematically depicted in Figure 2, feature extraction is the initial phase of our methodological approach. We extract five features for each dimension-modality (i.e., 20 for GDS and one for LDS). This process generates a 5-dimensional feature vector for GDS signals and a 100-dimensional feature vector for LDS signals, culminating in a combined 105-dimensional feature vector. The five crafted features are as follows:

• Download figure
• Open in new tab

Figure 2:

Flowchart illustrating the two-stage methodological approach for sleep analysis. The first stage involves feature extraction from Global Difference Sum (GDS) and Local Difference Sum (LDS), resulting in a 105-dimensional feature vector for each observation. Five specific features are extracted, each chosen for its relevance in sleep analysis. The second stage employs an ensemble of seven machine learning algorithms to make predictive assessments based on the extracted features.

1. 1-norm (Sum of Absolute Values): This measure captures the overall fluctuations within the data series and can be indicative of restless sleep or sleep quality. Larger values may signify frequent movements or disruptions.

   Here, x[i] represents a single dimension of either GDS or LDS in the ith time stamp. Let n denote the number of timestamps recorded in a single night for a resident.

2. Standard Deviation (s): Standard deviation gauges the variability in the data, which can be useful for detecting anomalies in sleep patterns.

   Here, μ is the mean of x and N (see Fig 1) is the total length of the signal over a night.

3. Number of Peaks (NumPeaks): Peaks in the data can represent moments of heightened activity or agitation, which could be of relevance in diagnosing sleep disorders like insomnia or sleep apnea.

4. 2-norm (Energy): 2-norm provides an overall sense of changes in consecutive frames. In this context, a high-energy signal can indicate restless or disrupted sleep.

5. Entropy (H_global): To quantify the complexity or unpredictability of sleep, we calculate the global entropy (H_global). Higher entropy values might suggest irregular sleep patterns.

   The global entropy is computed as an average of local entropies across the signal.

   In this equation, w is the window size used to calculate each local entropy (H_local,i). Specifically, for each window of size w starting at index i in the time series x[i], we calculate a local entropy using the following formula: where P_j represents the empirical probability of the j-th bin in a histogram constructed from the windowed segment of the signal, spanning from x[i] to x[i + w − 1]. This histogram is divided into m bins. The empirical probability P_j is calculated as the proportion of samples falling into the j-th bin within the window, thus defining an empirical probability distribution based on the observed values of x[i] over each window. In this study, we set w=100.

3.3 Prediction Models

Upon extracting features mentioned in Section 3.2, seven predictive models were applied as depicted in Figure 2. These algorithms include the support vector machine (SVM) [15], logistic regression [16], random forests [11], gradient boosting [20], AdaBoost [19], k-nearest neighbors (kNN), and multilayer perceptrons (MLP) [21]. Finally, a voting algorithm consolidates the predictive outcomes from all seven models, enhancing the system’s robustness and reliability by opting for the majority vote as the final prediction label.

Each model is trained using the 105-dimensional aggregated feature vector for prediction tasks, such as forecasting aggression. We repeated this analysis for morning and afternoon for:

1. Target-sensitive Prediction: In this study, the predictive models were designed to forecast both the occurrence and timing of specific forms of adverse behavior at designated times of day. For instance, a single data instance might be labeled with multiple behaviors such as aggression and self-injurious behaviors. The model aims to accurately predict the absence of a particular adverse behavior even when other adverse behaviors are concurrently present.

2. Target-insensitive Prediction: Here, the focus was on predicting the time and presence of any adverse behavior, regardless of its specific type, at a particular time of day.

To develop these models, we allocated 80% of the data for training purposes and reserved the most recent 20% to preserve temporal causality in each subject-specific scenario, such as forecasting aggressive behavior by a specific resident the following day.

3.4 Evaluation metrics

To assess the efficacy of our model, we utilized both conventional evaluation metrics and significance testing against a prevalence-aware baseline.

3.4.2 Validation Through Permutation Testing

To assess the statistical significance of our model’s predictive accuracy beyond mere random chance, we employed a permutation-based significance testing approach. This method involved reshuffling the test set labels to create a distribution of accuracy scores under the null hypothesis, where the model’s performance is assumed to be no better than random guessing. Through this process, we were able to compare the actual model accuracy against this distribution to ascertain whether the observed performance was statistically significant. The approach is detailed in Algorithm 1.

The null hypothesis, H₀₁, in this test, suggests that the observed accuracy is not superior to that achievable by random chance, given the label distribution.

Algorithm 1

Permutation Test

• Download figure
• Open in new tab

3.5 Assessment of Temporal Sleep-Behavior Patterns

We explored the impact of incorporating data from previous nights to improve the predictive accuracy of our models. This enhancement was accomplished by progressively expanding the input feature set, incorporating data from 1 to 7 preceding nights, in addition to the original features obtained from a single night’s data. To achieve this, we concatenated the feature vectors from consecutive nights, thereby forming an extended feature set for each subsequent night. Subsequently, we applied the model introduced in Section 3.3 to these enhanced feature sets to predict the day time behavior.

3.6 Evaluation of Predictive Models

To evaluate the role of each AI-based model presented in Section 3.3. We trained and evaluated each model with the composite 105-dimensional feature vector (single night), derived from sleep data. The training procedure for these models involved allocating 80% of the data for training, while the remaining 20% was set aside for testing purposes. This approach was adopted to maintain the temporal causality essential for predicting subject-specific behaviors, as detailed in Section 3.3.

4.1 Performance Analysis

Table 5 and Fig 3 show the morning and afternoon session performance metrics for each prediction task. We computed the standard deviation and the mean balanced accuracy, presenting the results as mean accuracy (MA)± standard deviation (SD).

• Download figure
• Open in new tab

Figure 3:

Confusion matrices for target-sensitive and target-insensitive models predicting the presence of adverse behavior at different times of the day: (a) Morning using the target-sensitive model, (b) Afternoon using the target-sensitive model, (c) Morning using the target-insensitive model, and (d) Afternoon using the target-insensitive model.

Table 5 presents model performance differences. In the morning, the target-sensitive model had a macro average accuracy of 0.74 ± 0.15, compared to 0.67 ± 0.15 for the target-insensitive model. In the afternoon, these accuracies changed to 0.74 ± 0.18 and 0.75 ± 0.21 for the target-sensitive and insensitive models. However, the afternoon F1 scores were 0.17 ± 0.13 and 0.23 ± 0.19, indicating the inflated accuracy for predicting the presence of adverse behavior in the afternoon.

Fig. 3 further illustrates the challenge in accurately predicting the presence of adverse behavior in the afternoon sessions. The confusion matrices for the morning, shown in Fig. 3 panels (a) and (c), reveal that the target-sensitive model achieved a micro balanced accuracy of 71%, compared to 66% for the target-insensitive model. Conversely, the afternoon session, depicted in panels (b) and (d) of Fig. 3, demonstrates a decrease in balanced accuracy to 54.5% for the target-sensitive model and 52.5% for the target-insensitive model.

Building on the earlier results of model performance in predicting adverse behavior, Table 3 and Figure 4 offer an in-depth look at how these models perform in identifying specific types of adverse behaviors during morning sessions. For aggression, self-injurious behavior, and disruptive behavior, the models achieved macro average balanced accuracies of 0.75 ± 0.19, 0.75 ± 0.33, and 0.76 ± 0.17 respectively. The corresponding macro average F1 scores were 0.74 ± 0.38, 0.74 ± 0.44, and 0.76 ± 0.4. In terms of micro average balanced accuracies, the values were 0.70 for aggression, 0.73 for self-injurious behavior, and 0.72 for disruptive behavior, accompanied by micro average F1 scores of 0.68, 0.8, and 0.75, respectively. These findings highlight the models’ overall effectiveness in predicting different adverse behaviors.

• Download figure
• Open in new tab

Figure 4:

Confusion matrices for predicting different adverse behaviors in the morning time. Confusion matrix for predicting (a) Aggression, (b) Self-injurious behavior, and (c) Disruptive behavior.

4.2 Significance Analysis

Following the evaluation of our model, we carried out the significance tests outlined in Section 3.4.2. These tests were applied to each resident for various adverse behaviors. The resulting p-values are shown in Table 4. This table highlights p-values falling below the 0.05 threshold in bold, denoting instances where the null hypothesis can be rejected. This suggests significant findings in contrast to the permutation test detailed in Section 3.4.2). For scenarios where data points are absent for specific resident-task combinations, “N/A” is indicated.

In predicting the type of adverse behaviors in the morning, the null hypothesis was rejected for aggression predictions in 6 out of 14 instances. For SIB, significant outcomes were observed in 3 out of 10 cases. In the case of disruptive behavior, the permutation test indicated significance in 2 out of 7 instances. Additionally, the permutation test contradicted the null hypothesis for predictions involving target-insensitive adverse behaviors in 7 out of 14 scenarios.

4.3 Predictive Value of Prior Nights

Figure 5, panels (a) and (b), illustrate the models’ accuracy and F1 score, respectively. The findings reveal a modest enhancement in performance with the incorporation of multiple nights’ data. Specifically, the average accuracy for the residents improved from 0.67 (using data from previous night) to 0.68 when data from 7 prior nights was used. A more pronounced improvement was observed in the mean F1 score, which increased from 0.69 (using data from previous night) to 0.74 with the inclusion of data from 7 preceding nights.

• Download figure
• Open in new tab

Figure 5:

Performance analysis of predictive models using different lengths of historical nights’ data(a) F1 Score and (b) Accuracy across different residents using various lengths of consecutive nights for prediction. The graphs showcase the performance outcomes, represented by F1 score and accuracy, for different residents when using varying lengths of consecutive nights’ data for prediction. It is notable that utilizing data from a 7-day period yields the highest results for both metrics, even slightly surpassing the performance achieved when only data from a single preceding night is used.

4.4 Model Assessment

The Ensemble Voting model achieved an F1 score of 0.69 ± 0.21 and an accuracy of 0.67 ± 0.15. This performance notably exceeds AdaBoost, Logistic Regression, and MLP. However, while Random Forest, kNN, and SVM recorded F1 scores of 0.7 ± 0.32, 0.7 ± 0.36, and 0.73 ± 0.38 respectively, the increased standard deviations in their Accuracy and Recall metrics indicate less consistency and reliability. Our Ensemble Voting model offers an advantageous balance, providing robust performance and reliability across various metrics in the target-insensitive model category.

For the target-insensitive task, we observed comparable performance across different metrics for both the SVM and Voting algorithms. These algorithms outperformed the remaining models, achieving an accuracy of 0.71 ± 0.15. Furthermore, the F1 scores registered were 0.76 ± 0.29 for the SVM and 0.75 ± 0.35 for the voting model.