Facial and vocal markers of schizophrenia measured using remote smartphone assessments

1 Introduction

There is rapidly increasing utilization of remote digital measurements in clinical research and practice. The development and validation of digital measurement tools in psychiatry come with both significant opportunities and risks. Significant opportunity arises as psychiatry is already undergoing a paradigm shift towards the utilization of objective markers to assess illness and disease progression (Insel, 2018) and towards the widespread use of telehealth platforms for psychiatric care. This is particularly important when face-to-face medical care is not possible, such as during the COVID-19 pandemic (Figueroa & Aguilera, 2020).

Many behavioral and physiological markers are now accessible through digital technology such as wearables, mobile/web apps, and application programming interfaces (APIs; Insel, 2017). Such advances hold promise in allowing new innovations in neuropsychiatry to truly scale in a manner where they can be utilized to develop and implement treatment for patients suffering from significant psychiatric impairment (Insel, 2019).

Schizophrenia represents a poignant example of both the benefits and challenges of remote digital measurement. Clinical trials for schizophrenia drug development are often site-centric, requiring patients to appear physically at the site for measurement of disease severity. The need to travel to sites can restrict study populations to those that live in geographical proximity to the site, restricting access to participation and limiting patient diversity (Lecomte et al., 2020). Current approaches for measurement of disease rely on clinician administered measures that are costly and time-consuming to administer, leading to infrequent assessment and low adherence to treatment regimens. The instruments themselves are not well-aligned with current neurobiological definitions of illness (Torous et al., 2018). For example, cognitive and motor dysfunction, which are symptoms of the negative subtype of schizophrenia, are not accurately assessed through existing scales.

Remote digital measurements are not prone to the same level of subjectivity as human raters, and can provide both scale and ease of use while better aligning clinical measurement with behavioral and physiological measurements of underlying neurobiological treatment targets (Marsch, 2018; Torous & Keshavan, 2018). Indeed, a large number of physiological and behavioral measures have demonstrated validity as markers of schizophrenia severity or caseness. Such markers, often examined in controlled laboratory settings, hold promise as accessible remote proxies to track clinical functioning in patients with schizophrenia. However, there is a need to determine their reliability when captured in real world settings, where differentiating between significant variability and noise can pose a challenge.

A number of behavioral characteristics of schizophrenia, such as alogia (poverty of speech) and affective flattening (diminished emotional expression/emotional withdrawal; (Tandon et al., 2013) can be quantified directly using standardized tasks and coding schemes (Alberto et al., 2019; de Boer et al., 2020; Kohler, Martin, Milonova, et al., 2008; Mandal et al., 1998; Mattes et al., 1995), which can be automated through use of computer vision (Baltrusaitis et al., 2016) and vocal acoustic (Jadoul et al., 2018) machine learning models. In addition to digital measures that are directly analogous to core schizophrenia symptomatology, there are a number of other acoustic measures including vocal loudness, pitch variability, fundamental frequency, and jitter that have demonstrated validity as markers of schizophrenia (Alberto et al., 2019; Covington et al., 2012; Martínez-Sánchez et al., 2015; Saxman & Burk, 1968). These markers have demonstrated specificity as measures of the negative symptom cluster in particular (Covington et al., 2012).

In the current investigation, we examine the ability to measure schizophrenia severity through facial and vocal analysis using videos recorded during a remote smartphone-based assessment composed of both evoked and spontaneous prompts. We compare these measures against standard clinical assessments of overall schizophrenia severity (i.e. Positive and Negative Syndrome Scale (PANSS) Total) as well as specific domains of positive (P Total), negative (N Total), and general (G Total) symptoms, measured during clinic visits. We further examine the relationship between digital measures and individual symptoms of schizophrenia.

2 Methods

2.2.2 Remote smartphone-based assessments

On the first day of the study, all study participants were trained by clinic staff on how to use the smartphone application (www.aicure.com) for remote data collection. The smartphone application was used to participate in remote assessments that would capture video and audio of participant behavior using the smartphone front-facing camera as they responded to on- screen prompts (Figure 1). Participants were allowed to use their own smartphones or use smartphones provisioned to them by the clinic staff for the duration of the study. The assessments were taken at scheduled time points over the course of the 14 days and were designed to capture two main kinds of behaviors as described below.

• Download figure
• Open in new tab

Figure 1:

Example screenshots from the smartphone assessment all study participants took for remote and automated collection of video and audio data. During each of the prompts, the app speaks the text displayed on the screen and awaits a verbal and visual response from the participant, all while recording video and audio from the front-facing camera and microphone. (a) Screen displayed before the participant begins the assessment. (b) Prompt for collection of free behavior in response to images, showing one example image. (c) Prompt for collection of evoked facial expression behavior. (d) Prompt for collection of evoked vocal expression behavior.

Free speech and spontaneous expressivity

Participants were shown images from the Open Affective Standard Image Set (Kurdi et al., 2017) and asked to describe the images and talk about how they made them feel (Figure 1b). The participants’ speech and facial expressivity in response to the prompts were captured (Alberto et al., 2019; Cohen et al., 2016; Kohler, Martin, Milonova, et al., 2008; Kohler, Martin, Stolar, et al., 2008; Mandal et al., 1998; Mattes et al., 1995; Schwartz et al., 2006). This assessment was conducted on days 2, 7, and 14 of the study. Measurements acquired from each timepoint of the assessments were averaged for reduction of noise before comparison with PANSS.

Evoked facial and vocal expressions

Participants were asked separately to make the most expressive face they could and hold it for 3 seconds (Figure 1c) and then say the names of the days of the week out loud (Figure 1d). These prompts were selected based on prior experimental tasks used to examine emotional activity and speech in schizophrenia (Alpert et al., 1997; Kohler, Martin, Stolar, et al., 2008). The captured video and audio was used to measure facial expressivity and acoustic characteristics of voice during the evoked expressions. These assessments were scheduled on days 1, 7, and 14 of the study. Measurements across the time points were averaged for reduction of noise before comparison with PANSS.

3 Results

4 Discussion

In the current investigation, we sought to test the hypothesis that facial and vocal markers of schizophrenia can be captured remotely in patients using brief automated smartphone-based assessments and that such measures would be well-correlated to standard clinical measures of schizophrenia symptom severity. Such measures show promise of objective and automated methods of assessing illness severity in the context of treatment development and decision making. Prompts and vocal/facial measures that have previously demonstrated accuracy in controlled research settings were simplified and deployed as a brief assessment via a smartphone application in an observational study with schizophrenia patients. Results support the ability to measure meaningful clinical markers of schizophrenia severity via a brief smartphone based assessment that captures data remotely and processes it through back-end deep machine learning algorithms to create vocal and facial markers.

Results demonstrate that vocal characteristics such as fundamental frequency, loudness, non- verbal vocal tones and prevalence of speech serve as specific markers of negative symptom severity. The majority of these markers demonstrate a robust signal of negative symptom severity regardless of whether prompts were evoked or spontaneous.

The observation that vocal markers provide specificity as a metric of negative symptom severity has significant practical implications for clinical research and decision making. Recent advances in the mechanistic understanding of negative symptomatology have led to a number of promising pharmacological and cognitive treatments for negative symptoms of schizophrenia (Erhart et al., 2006; Fusar-Poli et al., 2015; Millan et al., 2014; Singh et al., 2010). Such initiatives are important given the lack of FDA-approved treatments for negative symptoms (Kirkpatrick et al., 2006). However, reliable and change-sensitive measures of negative symptomatology to assess the efficacy of these treatments are sparse (King, 1998; Möller, 2007; Prikryl et al., 2007; Walther et al., 2009).

Facial expressivity only demonstrated a relationship with schizophrenia severity when captured using evoked prompts. This may indicate that either greater structure is needed to assess this marker remotely or that the prompts that were utilized were not a strong enough elicitation. Indeed, prior work has demonstrated that video rather than still images are stronger evocations to assess emotional variability in schizophrenia (Bersani et al., 2013). Despite this, we do observe that facial expressivity in response to evoked prompts provides a robust signal for overall symptom severity. Analysis of single symptoms demonstrates face validity for this marker as it reduced facial expressivity relates to greater blunted affect, social withdrawal, social avoidance, and motor retardation. We also observe that facial expressivity was decreased as subjects endorsed greater positive symptoms of hallucinations, delusions, and paranoid ideation. This finding is consistent with evidence that social and cognitive impairment, which falls into the negative symptom cluster, mechanistically relates to neurodegeneration that also impacts visual and auditory hallucinations (Bersani et al., 2013; Jenkins et al., 2018; Zhuo et al., 2020).

The current study presents with a number of important limitations. While the primary hypotheses were supported, not all effects were consistent across prompts. Given the small sample size, it is impossible to conclude definitively which markers can be utilized to robustly assess schizophrenia symptom severity or impairment. Indeed, a number of relatively large correlation coefficients demonstrated only marginal significance, likely due to sample size constraints. Further, despite the markers being hypothesized a priori, the current work is exploratory in nature given the small sample size, limited number of assessments, and the short duration of the study. A larger assessment will be needed to replicate the current findings. Despite the above limitations, the current work provides evidence that facial and vocal digital measures can be remotely captured in schizophrenia patients and that such measures demonstrate statistically significant relationships with established measures of schizophrenia symptom severity, demonstrating promise that these tools could be used to remotely measure and track schizophrenia symptoms and severity in an objective manner.

Finally, while the app-based video/audio capture utilizes a proprietary platform, this investigation utilized open-source Python-based software, available to all researchers (https://github.com/AiCure/open_dbm/). As an additional measure, the code that implemented the open-source software for this investigation and subsequent analyses of results have been provided by the authors in the Methods section. This allows for the expansion of the experiment to a wider patient population as mentioned above and the independent validation of the methods and their implementation in this investigation by other researchers in academic and clinical research, following an open-science framework for the development of digital tools for objective, accurate, and scalable measurement of disease symptomatology in both mental and physical health.

5 Conclusions

In this investigation, we demonstrate that facial and vocal markers, measured using computer vision and vocal analytics from video captured remotely via smartphones demonstrates validity as a marker of schizophrenia and is a promising metric for negative symptom severity. Use of such technology in clinical care and clinical research settings could allow for more frequent, remotely assessed, objective measurement of disease symptomatology and treatment response in a scalable and accessible manner, which can support development of novel treatments and risk assessment among individuals with schizophrenia.

Declaration of Interest

Authors IGL, AA, VY and VK were employed and own shares at AiCure, LLC at the time of the study. Authors OP, MD, MM, LS, and BH are employees of Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA and may own stock/stock options in Merck & Co., Inc., Kenilworth, NJ, USA.

Supplementary Materials