Remote digital measurement of visual and auditory markers of Major Depressive Disorder severity and treatment response
======================================================================================================================

* Isaac R. Galatzer-Levy
* Anzar Abbas
* Vijay Yadav
* Vidya Koesmahargyo
* Allison Aghjayan
* Serena Marecki
* Miriam Evans
* Colin Sauder

## Abstract

**Objectives** Multiple machine learning-based visual and auditory digital markers have demonstrated associations between Major Depressive Disorder (MDD) status and severity. The current study examines if such measurements can quantify response to antidepressant treatment (ADT) with selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine uptake inhibitors (SNRIs).

**Methods** Visual and auditory markers were acquired through an automated smartphone task that measures facial, vocal, and head movement characteristics across four weeks of treatment (with timepoints at baseline, 2 weeks, and 4 weeks) on ADT *(n* = 12). The Montgomery-Asberg Depression Rating Scale (MADRS) was collected concordantly through clinical interviews to confirm diagnosis and assess changes in MDD severity.

**Results** Patient responses to ADT demonstrated clinically and statistically significant changes in the MADRS F(2,34) = 51.62, *p* <.0001. Additionally, patients demonstrated significant increases in multiple digital markers including facial expressivity, head movement, and amount of speech. Finally, patients demonstrated significant decreased frequency of fear and anger facial expressions.

**Conclusion** Digital markers associated with MDD demonstrate validity as measures of treatment response.

Keywords
*   digital biomarkers
*   digital phenotyping
*   major depressive disorder

## 1 Introduction

Patients with Major Depressive Disorder (MDD) are both heterogeneous in their clinical presentation and their response to antidepressant treatments (ADTs) (Lux and Kendler, 2010; Ostergaard et al., 2011). It is theorized that treatment effects may be obfuscated because MDD measurements combine heterogeneous symptoms that reflect distinct neurobiological and social processes while pharmacological treatments target specific neurobiological processes such as serotonergic tone. For example, patients with different subtypes of MDD, such as cognitive and neurovegetative phenotypes, have demonstrated differential treatment response to distinct classes of ADTs. (Jaracz et al., 2015; Uher et al., 2009) As such, there are significant efforts to refocus treatment research on measures that match the underlying neurobiological treatment target (Cuthbert and Insel, 2013). Disentangling the heterogeneity in MDD can lead to better risk and treatment response assessment by shifting the focus of investigation to narrow phenotypes that reflect the underlying neurological deficit and target of treatment (Cuthbert and Insel, 2013; Insel, 2014).

The use of digital measurements that relate to underlying biological phenotypes, termed digital phenotyping (Insel, 2017), has been proposed as a methodology to improve measurement of underlying illness by capturing digital proxy measures of clinical functioning. An example of digital phenotyping is the measurement of activity as a proxy measure of mood or anxiety states using actigraphy or geolocation captured from an individual’s smartphone (Onnela and Rauch, 2016; Torous et al., 2017). While novel measurements are promising, validation is required before such metrics can be interpreted clinically. Key steps to validation include comparison to traditional clinical measures, both cross-sectionally and as they change with the disease or treatment course (Coravos et al., 2019). Such measures should strive for ease of collection and increased sensitivity to facilitate frequent, accurate assessment and should be validated in relationship to narrower biological phenotypes and treatment targets than those that traditional endpoints assess. This will ultimately lead to improved, dynamic treatment research and clinical decision making (Torous et al., 2017) based on modulation of underlying neurobiological deficits (Lenze et al., 2020).

Based on prior knowledge, visual and auditory data sources represent a compelling direction for objective measurement of patient functioning in MDD. Beginning with observations by Emil Kraepelin, patients with depression have been shown to produce slowed and spaced out speech, where they appear to "become mute in the middle of a sentence" and demonstrate altered facial behavior, regarding which he states, “the facial expression and the general attitude are sleepy and languid” (Kraepelin, 1907). These clinical observations by Kraepelin have been corroborated and extended with standardized methods to assess facial expressions, vocal characteristics, and movement patterns using audio and video data sources. The same paucity of speech has been observed in acutely suicidal patients (Cummins et al., 2015). Indeed, both speech and facial/bodily movement represent sensitive biological outputs that change with physiological and cognitive variability (Cummins et al., 2015; Dagum, 2018; Jabbi and Keysers, 2008).

A number of visual and auditory characteristics that correspond to known MDD symptoms can now be directly quantified. This includes reduced gross motor activity (Buyukdura et al., 2011), slumped posture (Kim et al., 2018), reduced head movement variability (Alghowinem et al., 2013; Dibeklioglu et al., 2018; Kim et al., 2018), reduced facial expressivity (Girard et al., 2013), reduced speech production (Garcia-Toro et al., 2000), and increased negative affect (Berenbaum and Oltmanns, 2005; Ekman and Rosenberg, 2005). The automated measurement of these clinical features introduces the possibility of objective automated assessment. Given that audio and video data sources can be captured remotely, this further introduces the possibility of greatly scaling the reach and frequency of assessment. Increased scale and objectivity can facilitate increased accuracy and accessibility of clinical risk and treatment response assessments.

Serotonin signaling deficits represent a primary biological target for treatment in MDD. Serotonergic tone mechanistically impacts motor functioning directly through interactions with dopamine and norepinephrine signalling (El Mansari et al., 2010; Herrera-Guzmán et al., 2009; Morrissette and Stahl, 2014). Postmortem comparison of suicides compared to controls demonstrates significant reductions of brain serotonin (Stockmeier, 2003; Stockmeier et al., 1998). More specific mapping of mRNA expression patterns demonstrates reduced expression of Serotonin mRNA subtypes that are relatively widespread and other subtypes that are specific to the frontopolar cortex amygdala circuitry (Anisman et al., 2008). This circuitry governs the expression and regulation of threat and anxiety (Gold et al., 2015).

In the current study, we tested for changes in digital markers of (1) facial expressivity, (2) negative affect measured in facial expression, (3) pauses in speech, and (4) head movement activity. We hypothesized significant positive change in measures of negative affect and significant negative change in measures of motor functioning (such as percentage of audio with detected speech, facial expressivity, and head movement activity) in patients diagnosed with MDD over the first 4 weeks of ADT.

## 2. Methods

### 2.1 Study Participants

Participants were identified through advertisements posted on social media. Individuals who self-identified as experiencing depression were screened over the telephone to assess depression symptoms. Potentially eligible subjects were then scheduled for an in-person pre-screening visit with a clinician to assess primary eligibility criteria. Individuals who met criteria consented to participate in a screening assessment with a psychological rater, which included the Mini-International Neuropsychiatric Interview (MINI), Structured Interview Guide for the Montgomery-Asberg Depression Rating Scale (SIGMA-MADRS), Columbia Suicide Severity Rating Scale (C-SSRS), and the Quick Inventory of Depressive Symptomatology Self Report (QIDS-SR16).

To be included in the study, subjects had to meet DSM-5 criteria for single or recurrent MDD based on the MINI with a current major depressive episode of ≥ 8 weeks and a MADRS total score of ≥ 20. Participants must have also been, in the opinion of the study psychiatrist, medically stable and a good candidate for treatment with a monoamine ADT. Key exclusion criteria included significant medical complications (e.g., uncontrolled cardiac or endocrine disorders, diagnosis or treatment for cancer within the past 2 years), significant psychiatric complications (e.g., suicidal behavior that in the judgment of the investigator confers significant risk, other primary psychiatric diagnosis, substance use disorder), or the use of certain prohibited concomitant medications (e.g., prescription painkillers [opioids]).

Participants who met screening eligibility criteria subsequently completed a visit with a study psychiatrist and were prescribed an ADT consistent with standard of care. Participants who demonstrated significant decreases in depression severity, indicated by a 30% reduction in MADRS total score over 4 weeks of ADT, were included in the sample (n = 18). The sample included 7 men and 11 women (mean age = 30.2 ± 8.6), who met criteria for MDD. The mean BMI was 28.7 ± 5.6. Baseline total MADRS scores ranged from 25 to 45 (mean = 34.1 ± 4.9).

### 2.2 Treatment and Assessment Conditions

All patients were prescribed either an SSRI or SNRI at variable doses based on the clinician’s discretion. Treatment response was measured at bi-weekly intervals using two independent assessments described below.

### 2.3 Assessments

#### 2.3.1 Clinical Assessment

Montgomery-Asberg Depression Rating Scale (MADRS): The MADRS is a 10-item clinician administered scale for the measurement of MDD with validated clinical cut points for Severe (>34), Moderate (17-34), Mild (7-19), and Asymptomatic (< 7) depression. The MADRS has demonstrated validity as a sensitive measure of ADT response (Montgomery et al., 1985). The MADRS was administered by trained psychological raters with prior scale experience at Week 0 (Baseline) and at approximately 2 and 4 weeks post treatment initiation.

#### 2.3.2 Remote smartphone-based video assessments

All participants were asked to download the AiCure app (AiCure, LLC, New York, NY [www.aicure.com](http://www.aicure.com)) on their personal smartphone for measurement of digital markers of MDD. They were then trained by the study team on how to use the app to participate in remote assessments.

Participants completed remote assessments once a week for the length of the study. The assessment consisted of a smartphone-based adaptation of a paradigm to examine emotional valence in response to varied emotional imagery (Carretié et al., 2019; Stockmeier, 2003; Stockmeier et al., 1998). Specifically, at each assessment time point they were prompted to view images selected at random from the Open Affective Standardized Image Set (OASIS) from one of three emotional valence categories: positive valence, neutral valence, and negative valence. Participants were asked to observe each image and then describe what they saw and how it made them feel for a minimum of 10 seconds (Figure 1).

![Figure 1:](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2020/08/26/2020.08.24.20178004/F1.medium.gif)

[Figure 1:](http://medrxiv.org/content/early/2020/08/26/2020.08.24.20178004/F1)

Figure 1: 
Depiction of the smartphone-based assessment that all individuals completed. Video and audio of participant responses was recorded during the assessment and used to quantify behavioral characteristics and subsequently measure digital markers of MDD severity.

### 2.4 Digital marker calculation

Video and audio were captured continuously during the smartphone assessment using the smartphone front-facing camera and microphone. Data was uploaded and processed through HIPAA compliant backend services for transfer and storage of Protected Health Information (PHI). Video was extracted for analysis for the portion of the task where the participant is observing the image and responding to it. Both video and audio was extracted and analysed for the portion of the task when the participant was describing the image.

All analysis was conducted in a Python environment using open-source tools. All code used to calculate visual and auditory markers of MDD, the resulting data output, and the executable python script for the analysis of the data output presented in this manuscript is publicly available on Github: [https://github.com/AiCure/ms\_dbm\_adamsclinicalstudy](https://github.com/AiCure/ms_dbm_adamsclinicalstudy). A total of 18 variables inclusive of the MADRS, facial markers, voice markers, and movement markers were calculated (see Supplementary Table 1 for descriptive statistics for all variables).

#### 2.4.1 Facial marker calculation

First, all videos were segmented into individual video frames at 30 frames per second. Next, each frame was segmented into three matrices consisting of red, blue, green spectrum pixels for use in computer vision (CV) modeling using OpenCV, a open source computer vision software package (Brahmbhatt, 2013). Subsequently, each frame was analysed using OpenFace (Baltrusaitis et al., 2016), an open source software package that has demonstrated validity next to expert human ratings of Facial Action Coding System (FACS) (Ekman and Rosenberg, 2005), a standardized methodology to measure facial movements that reflect activity in the underlying human facial musculature used in the production of basic emotions (i.e., *happiness, fear, anger, surprise, sadness, disgust)*.

Specifically, for each frame OpenFace outputs **(1)** binary activation of each facial action unit (AU) utilized to calculate presence of facial emotions and **(2)** the degree of expressivity for that action unit utilized to calculate intensity of facial emotions. From AU measurements, emotion behavior was calculated including **(1)** the presence or absence of each emotion for each frame selected as the most probable based on the observed AU activation, termed ‘count’ and **(2)** the level of activation for each emotion and across all emotions, termed ‘intensity’. Following the calculation of these variables for each frame, a set of variables was calculated that represented the count of emotions expressed across across all frames divided by number of frames (*fear count, anger count, surprise count, sadness count, happy count, disgust count)* and the intensity of emotion averaged over all frames (fear intensity, anger intensity, surprise intensity, sadness intensity, surprise intensity, disgust intensity). Additionally, a composite score of overall facial intensity summed across all emotions was calculated *(overall facial expressivity)*.

#### 2.4.2 Voice marker calculation

Recordings were segmented into speech and non-speech parts using parselmouth, an open source software package that utilizes Praat software library (Weenink and Boersma, 2018) functions for vocal analysis (Jadoul et al., 2018). The ratio of speech to white space between words was calculated to represent the amount of time participants spoke compared to non-speech (*voice percentage*).

#### 2.4.3 Movement marker calculation

For each frame of video, head position and angle were acquired using OpenFace. The average framewise displacement of the head between frames *(head movement mean)* and its standard deviation *(head movement standard deviation)* were calculated as measures of head movement. The mean change in angle of the head(head *pose change mean)* was calculated as an additional measure of head movement.

### 2.5 Data Analysis

Change over time in MADRS and facial, voice, and movement variables (termed digital markers) was calculated using repeated measures analysis of variance (ANOVA). To avoid capitalizing on change, p-values were corrected using false discovery rate (FDR) correction (Li and Barber, 2019). The sphericity assumption was formally tested for each ANOVA. When this assumption proved to hold, the F-statistic and corresponding p-value was used. When the sphericity assumption was violated, Mauchly’s W statistic and corresponding p-value were used (Mendoza, 1980). Additionally, pairwise comparisons were calculated between each time point to determine where change across time points occurs (i.e., baseline to 2 weeks, baseline to 4 weeks, 2 weeks to 4 weeks) controlling for FDR using Tukey’s test.

## 3 Results

### 3.1 Depression Response

Participants demonstrated a main effect for change in MADRS scores from baseline to week 4 [F(2,34) = 51.62, *p* < .0001]. Descriptive statistics demonstrate clinically relevant change with patients moving from the clinical to non-clinical range (Supplementary Table 1; Figure 2).

![Figure 2:](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2020/08/26/2020.08.24.20178004/F2.medium.gif)

[Figure 2:](http://medrxiv.org/content/early/2020/08/26/2020.08.24.20178004/F2)

Figure 2: 
Response to treatment as measured by two independent assessments. Mean for each timepoint and standard error bars are shown. **(a)** MADRS scores acquired at baseline (BL), week 2, and week 4 showed significant decrease in response to ADT [F(2,34) = 51.62, p < .0001]. **(b)** Overall facial expressivity measured in response to positive, neutral, and negative images demonstrated a significant increase in response to ADT [F(2,28) = 39.27, p < .0001] as MADRS scores decreased. **(c)** Percentage of frames with voice measured in response to positive, neutral, and negative images also demonstrated a significant increase in response to ADT [F(2,26) = 5.77, p= 0.008] as MADRS scores decreased. All values in **b** and **c** were normalized between 0 and 1.

Participants demonstrated change in MDD severity as measured by digital markers. To align time points between digital markers and the MADRS scores, measurements from days 7-21 were averaged as the week 2 time point and measurements from days 22-35 were averaged as the week 4 time point. Due to missed remote assessments, a subset of the total sample of 18 had complete data across time points, with *n* = 12 for facial markers and *n* = 11 for voice markers. All statistical results for digital markers are presented in Table 1. Examples of marker profiles across treatment are presented in Figure 2 alongside the participants’ MADRS profile across treatment. All scores, including MADRS, were normalized to a range of 0-1 to allow visual comparison of the magnitude of change on digital markers in comparison to change in MADRS clinical scores (Figure 2).

View this table:
[Table 1:](http://medrxiv.org/content/early/2020/08/26/2020.08.24.20178004/T1)

Table 1: 
Repeated Measures ANOVA results for all facial and voice markers measured in response to positive, negative, and neutral visual stimuli from baseline to 4 weeks of ADT. Sph. indicates sphericity, F indicates F-statistic, P indicates p-value of F-statistic, and +/- indicates increased or decreased values.

### 3.2 Facial markers

All facial activity measures across all emotions (fear intensity, anger intensity, surprise intensity, sadness intensity, surprise intensity, disgust intensity and overall expressivity) along with the overall expressivity score demonstrated significant positive change from baseline to week 4 in response to all image prompts (positive, neutral, and negative; see Table 1). This result indicates that ADT produces a main effect on facial activity overall which is not bound to one particular facial musculature group or type of external stimulus (Figure 2).

Across conditions, the frequency of expressions of anger (anger count) decreases. The frequency of expressions of fear also decreases, but only in response to neutral and negative stimuli (fear count). Additionally, the frequency of expressions of happiness (happy count) decreases in response to negative stimuli only. Together, results indicate a general decrease in expressions of anger, and context specific decreases in fear and happiness expressions.

### 3.3 Voice markers

The single variable representing the ratio of speech to silence across sentences uttered (voice percentage) additionally demonstrated significant positive change in response to ADT across all conditions, indicating an increase in speech relative to silence. This result is consistent with increased motor/muscle activity observed in facial activity (Figure 2).

### 3.4 Movement markers

Additionally, movement parameters demonstrated consistent effects across conditions. The rate of head movement *(head movement mean)* and the degree of variability in the rate of head movement *(head movement standard deviation)* both demonstrated significant increases in response to ADT. *Head pose change mean* also demonstrated significant increase during neutral and positive stimuli.

## 4 Discussion

Results demonstrate a consistent effect of monoamine ADTs (SSRIs/SNRIs) on digital markers of motor functioning, which are highly concordant with change in MDD symptom severity. Specifically, facial and vocal activity demonstrated robust increases across 4 weeks following the initiation of treatment, which mirrored decreases in symptom severity as assessed by the clinician administered MADRS. The current findings suggest that SSRI/SNRI treatment, which produces graded increases in serotonin, reduces depression severity in part by rescuing motor functioning (e.g., increased facial expressivity, increased speech production).

Additionally, a decrease was observed across conditions in the expression of anger. Patients with depression have long demonstrated increased rates of anger compared to healthy counterparts (Koh et al., 2005; Riley et al., 1989). Further, polymorphisms of the Serotonin 1B receptor that are associated with increased depression and suicide risk are also associated with increased anger and fear (Hakulinen et al., 2013; McDevitt et al., 2011). These results further indicate that the observed change in digital markers in response to serotonin reuptake inhibitors reflects a more specific phenotypic change in measurement of serotonergic profile in the central nervous system.

Serotonin levels in the central nervous system are known to have both direct and indirect effects (via dopamine) on motor activity (Gerson and Baldessarini, 1980; Jacobs, 1991). Both suicide (as measured in post-mortem brain tissue) and suicidal attempts, a key symptom class of MDD, are associated with depleted serotonin (Mann and John Mann, 2013). As such, digital measurements that reflect motor behavior may represent a sensitive measure of serotonergic tone and potentially other neurotransmitter activity that affects motor functioning and ultimately the overall clinical presentation.

The current work presents a number of limitations that should be overcome through research that confirms and extends the findings reported. First, while treatment success was confirmed with clinical measures of MDD, dosage and treatment type were not controlled in a manner to make direct inferences about dose-response relationships. In addition, the current study was not adequately powered to assess the intra-subject variability in treatment response. Future research should provide more extensive experimental control of medication and dosage, and to assess the relationship between magnitude of clinical response and digital markers of motor activity. Next, while facial movement results were robust, we do not know if findings related to specific emotions would rise to significance given a larger sample size or more sampling occasions of the stimuli. One of the goals of the data collection was to implement a very simple remote assessment of objective visual and auditory markers to facilitate ease of frequent assessment. However, the minimum sample to accurately measure each marker needs to be assessed through the use of larger samples. For example, we observed decreases in happiness in response to negative images. This result is difficult to directly interpret. However, given a larger sample, we may be powered to identify increases in happiness in response to positive images, consistent with observations that depressed patients display context inappropriate affect (Coifman et al., 2007; Coifman and Bonanno, 2010).

Ultimately, the current work holds promise as an example of the potential to observe treatment effects that reflect underlying neurobiological target engagement by shifting the focus to monotonic neurobiologically based domains rather than heterogeneous diagnoses (Insel, 2014). Further work should determine if these same markers are relevant in other disorders and treatments that are mechanistically affected by serotonergic tone, as well as their relevance to other disorders with motor and movement profiles including Parkinson’s Disease and schizophrenia (Stahl, 2016). Second, the current work demonstrates the success of non-invasive objective digital assessment as a tool to assess treatment effects in Major Depressive Disorder. Importantly, no markers were scientifically novel, rather they were based on validated methods that are open and public and have been previously reported in scientific literature.

The current work demonstrates, in the context of MDD, that these data sources can be captured remotely through ubiquitously available digital tools to provide measurements that are at least as robust as traditional rating scales. It will be important to determine if such models reliably track with other disease states and treatment responses as such models and applications have significant potential to increase the rate and accuracy of treatment decision making.

Together, the current study demonstrates that scalability, through digital measurement, of monotonic characteristics that reflect underlying central nervous system activity. This observation holds promise that frequent remote digital assessment can be used to monitor, titrate, and even personalize treatment for MDD and other psychiatric or neurological conditions by grounding the measurements in narrow phenotypes that match the underlying mechanistic target of the treatment.

## Disclosures

Isaac R. Galatzer-Levy, Anzar Abbas, Vidya Koesmahargyo, and Vijay Yadav are employees of AiCure, LLC and hold stock options in AiCure, LLC. Colin Sauder is an employee of Karuna Therapeutics. Funding and staff resources for this study was jointly provided by AiCure, LLC and Adams Clinical.

## Data Availability

All code used to calculate visual and auditory markers of MDD, the resulting data output, and the executable python script for the analysis of the data output presented in this manuscript is publicly available on Github.

[https://github.com/AiCure/ms\_dbm\_adamsclinicalstudy](https://github.com/AiCure/ms_dbm_adamsclinicalstudy) 

## Supplementary Materials

View this table:
[Supplementary Table 1:](http://medrxiv.org/content/early/2020/08/26/2020.08.24.20178004/T2)

Supplementary Table 1: 
Means and standard errors across time points of each digital marker in response to neutral, positive and negative stimuli.

View this table:
[Supplementary Table 2:](http://medrxiv.org/content/early/2020/08/26/2020.08.24.20178004/T3)

Supplementary Table 2: 
Pairwise comparisons between time points for each digital marker in response to neutral, positive and negative stimuli.

*   Received August 24, 2020.
*   Revision received August 24, 2020.
*   Accepted August 26, 2020.


*   © 2020, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  Alghowinem, S., Goecke, R., Wagner, M., Parkerx, G., Breakspear, M., 2013. Head Pose and Movement Analysis as an Indicator of Depression. 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. [https://doi.org/10.1109/acii.2013.53](https://doi.org/10.1109/acii.2013.53)
    
    

2.  Anisman, H., Du, L., Palkovits, M., Faludi, G., Kovacs, G.G., Szontagh-Kishazi, P., Merali, Z., Poulter, M.O., 2008. Serotonin receptor subtype and p11 mRNA expression in stress-relevant brain regions of suicide and control subjects. J. Psychiatry Neurosci. 33, 131–141.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18330459&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000253761400006&link_type=ISI) 

3.  Baltrusaitis, T., Robinson, P., Morency, L.-P., 2016. OpenFace: An open source facial behavior analysis toolkit. 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). [https://doi.org/10.1109/wacv.2016.7477553](https://doi.org/10.1109/wacv.2016.7477553)
    
    

4.  Berenbaum, H., Oltmanns, T.F., 2005. Emotional Experience and Expression in Schizophrenia and Depression. What the Face RevealsBasic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). [https://doi.org/10.1093/acprof:oso/9780195179644.003.0021](https://doi.org/10.1093/acprof:oso/9780195179644.003.0021)
    
    

5.  Brahmbhatt, S., 2013. Introduction to Computer Vision and OpenCV. Practical OpenCV. [https://doi.org/10.1007/978-1-4302-6080-6\_1](https://doi.org/10.1007/978-1-4302-6080-6_1)
    
    

6.  Buyukdura, J.S., McClintock, S.M., Croarkin, P.E., 2011. Psychomotor retardation in depression: biological underpinnings, measurement, and treatment. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 395–409.
    
    

7.  Carretié, L., Tapia, M., López-Martín, S., Albert, J., 2019. EmoMadrid: An emotional pictures database for affect research. Motivation and Emotion. [https://doi.org/10.1007/s11031-019-09780-y](https://doi.org/10.1007/s11031-019-09780-y)
    
    

8.  Coifman, K.G., Bonanno, G.A., 2010. When distress does not become depression: Emotion context sensitivity and adjustment to bereavement. Journal of Abnormal Psychology. [https://doi.org/10.1037/a0020113](https://doi.org/10.1037/a0020113)
    
    

9.  Coifman, K.G., Bonanno, G.A., Ray, R.D., Gross, J.J., 2007. Does repressive coping promote resilience? Affective-autonomic response discrepancy during bereavement. Journal of Personality and Social Psychology. [https://doi.org/10.1037/0022-3514.92.4.745](https://doi.org/10.1037/0022-3514.92.4.745)
    
    

10. Coravos, A., Khozin, S., Mandl, K.D., 2019. Developing and adopting safe and effective digital biomarkers to improve patient outcomes. NPJ Digit Med 2. [https://doi.org/10.1038/s41746-019-0090-4](https://doi.org/10.1038/s41746-019-0090-4)
    
    

11. Cummins, N., Scherer, S., Krajewski, J., Schnieder, S., Epps, J., Quatieri, T.F., 2015. A review of depression and suicide risk assessment using speech analysis. Speech Communication. [https://doi.org/10.1016/j.specom.2015.03.004](https://doi.org/10.1016/j.specom.2015.03.004)
    
    

12. Cuthbert, B.N., Insel, T.R., 2013. Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med. 11, 126.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1741-7015-11-126&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23672542&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

13. Dagum, P., 2018. Digital biomarkers of cognitive function. npj Digital Medicine. [https://doi.org/10.1038/s41746-018-0018-4](https://doi.org/10.1038/s41746-018-0018-4)
    
    

14. Dibeklioglu, H., Hammal, Z., Cohn, J.F., 2018. Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding. IEEE J Biomed Health Inform 22, 525–536.
    
    

15. Ekman, P., Rosenberg, E.L., 2005. What the Face RevealsBasic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). [https://doi.org/10.1093/acprof:oso/9780195179644.001.0001](https://doi.org/10.1093/acprof:oso/9780195179644.001.0001)
    
    

16. El Mansari, M., Guiard, B.P., Chernoloz, O., Ghanbari, R., Katz, N., Blier, P., 2010. Relevance of norepinephrine-dopamine interactions in the treatment of major depressive disorder. CNS Neurosci. Ther. 16, e1–17.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1755-5949.2010.00146.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20406250&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000277403800001&link_type=ISI) 

17. Garcia-Toro, M., Talavera, J.A., Saiz-Ruiz, J., Gonzalez, A., 2000. Prosody impairment in depression measured through acoustic analysis. J. Nerv. Ment. Dis. 188, 824–829.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11191583&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

18. Gerson, S.C., Baldessarini, R.J., 1980. Motor effects of serotonin in the central nervous system. Life Sciences. [https://doi.org/10.1016/0024-3205(80)90368-9](https://doi.org/10.1016/0024-3205(80)90368-9)
    
    

19. Girard, J.M., Cohn, J.F., Mahoor, M.H., Mavadati, S., Rosenwald, D.P., 2013. Social Risk and Depression: Evidence from Manual and Automatic Facial Expression Analysis. Proc. Int. Conf. Autom. Face Gesture Recognit. 1–8.
    
    

20. Gold, A.L., Morey, R.A., McCarthy, G., 2015. Amygdala–Prefrontal Cortex Functional Connectivity During Threat-Induced Anxiety and Goal Distraction. Biological Psychiatry. [https://doi.org/10.1016/j.biopsych.2014.03.030](https://doi.org/10.1016/j.biopsych.2014.03.030)
    
    

21. Hakulinen, C., Jokela, M., Hintsanen, M., Merjonen, P., Pulkki-Råback, L., Seppälä, I., Lyytikäinen, L.-P., Lehtimäki, T., Kähönen, M., Viikari, J., Raitakari, O.T., Keltikangas-Järvinen, L., 2013. Serotonin receptor 1B genotype and hostility, anger and aggressive behavior through the lifespan: the Young Finns study. Journal of Behavioral Medicine. [https://doi.org/10.1007/s10865-012-9452-y](https://doi.org/10.1007/s10865-012-9452-y)
    
    

22. Herrera-Guzmán, I., Gudayol-Ferré, E., Herrera-Guzmán, D., Guàrdia-Olmos, J., Hinojosa-Calvo, E., Herrera-Abarca, J.E., 2009. Effects of selective serotonin reuptake and dual serotonergic-noradrenergic reuptake treatments on memory and mental processing speed in patients with major depressive disorder. Journal of Psychiatric Research. [https://doi.org/10.1016/jjpsychires.2008.10.015](https://doi.org/10.1016/jjpsychires.2008.10.015)
    
    

23. Insel, T.R., 2017. Digital Phenotyping: Technology for a New Science of Behavior. JAMA 318, 1215–1216.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2017.11295&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28973224&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

24. Insel, T.R., 2014. The NIMH Research Domain Criteria (RDoC) Project: precision medicine for psychiatry. Am. J. Psychiatry 171, 395–397.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1176/appi.ajp.2014.14020138&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24687194&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000333501000005&link_type=ISI) 

25. Jabbi, M., Keysers, C., 2008. Inferior frontal gyrus activity triggers anterior insula response to emotional facial expressions. Emotion 8, 775–780.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/a0014194&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19102588&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000261678100004&link_type=ISI) 

26. Jacobs, B.L., 1991. Serotonin and behavior: emphasis on motor control. J. Clin. Psychiatry 52 Suppl, 17–23.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=1988412&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1991EV49300007&link_type=ISI) 

27. Jadoul, Y., Thompson, B., de Boer, B., 2018. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics. [https://doi.org/10.1016/j.wocn.2018.07.001](https://doi.org/10.1016/j.wocn.2018.07.001)
    
    

28. Jaracz, J., Gattner, K., Moczko, J., Hauser, J., 2015. Comparison of the effects of escitalopram and nortriptyline on painful symptoms in patients with major depression. General Hospital Psychiatry. [https://doi.org/10.1016/j.genhosppsych.2014.10.005](https://doi.org/10.1016/j.genhosppsych.2014.10.005)
    
    

29. Kim, Y., Cheon, S.-M., Youm, C., Son, M., Kim, J.W., 2018. Depression and posture in patients with Parkinson’s disease. Gait & Posture. [https://doi.org/10.1016/j.gaitpost.2017.12.026](https://doi.org/10.1016/j.gaitpost.2017.12.026)
    
    

30. Koh, K.B., Kim, D.K., Kim, S.Y., Park, J.K., 2005. The relation between anger expression, depression, and somatic symptoms in depressive disorders and somatoform disorders. J. Clin. Psychiatry 66, 485–491.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15816791&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

31. Kraepelin, E., 1907. Clinical psychiatry.
    
    

32. Lenze, E.J., Rodebaugh, T.L., Nicol, G.E., 2020. A Framework for Advancing Precision Medicine in Clinical Trials for Mental Disorders. JAMA Psychiatry. [https://doi.org/10.1001/jamapsychiatry.2020.0114](https://doi.org/10.1001/jamapsychiatry.2020.0114)
    
    

33. Li, A., Barber, R.F., 2019. Multiple testing with the structure-adaptive Benjamini-Hochberg algorithm. Journal of the Royal Statistical Society: Series B (Statistical Methodology). [https://doi.org/10.1111/rssb.12298](https://doi.org/10.1111/rssb.12298)
    
    

34. Lux, V., Kendler, K.S., 2010. Deconstructing major depression: a validation study of the DSM-IV symptomatic criteria. Psychol. Med. 40, 1679–1690.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S0033291709992157&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20059797&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

35. Mann, J.J., John Mann, J., 2013. The serotonergic system in mood disorders and suicidal behaviour. Philosophical Transactions of the Royal Society B: Biological Sciences. [https://doi.org/10.1098/rstb.2012.0537](https://doi.org/10.1098/rstb.2012.0537)
    
    

36. McDevitt, R.A., Hiroi, R., Mackenzie, S.M., Robin, N.C., Cohn, A., Kim, J.J., Neumaier, J.F., 2011. Serotonin 1B autoreceptors originating in the caudal dorsal raphe nucleus reduce expression of fear and depression-like behavior. Biol. Psychiatry 69, 780–787.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.biopsych.2010.12.029&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21353664&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000288852800016&link_type=ISI) 

37. Mendoza, J.L., 1980. A significance test for multisample sphericity. Psychometrika. [https://doi.org/10.1007/bf02293611](https://doi.org/10.1007/bf02293611)
    
    

38. Montgomery, S.A., Smeyatsky, N., de Ruiter, M., Montgomery, D.B., 1985. Profiles of antidepressant activity with the Montgomery-Asberg Depression Rating Scale. Acta Psychiatr. Scand. Suppl. 320, 38–42.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=2931947&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

39. Morrissette, D.A., Stahl, S.M., 2014. Modulating the serotonin system in the treatment of major depressive disorder. CNS Spectr. 19 Suppl 1, 57–67; quiz 54-7, 68.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25544378&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 

40. &na;, &NA;, 2011. Praat: Doing Phonetics by Computer. Ear and Hearing. [https://doi.org/10.1097/aud.0b013e31821473f7](https://doi.org/10.1097/aud.0b013e31821473f7)
    
    

41. Onnela, J.-P., Rauch, S.L., 2016. Harnessing Smartphone-Based Digital Phenotyping to Enhance Behavioral and Mental Health. Neuropsychopharmacology 41, 1691–1696.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/npp.2016.7&link_type=DOI) 

42. Ostergaard, S.D., Jensen, S.O.W., Bech, P., 2011. The heterogeneity of the depressive syndrome: when numbers get serious. Acta Psychiatr. Scand. 124, 495–496.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1600-0447.2011.01744.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21838736&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000297004300010&link_type=ISI) 

43. Riley, W.T., Treiber, F.A., Woods, M.G., 1989. Anger and hostility in depression. J. Nerv. Ment. Dis. 177, 668–674.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/00005053-198911000-00002&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=2809577&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1989AY51100002&link_type=ISI) 

44. Stahl, S.M., 2016. Parkinson’s disease psychosis as a serotonin-dopamine imbalance syndrome. CNS Spectr. 21, 355–359.
    
    

45. Stockmeier, C.A., 2003. Involvement of serotonin in depression: evidence from postmortem and imaging studies of serotonin receptors and the serotonin transporter. J. Psychiatr. Res. 37, 357–373.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0022-3956(03)00050-5&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12849929&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F08%2F26%2F2020.08.24.20178004.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000184238200002&link_type=ISI) 

46. Stockmeier, C.A., Shapiro, L.A., Dilley, G.E., Kolli, T.N., Friedman, L., Rajkowska, G., 1998. Increase in Serotonin-1A Autoreceptors in the Midbrain of Suicide Victims with Major Depression—Postmortem Evidence for Decreased Serotonin Activity. The Journal of Neuroscience. [https://doi.org/10.1523/jneurosci.18-18-07394.1998](https://doi.org/10.1523/jneurosci.18-18-07394.1998)
    
    

47. Torous, J., Onnela, J.-P., Keshavan, M., 2017. New dimensions and new tools to realize the potential of RDoC: digital phenotyping via smartphones and connected devices. Transl. Psychiatry 7, e1053.
    
    

48. Weenink, D., Boersma, P., 2018. Praat: doing phonetics by computer.