Smartphone Movement Sensors for Remote Monitoring of Respiratory Rate: Observational Study

Sophie Valentine; Benjamin Klasmer; Mohammad Dabbah; Marko Balabanovic; David Plans

doi:10.1101/2021.12.03.21267247

Abstract

Background Mobile health offers potential benefits to patients and healthcare systems alike. Existing remote technologies to measure respiratory rate (RR) have limitations, such as cost, accessibility and reliability. Using smartphone sensors to measure RR may offer a potential solution.

Objective The aim of this study was to conduct a comprehensive assessment of a novel mHealth smartphone application designed to measure RR using movement sensors.

Methods In Study 1, 15 participants simultaneously measured their RR with the app, and an FDA cleared reference device. A novel reference method to allow the app to be evaluated ‘in the wild’ was also developed. In Study 2, 165 participants measured their RR using the app, and these measures were compared to the novel reference. Usability of the app was also assessed in both studies.

Results The app, when compared to the FDA-cleared and novel references, respectively, showed a mean absolute error (MAE) of 1.65 (SD=1.49) and 1.14 (1.44), relative MAE of 12.2 (9.23) and 9.5 (18.70) and bias of 0.81 (limits of agreement (LoA) =-3.27-4.89) and 0.08 (−3.68-3.51). Pearson correlation coefficients were 0.700 and 0.885. 93% of participants successfully operated the app on their first use.

Conclusions The accuracy and usability of the app demonstrated here show promise for the use of mHealth solutions employing smartphone sensors to remotely monitor RR. Further research should validate the benefits that this technology may offer patients and healthcare systems.

Introduction

Extensive growth in the development and adoption of remote healthcare tools has been seen in recent years in response to increasing demand for traditional offerings. Notably, the COVID-19 global health pandemic has made salient how these mobile health (mHealth) tools may support healthcare systems to manage their patients when resources are pushed to breaking point.^1-3 As more widely accessible tools can be used by more people - and therefore offer greater impact - many mHealth smartphone applications (apps) have been developed, due to the high global penetration of smartphones. These systems offer a wide variety of services from telemedicine to remote monitoring and self-care, and evidence suggests they may produce improved economic⁴ and health outcomes.⁵

Respiratory rate (RR) is a fundamental indicator of health status for many health conditions, both general and specific to the respiratory system.^6-11 As such, mHealth solutions for monitoring of RR may offer significant value to patients and healthcare professionals (HCPs) alike. Although several such solutions exist, many fall short on various factors. Hardware-based solutions, including piezoelectric sensors¹² pulse oximeters,¹³ and multi-sensor devices,^14-15 are typically expensive, vulnerable to limited means of manufacture and distribution,¹⁶ and may lack interoperability with other health records, which is cited as a critical risk to decentralisation of national healthcare systems.¹⁷ Software-based solutions address limitations of cost, manufacture and distribution; however, they typically employ less-stable mechanisms of action. These mHealth apps often use smartphone cameras or microphones,^18-20 the latter of which have been evidenced to be vulnerable to environmental noise at the cost of accuracy and usability.^21-22

Movement sensors may present a promising alternative software-based solution for mHealth RR monitoring. Research indicates that multi-axial accelerometers and gyroscopes - as found ubiquitously in modern smartphones - can accurately capture RR based on chest movements.^23-30 Additionally, due to their mechanism of action, these sensors are significantly less affected by environmental noise. Overall, smartphone-based measurement of RR provides a potential low cost, and widely available method for RR measurement, both in a remote monitoring environment, or in locations where specialised hardware and software are not available.

This article presents an observational assessment of a novel user-centric mHealth smartphone app that measures RR using smartphone movement sensors. We first conducted a preliminary evaluation of the device and study methods via a small lab-based study, then jointly assessed accuracy and usability on a greater scale and ecological valid environment via a remote study. Ethical approval was provided by the University of Exeter’s Research Ethics Board (application ID eUEBS004088) and all research was conducted in compliance with the Declaration of Helsinki.

Study 1

Methods

The preliminary evaluation pursued three aims: (1) to establish the accuracy of the novel mHealth smartphone app relative to a reference device cleared by the US Food and Drug Administration (FDA), (2) to understand the usability of the mHealth app and (3) to evaluate the suitability of a novel reference method that would permit accuracy assessments to be conducted via remote and real-world studies. Through a prospective, non-interventional, non-randomised study conducted on healthy volunteers, RR estimates provided by the FDA-cleared reference device were compared to those from the novel mHealth smartphone app and the novel reference.

Measurements

Novel mHealth smartphone app

The mHealth app contained a purpose-built user interface (Figure 1). The user is instructed to hold their smartphone to their upper-middle chest with the screen facing outwards while sitting still and breathing normally for the duration of the 30-second sensor recording. Data is captured from the smartphone’s tri-axial gyroscope and interpolated to achieve an even 100Hz sample frequency. A low-pass Butterworth filter with 0.4 Hz cut-off is applied to remove high-frequency noise while retaining activity associated with breathing, typically in the 0.16-0.33Hz range (10-20 breaths per minute (BPM)). RR is calculated by performing an autocorrelation before normalising the resulting signal. A peak-finding routine then identifies prominent peaks corresponding to the cyclical property of breathing movements. The mean inter-peak interval (IPI) is then calculated and converted to a ‘per minute’ RR estimation by division by 60 (seconds) (Figure 2).

Figure 1.

Selected wireframes from the user interface of the mHealth app, depicting (from left to right) an option given to the user to view operation instructions in video or written format, an instruction for the user to hold their smartphone to their chest, a clinical safety feature allowing the user to retake a recording if they were disturbed while taking the original recording, and feedback given to the user if their recording fails the signal check.

Figure 2.

Graphical depiction of peak finding following application of an autocorrelation method. The grey line shows an example of correlation coefficients for a movement sensor (gyroscope) signal correlated with itself at progressive temporal shifts. Black crosses indicate prominent peaks corresponding to the cyclical property of breathing movements. IPIs are depicted by d_i.

An additional ‘signal check’ routine assesses whether the signal quality is sufficient to accurately derive RR, based on whether the number of autocorrelation peaks or standard deviation (SD) of the individual IPIs meet predetermined thresholds identified via preliminary bench-testing. If a recording fails the signal check, the user is informed via the app’s UI, redirected to the operation instructions and prompted to try again. Passing the signal check within three recordings attempts constitutes a successful use of the system, and three consecutive signal check failures constitute an unsuccessful use of the system, after which the user is instructed to seek support or try again later.

FDA-cleared reference

The MightySat Rx¹³, developed by Masimo Corporation, was selected as a reference due to its FDA-cleared status, continuous measurement and ease of use. The fingertip pulse oximeter derives RR using photoplethysmography (an optical measure of volumetric changes in peripheral blood flow). Continuous estimates of RR produced by this reference were converted to single weighted averages to facilitate comparison with data derived from the mHealth app.

Novel reference

The novel reference method involves identification of repeated cyclical peak-trough complexes within smartphone movement sensor signals(Figure 3). Signals of insufficient quality to derive RR are considered to fail the reference method. This method is conceptually similar to reference methods described in peer-reviewed literature reporting accuracy assessments of multiple RR devices, including successful FDA market clearance applications.^{13; 31-33} This method would permit accuracy assessments to be conducted via remote and real-world studies without a need for additional hardware, offering significant value in terms of research scale, cost and ecological validity via avoidance of observation bias.

Figure 3.

Graphical depiction of the novel reference method involving visual inspection of smartphone movement sensor signals by trained clinical and scientific researchers. The grey solid line shows an example of a smartphone movement sensor (gyroscope) signal, with black solid arrows depicting nine full repeated cyclical peak-trough complexes. The grey dotted line indicates the projected continuation of the movement sensor signal past the end of the recording period, with the grey dashed arrow indicating where a tenth full peak-trough complex would end. Hence, the movement sensor signal depicts a total of approximately 9.75 peak-trough complexes, with the final .75 peak-trough complex indicated by the black dashed arrow.

Participants and recruitment

Participants were recruited via convenience sampling. All were employees of the mHealth app manufacturer. Inclusion criteria included being aged 18 or over and being willing and able to follow the study protocol and complete an informed consent form.

Procedure

The study took place at the offices of the mHealth app manufacturer. Participants were provided with complete information concerning the study procedures and gave written informed consent to participate. The FDA-cleared reference device was applied to the forefinger of the participant’s left hand. Participants were provided with an iPhone XR with the mHealth app installed and received verbal instructions on operating the device: namely, to hold the smartphone to their upper middle chest with the screen facing outwards while sitting still and breathing normally during the 30-second recording. Participants were instructed to capture six recordings, disregarding whether each recording passed or failed the signal check. Audiovisual footage was captured during the study and used for offline synchronisation of data captured via the mHealth app and FDA-cleared reference. Specifically, this included sounds produced by the mHealth app indicating the start and end of the app’s recording period and depicting RR estimates displayed on the FDA-cleared reference’s monitor. Participation took around 10 minutes per participant.

Statistics

The error of the mHealth app and novel reference relative to the FDA-cleared reference was assessed through measures of mean absolute error (MAE), relative MAE and using the Bland-Altman method.³⁴ Due to the non-normal distribution of absolute error data, confidence intervals for MAE and relative MAE were derived via bootstrapping with replacement employing 1000-iterations and a sample size of 100%. The proportion of clinically significant errors, defined as an absolute error greater than three breaths per minute,^35-36 was also calculated. Direct relationships between RR estimates generated through the mHealth app, novel reference and FDA-cleared reference were assessed via Pearson Product Moment Correlation (PPMC). The usability of the mHealth app was assessed using the proportion and position of recordings that failed the signal check.

Results

Participants and data

15 participants took part in Study 1 (9 female), for whom 6 recordings each were collected for a total of 90. 26 (28%) mHealth app recordings failed the signal check and were excluded from analyses, resulting in a dataset of 64 paired samples. 29 (32%) of recordings failed the novel reference method, so were excluded from analyses, resulting in a dataset of 61 paired samples.

Accuracy

mHealth app versus FDA-cleared reference

Error results indicated an MAE of 1.65 BPM (SD = 1.49) with a 95% confidence interval (CI) of 1.32-2.06. Relative MAE was 12.2% (SD = 9.23) with 95% CI of 10.06 - 14.57. Bias (FDA-cleared reference - mHealth app) was 0.81 (SD = 2.08) with limits of agreement (LoA) of -3.27 - 4.89, indicating RR underestimation by the mHealth app. 8 comparisons (12.5%) had an absolute error greater than 3 BPM. A Bland-Altman plot indicated error values as a function of RR averaged between the reference and mHealth app (Figure 4). PPMC produced a coefficient of r(63) = 0.700, p < .000, indicating a high or strong association between the reference RR estimates and mHealth app RR estimate³⁷ (Figure 5).

Figure 4.

Bland-Altman plot for RR estimates provided by the mHealth app and FDA-cleared reference. The x-axis indicates RR estimates averaged between the mHealth app and FDA-cleared reference and the y-axis indicates the difference between RR estimates from each source (FDA-cleared reference - mHealth app). The solid horizontal line depicts a mean difference (bias) of 0 and dashed lines from top to bottom represent the upper limit of agreement (4.89), the observed mean difference (bias; 0.81), and the lower limit of agreement (−3.27). Marker size is proportional to the number of observations for each combination of values.

Figure 5.

Scatterplot for simultaneous RR estimates provided by the mHealth app (x-axis) and FDA-cleared reference (y-axis). The solid line indicates the gradient y=x. Marker size is proportional to the number of observations for each combination of values.

Novel reference versus FDA-cleared reference

Error results indicated that MAE was 1.69 BPM (SD = 1.61) with a 95% CI of 1.23 - 2.22. Relative MAE was 12.8% (SD = 11.60) with 95% CI of 9.96 - 15.64. Bias (FDA-cleared reference - novel reference) was 0.22 (SD = 2.34) with LoA of -4.36 - 4.79, indicating slight RR underestimation by the mHealth app. 9 comparisons (15%) had an absolute error greater than 3 BPM. A Bland-Altman plot indicated error values as a function of RR averaged between the FDA-cleared and novel references (Figure 6). PPMC produced a coefficient of r(59) = 0.701, p < .000, indicating a high or strong association³⁷ (Figure 7).

Figure 6.

Bland-Altman plot for RR estimates provided by the novel reference and FDA-cleared reference. The x-axis indicates RR estimates averaged between the novel and FDA-cleared references and the y-axis indicates the difference between RR estimates from each source (FDA-cleared reference - novel reference). The solid horizontal line depicts a mean difference (bias) of 0 and dashed lines from top to bottom represent the upper limit of agreement (4.79), the observed mean difference (bias; 0.22), and the lower limit of agreement (−4.36). Marker size is proportional to the number of observations for each combination of values.

Figure 7.

Scatterplot for simultaneous RR estimates provided by the novel reference (x-axis) and FDA-cleared reference (y-axis). The solid line indicates the gradient y=x. Marker size is proportional to the number of observations for each combination of values.

Usability

14 of a total of 15 participants (93.3%) were able to use the system successfully on their first try (Table 1; Figure 8). Specifically, this indicates that they could capture one or more recordings that passed the signal check within the first three attempts. All participants were able to use the system successfully by the end of their second try.

View this table:

Table 1.

Number and proportion of participants able to use the system successfully on consecutive attempts in Study 1 (n = 15).

Figure 8.

Line graph indicating the proportion of Study 1 participants who were able to generate a recording that passed the signal check on each of six consecutive recording attempts. The black dashed line indicates the proportion of individuals who were able to generate a recording that passed the signal check by the end of their first use of the system (three consecutive recordings), and the grey dashed line indicates the proportion of individuals who were able to do so by the end of their second use of the system.

Interim conclusion

Study 1 results indicated strong relationships between the FDA-cleared reference and both the mHealth app and the novel reference. Notably, these relationships were highly comparable to functional outcomes for alternative FDA-cleared RR monitoring devices.^{12, 14, 36, 38} Accordingly, these results supported both continued assessment of the mHealth app and application of the novel reference to accuracy analyses in Study 2, as described below.