Implementation, adoption and perceptions of telemental health during the COVID-19 pandemic: a systematic review
===============================================================================================================

* Rebecca Appleton
* Julie Williams
* Norha Vera San Juan
* Justin J. Needle
* Merle Schlief
* Harriet Jordan
* Luke Sheridan Rains
* Lucy Goulding
* Monika Badhan
* Emily Roxburgh
* Phoebe Barnett
* Spyros Spyridonidis
* Magdalena Tomaskova
* Jiping Mo
* Jasmine Harju-Seppänen
* Zoë Haime
* Cecilia Casetta
* Alexandra Papamichail
* Brynmor Lloyd-Evans
* Alan Simpson
* Nick Sevdalis
* Fiona Gaughran
* Sonia Johnson

## Abstract

**Background** Early in 2020, mental health services had to rapidly shift from face-to-face models of care to delivering the majority of treatments remotely (by video or phone call or occasionally messaging) due to the COVID-19 pandemic. This resulted in several challenges for staff and patients, but also in benefits such as convenience or increased access for people with impaired mobility or in rural areas. There is a need to understand the extent and impacts of telemental health implementation, and barriers and facilitators to its effective and acceptable use. This is relevant both to future emergency adoption of telemental health, and to debates on its future use in routine mental health care.

**Objective** To investigate the adoption and impacts of telemental health approaches during the COVID-19 Pandemic, and facilitators and barriers to optimal implementation.

**Methods** Four databases (PubMed, PsycINFO, CINAHL and Web of Science) were searched for primary research relating to remote working, mental health care, and the COVID-19 pandemic. Preprint servers were also searched. Results of studies were synthesised using framework synthesis.

**Results** A total of 77 papers met our inclusion criteria. In most studies, the majority of contacts could be transferred to a remote form during the pandemic, and good acceptability to service users and clinicians tended to be reported, at least where the alternative to remote contacts was interrupting care. However, a range of impediments to dealing optimal care by this means were also identified.

**Conclusions** Implementation of telemental health allowed some continuing support to the majority of service users during the COVID-19 pandemic and has value in an emergency situation. However, not all service users can be reached by this means, and better evidence is now needed on long-term impacts on therapeutic relationships and quality of care, and on impacts on groups at risk of digital exclusion and how to mitigate these.

Keywords
*   telemental health
*   COVID-19
*   remote care
*   telemedicine
*   mental health
*   systematic review
*   implementation science

## Introduction

Since the onset of the COVID-19 pandemic in the first few months of 2020, most countries have experienced a severe disruption of mental health service delivery in its usual forms (1). Community-based outpatient, day and home treatment programmes, prevention and mental health promotion programmes, and services for specific age groups, such as older adults, children and adolescents and people with substance misuse problems, have been among those severely affected at a time of potentially increased demand due to the adverse mental health consequences of the pandemic (2, 3).

Mental health care providers around the world responded to the disruption of services in many ways, including the significant and widely documented shift to remote delivery of mental health services to replace in-person consultations (1, 4, 5). Telemental health, defined as “the provision of behavioral and/or mental health care services using technological modalities in lieu of, or in addition to, traditional face-to-face methods” (6), including video conferencing, telephone, email or text messaging, has been central to continuing assessment and support in the community. Additionally, technological innovations are helping to address isolation and service disruption in hospital and residential settings (4, 7).

Multiple research studies conducted both before and during the pandemic have reported evidence of the effectiveness of telemental health in reducing treatment gaps and improving access to care for a range of service users (8-10). Findings from studies, often of telemental health programmes established for purposes of research, have suggested that, overall, synchronous modalities such as video conferencing are comparable to face-to-face delivery in terms of quality of care, reliability of clinical assessments and treatment outcomes and adherence (11-15). Good levels of service user acceptance and satisfaction with telemental health services have also been reported (10). Successful adoption of telemental health has been described across a wide range of populations (adult, child and adolescent, older people, ethnic minorities), settings (hospital, primary care, community) and conditions (11, 13, 16). For certain populations, including some with autism and severe anxiety disorders, and those with physical disabilities or geographical barriers to accessing services, telemental health can be preferable for some service users (6, 17), although individuals experiencing significant social disadvantage or severe mental health problems, such as psychosis, have been found to benefit less (18). Research suggests that telemental health can also work for group interventions (19). The attitudes of clinicians who have delivered care via synchronous telemental health appear to be largely positive, with professionals finding it both effective and acceptable (20) and recognising its potential to enhance communication within and between mental health teams (4, 7). There is also some positive health economic evidence, with several studies suggesting telemental health is no more expensive than face-to-face delivery and tends to be more cost-effective (12). This approach also appears to be a viable and inexpensive treatment option where access to emergency services is limited, and associations have been found with reduced psychiatric admissions (10).

However, despite this evidence base, integration of telemental health approaches into routine mental health care or the widespread adoption of remote working across whole systems has rarely been reported. Even during the pandemic, adoption of such technologies has been piecemeal, with utilisation varying substantially both between and within countries (1, 7). Barriers to the wider adoption of telemental health include: the risk of digital exclusion of some service users, such as those facing significant social disadvantage or with limited technological access and expertise; lack of technological infrastructure and clear protocols within services, impeding the integration of telemental health with face-to-face care; difficulty in establishing and maintaining therapeutic relationships and in conducting high quality assessments; service users who lack private space or find participating in sometimes intimate and distressing discussions from home intrusive (4, 11, 12, 18, 21-23). A range of other ethical, regulatory, technological, cultural and organisational barriers have also been identified, both before and during the pandemic (12, 24-28).

The widespread emergency adoption of telemental health since the onset of the pandemic has generated a substantial literature. Numerous commentaries, service evaluations and reports of telemental health innovations and service user, carer and staff experiences, and a growing number of research studies addressing effectiveness and implementation issues (29-32), have been published internationally. Clinical guidelines have been rapidly produced in a number of countries (33).

A synthesis of the relevant empirical evidence gathered during the pandemic is therefore timely, and informative for future planning by generating evidence of effects of adopting telemental health across whole populations and service systems rather than in the context of relatively small-scale research studies involving volunteer participants. Capturing the learning and experiences gained through the rapid shift to telemental health will help optimise remote healthcare in a population that presents unique relational challenges associated with mental distress. It will also help to understand and overcome implementation barriers and inform strategies for improving the flexibility, effectiveness, and efficiency of mental health services through the sustained integration into routine care of telemental health approaches, to ensure that it brings the greatest benefits for patients, carers, and staff.

The aim of this rapid review is to synthesise the international literature specific to remote working in mental health services (as a replacement for or in conjunction with face-to-face service delivery) in the context of the COVID-19 pandemic. Our research questions are:

1.  What evidence has been obtained during the COVID-19 pandemic regarding the effectiveness and cost-effectiveness of telemental health, and regarding its safety (including adverse events due to breaches of privacy and safety)?

2.  What coverage has been achieved through telemental health adoption in the pandemic (including extent of adoption by services and reach among clinical populations in which it is adopted); in which groups and for which service settings is telemental health more or less likely to be implemented successfully; what are potential risks associated with digital exclusion for those not reached; and what barriers and facilitators influence success in implementation?

3.  How acceptable are telemental health approaches to service users, carers and staff as applied during the pandemic, including perceived impacts on therapeutic relationships, communication and privacy?

4.  What innovations and improvements have been introduced to make clinical care via telemental health more effective and acceptable, achieve greater coverage, and address barriers to delivering care in this way? (This includes descriptions and evaluations of specific strategies designed to make telemental health work better than usual delivery, and of adaptations of telemental health to specific settings, such as inpatient wards and crisis services).

## Methods

A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (34).

The protocol for this review was registered with PROSPERO (CRD42021211025).

### Search strategy

Four electronic databases (PubMed, PsycINFO, CINAHL and Web of Science), preprint servers medRxiv, PsyArXiv, Wellcome Open Research, and JMIR were searched for research relating to COVID-19, mental illness, and remote working from 1st January – 9th December 2020. An example search strategy can be found in Supplementary Table 1.

This search was supplemented by searching the references listed in included studies for any additional studies that met our inclusion criteria.

### Screening

The resulting list of articles was de-duplicated using Endnote (35) and all references were imported into Rayyan (36) for title and abstract screening. Full texts were sourced for articles deemed relevant for inclusion, and these were screened against the full review eligibility criteria. To establish consistency in study selection, title and abstract screening was conducted by four reviewers (MS, ZH, JHS & LSR), with 100% of included and 25% of excluded references checked by another reviewer (RA). Full texts were screened by three reviewers (RA, MB & MS), with 100% of included and 25% of excluded papers checked by another reviewer (task divided between LG, HJ, JW, PB & LSR). Any disagreements were resolved through team discussion.

### Inclusion criteria

#### Participants

Staff working within the field of mental health, people receiving organised mental health care for any condition (including addictions, dementia and intellectual disability), family members or carers of people receiving mental health care (regarding their views on the impact of remote working on the service user, and interventions aimed at reducing carer distress).

#### Interventions

Any form of spoken or written communication carried out between mental health professionals or between mental health professionals and service users / family members / unpaid carers or peer support communications using the internet, the telephone, text messaging platforms or hybrid approaches combining different platforms.

#### Comparator(s)/control

Any mental health communication delivered face-to-face, digitally or remotely, waitlist control or placebo. Studies comparing different modes of delivery during the pandemic, and those comparing care delivery and outcomes during the pandemic with those before the pandemic were included. Relevant studies with no comparator were also included.

#### Outcomes

Qualitative and quantitative outcomes describing: implementation effectiveness (including process evaluations) and barriers and facilitators to digital engagement, clinical effectiveness, cost-effectiveness, acceptability (including service user, carer and staff satisfaction), impacts on communication and therapeutic relationships, coverage and impacts of digital exclusion, interventions to improve quality or coverage, improvements in quality of life, and economic impacts.

#### Design

Any papers that present qualitative or quantitative data from study designs of any type (including relevant service evaluations and case series). If the focus of the study was not solely remote working but the results section contained substantial data relevant to our research questions, these were also included. Any relevant reviews identified in the searches were checked for included research which met our inclusion criteria.

### Exclusion criteria

We excluded studies that were:

1.  not specific to the pandemic response.

2.  reporting on interventions with patients with primary sleep disorders.

3.  reporting on those with sub-clinical symptoms (unless combined with another included mental health problem).

4.  focused on digital interventions such as apps, websites, and virtual reality tools, except where the sole purpose of the digital intervention was to facilitate direct spoken or written communication.

5.  focused on interventions aimed at improving the mental health or wellbeing of healthcare professionals.

6.  editorials, opinion pieces, guidance documents, protocols, conference abstracts and letters, with the exception of editorials or letters which contained primary research findings.

No language or location restrictions were applied in this review.

### Data extraction

Data extraction was supported by well-established implementation science frameworks. A data extraction form was developed based on a brief version of the Consolidated Framework for Implementation Research (CFIR) (37) and the taxonomy of implementation outcomes (38). We used the higher level CFIR constructs (see Table 2 for a brief definition of each one of the implementation facets that CFIR constructs capture) to extract data on factors influencing implementation success (39), and the taxonomy of implementation outcomes including acceptability, adoption and feasibility. We also extracted information deemed relevant based on previous studies conducted by the research team, including an umbrella review of pre-COVID 19 systematic reviews on telemental health and a qualitative study (15, 21). Data extracted consisted of study details, including design and focus of study; gender, ethnicity, age; diagnosis of participants; details of staff occupation; setting and context of study; intervention details, implementation outcomes (including acceptability, adoption, appropriateness, feasibility, fidelity, cost effectiveness, penetration and sustainability), barriers and facilitators to implementation and clinical and safety outcomes. The full data extraction form can be viewed in Supplementary File 1.

View this table:
[Table 1.](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T1)

Table 1. 
Study Characteristics

View this table:
[Table 2.](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T2)

Table 2. 
Implementation barriers and drivers for telemental health grouped according to condensed CFIR domains (Consolidated Framework for Implementation Research)

Data extraction was completed by nine reviewers (approximately eight studies each) using EPPI-Reviewer 4 (40). All reviewers were trained on how to extract data to ensure consistency, and extracted data was checked by a second reviewer. The extraction form was first piloted on 10% of included studies to assess usability and content, with amendments made before completing extraction for the whole dataset.

### Quality appraisal

Given the diversity of the included article types and methods, two quality appraisal tools were used. Primary research studies were assessed with the Mixed Methods Appraisal Tool (MMAT) (41). Commentaries and service evaluations were assessed using AACODS, which appraises the veracity, clarity, acknowledgement of bias, and relevance of the contribution to the field (42). Study quality was assessed by RA and verified by NVSJ. Disagreements were resolved through discussion.

### Evidence synthesis

We conducted a framework synthesis of study characteristics and outputs. Study outcomes were tabulated applying existing implementation science frameworks, i.e., the CFIR framework (37), Proctor et al.’s (38) taxonomy of implementation outcomes; and also relevant topics/themes that emerged during data extraction. This table-based synthesis of the study outcomes combined a deductive and inductive approach to data analysis by using existing frameworks, while at the same time, identifying emerging themes. Results reported in this paper include a narrative synthesis of the study characteristics and quantitative study outputs (43) and the tabulated results.

## Results

### Study selection

A total of 3956 references were identified through searching databases of published papers. medRxiv was the only preprint database where included papers were found (n=10), one more relevant paper was identified by a member of the research team and a further paper was found through reference searching of included studies.

A PRISMA flow diagram (34) of the screening and selection process is presented in Figure 1.

![Figure 1:](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2021/07/06/2021.07.05.21260018/F1.medium.gif)

[Figure 1:](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/F1)

Figure 1: 
PRISMA diagram showing screening and included studies

### Quality of included studies

The quality of included primary research studies was moderate to high: 23 out of 48 studies appraised using the MMAT met above 80% of quality criteria, whilst 21 out of 48 met between 50 and 79%. The quality of included service evaluations or audits were generally high: 27 out of 29 studies appraised using the AACODS met at least four out of the six quality criteria assessed. These include: being written by recognised experts, including reference lists, having a clear aim, stating details such as date, location and limitations, and making a meaningful contribution to the research literature.

### Study characteristics

Of the 77 studies which were eligible for inclusion in the review, 45 were primary research studies, 24 were service evaluations or audits and eight were editorials or letters that included data. Thirty-three were conducted in the USA, nine in the UK, and five each in Australia, Canada and Spain. Five were conducted across more than one country.

Of the 45 primary research studies, 32 involved staff and 9 involved service users. The remaining four analysed service use data (three evaluated contacts with hotlines and one evaluated service use in one UK National Health Service (NHS) service provider).

Most studies were conducted in services that worked with people with mixed psychiatric diagnoses (n=30), although we also found studies conducted with groups with a single diagnosis, e.g. dementia or eating disorders. Studies could include more than one service type, the most commonly studied being community mental health teams (CMHTs) or outpatient settings (n=39), followed by psychology or psychotherapy services (n=17). Inpatient or residential services were included in 15 studies, whilst 12 included general hospitals. Eight studies included private hospitals or clinics, whilst four explored telemental health use in helplines, voluntary sector services, crisis teams, or veterans’ health services respectively. Five studies did not report any specific setting.

The aims of most studies were either a description of changes made due to the pandemic, new services set up because of the pandemic, or an evaluation of the impact of the pandemic on either staff or service users. The descriptions of changes either focused specifically on the move to telemental health or were wider descriptions of changes to services including the use of telemental health. The characteristics of each of the included studies are shown in Table 1, with a more detailed summary in Supplementary Table 2.

### Data synthesis

#### Barriers and facilitators to telemental health

Implementation barriers and facilitators were categorised using a condensed version of the CFIR framework (see Table 2, where definitions of the CFIR constructs are also provided). The key findings are summarised below.

#### Intervention characteristics

Video and phone calls were the most common modalities used for remote care; studies also reported the use of emails, instant messaging services, apps, pre-recorded videos and forums (further details about the modality used in each study can be found in Table 1).

When comparing remote care to traditional face-to-face settings, studies identified advantages for both methods. Benefits for remote care included being more convenient (for both staff and service users), making care more accessible to groups who may previously have been excluded, reducing travel (resulting in both time and cost savings), and helping clinicians understand more about the service user, as they had more insight into their home lives. A further benefit is that more family members were readily able to attend family therapy or family education sessions since care was moved online e.g. (44). However, clinicians reported difficulties in picking up on non-verbal cues in remote compared to face-to-face care, and that remote care could sometimes require more concentration.

#### Outer Setting

Services commonly implemented remote methods of working due to ‘stay at home’ orders or national lockdowns, or due to a high level of cases in their local area. Overall, all settings described in papers had sufficient capacity to make a rapid shift to remote forms of care. Several studies in the USA in particular mentioned the impact of health insurance regarding uptake of telemental health e.g. (57), as not all insurance providers covered remote care. However, this did change during the course of the pandemic as telemental health delivery was eased by the relaxation of policy and billing reimbursements (57, 76). The change from face-to-face to remote delivery of care was also facilitated by professional societies who posted guidelines on their websites to assist clinicians.

#### Staff Characteristics

Enablers for clinician uptake included supporting clinicians by ensuring supervision, supportive leadership, clear communication, keeping track of clinicians’ needs, optimising physical space for comfort and privacy (for example using headphones or ergonomic seating), and arranging times away from the computer.

However, staff in several studies reported a lack of initial training for telemental health, and therefore identified training needs regarding the use of online platforms and meeting privacy regulations in particular. In some studies, having no previous experience with telemental health was also found to be associated with higher anxiety (44) and lower uptake (53) of remote care. However, others found previous experience did not impact clinicians’ views of telemental health during the pandemic (48).

#### Process

As telemental health was not commonly used in most services pre-COVID, staff had to rapidly adjust to a new way of working. Several studies discussed the training which was put in place for staff, which included training courses, shadowing or observing senior colleagues, discussion within clinical teams, facility-level telehealth coordinators and clinical champions providing training, webinars, and checking official guidelines. New workflows also had to be developed to allow staff to access patient records remotely, and service users had to be informed about the transition to telemental health.

#### Service user needs/resources

In addition to the needs of staff, service users also identified certain needs and resources to enable them to effectively transition to telemental health care. A commonly reported issue was access to technology, particularly amongst service users with diagnoses such as schizophrenia e.g. (117), older adults e.g. (51), and service users from lower socio-economic backgrounds e.g. (79). Service users also reported problems having a stable internet connection to allow for uninterrupted communication, which could negatively impact the therapeutic relationship. Concerns were also raised by both clinicians and service users regarding privacy and confidentiality, and in some cases service users had difficulties concentrating on remote care. Several studies e.g. (87, 99), mention the need for an agreed ‘Zoom etiquette’ for service users, including attire, audio/visual set-up and background distractions.

### Implementation outcomes

Outcomes of the implementation of telemental health have been summarised below using Proctor et al.’s (38) taxonomy of implementation outcomes. Further information can be found in Table 5.

#### Acceptability

Remote care was seen as satisfactory by the majority of clinicians and service users in most studies in the context of the pandemic. A number of studies also reported that telemental health enabled some groups to access care who found it difficult to engage with face-to-face support e.g. (7). Some clinicians reported that they would also be willing to continue with some aspects of remote care in the future e.g. (44, 78). However, it is important to note that whilst acceptability was high overall, this was not the case for all groups e.g. Grover et al (72) reported acceptability rates of around 45% for both clinicians and service users using services in a range of settings in India. Further details of satisfaction and acceptability outcomes are shown in Table 3.

View this table:
[Table 3:](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T3)

Table 3: 
Levels of Acceptability of telemental health during the COVID-19 pandemic

Telemental health services were acceptable to people during the pandemic as a way of continuing their treatment, however findings from several studies also indicated that participants wanted at least some appointments to be face-to-face once restrictions on in-person contact had loosened.

#### Adoption

Adoption rates were relatively high across studies, with most services or clinicians moving their appointments to remote methods. Rates of adoption of telemental health for service users who were already receiving care at the start of the pandemic ranged from 48% (90) to 100% (45, 68, 111). Some studies reported face-to-face appointments still took place if necessary, for example for initial assessments or for medication reviews e.g. (117). Most studies which examined impact on attendance reported no adverse effects on attendance rates after introducing telemental health: there was either no difference in missed appointments when comparing remote to face-to-face care (46, 87), or non-attendance after adoption of telemental health decreased (80, 92, 104). Further details about adoption of telemental health across studies can be found in Table 4.

View this table:
[Table 4:](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T4)

Table 4: 
Levels of Adoption and Coverage of telemental health during the COVID-19 pandemic

View this table:
[Table 5.](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T5)

Table 5. 
Implementation outcomes summary findings for telemental health

Whilst most studies reported high adoption rates, a few studies reported a decrease in attendance: for example, Erekson et al (62) (though possibly because of the university setting) and Dores et al (61) who identified challenges in retention due to: low client adherence, lack of privacy, interruptions at home, lack of appropriate technology and/or simply preference for face-to-face contact.

There was also evidence to indicate that adoption rates of telemental health fell as COVID cases decreased e.g. (85, 86).

#### Appropriateness

There were some concerns raised over the appropriateness of remote care, for example studies reported difficulties managing medication e.g. (46, 109, 110) and concerns around engaging and assessing new patients e.g. (79). Clinicians also found it harder to assess some physical indicators of mental health status (e.g. hygiene, eye contact, physical symptoms of opioid withdrawal), without being able to see the service user in person. However, in contrast, remote methods of working felt safer for clinicians who worked with service users at risk for violence and behavioural dysregulation e.g. (57). Online care was also not necessarily appropriate for patients with auditory or visual impairments, or with conditions such as migraines.

Staff reported concerns around the management of risk and safeguarding of service users when using remote methods of care e.g. (80, 94). Some helpful features of platforms which were thought to improve safety were using the waiting room function, being able to remove call participants, renaming participants (to protect anonymity) and using the private chat function.

#### Feasibility

In general, all studies reported good feasibility, at least for the short-term emergency response during the pandemic. However, some studies reported that telemental health is not suitable for all types of therapy, for example those which require a physical presence (exposure therapy, role play, collaborative models) e.g. (107). Telemental health may also be less suitable for treating trauma (63, 103), for clients with severe anxiety (63), learning difficulties or autism (106), children (63), and clients with cognitive impairment (63, 91).

#### Cost-effectiveness

There was limited information about costs of implementation of remote care in the included studies and no actual costs of telemental health were reported in the papers. However, initial evidence suggests remote care is not a costly intervention, with one paper stating that telemental health is “cost-effective” (70), whilst another mentions the use of “low-cost technologies” used by clinicians (73).

#### Penetration

There was widespread penetration (the extent to which telemental health was integrated into mental health services) of remote methods of care delivery due to the COVID-19 pandemic, despite few services utilising telemental health previously. Services were able to rapidly adapt to this new way of working, with the majority of appointments conducted remotely after the first few weeks of ‘stay at home’ orders.

#### Sustainability

The sustainability of telemental health cannot be completely determined from the included studies, as they present data mostly from the early stages of the COVID-19 pandemic. However, there was some indication that although remote working was widely accepted as a necessity, once restrictions loosened, rates of telemental health use declined (corresponding with the drop in cases in Europe in summer 2020). This correlates with findings that not all staff and service users would want to continue using remote methods of care after the pandemic ends. However, there are some aspects of remote working that both clinicians and service users would like to keep in the future in combination with face-to-face care e.g. (94, 105), as this approach has benefits such as being more efficient, flexible, and enabling access for certain groups e.g. (7, 61, 63).

### Clinical outcomes

Comparing the clinical outcomes of face-to-face and remote care using quantitative measures indicated that telemental health approaches could be as effective as face-to-face care e.g. (56, 104), although it should be noted that most studies were on a small scale. Several studies also reported no psychiatric decompensations after switching to remote care e.g. (87, 89). However, it is important to note that clinical outcomes for telemental health were not comparable for all service users, for example Dores et al (61) found a quarter of psychiatrists reported poorer clinical outcomes after switching to remote care. Another study also indicated that only a third of clinicians felt as though telemental health consultations were comparable to pre-pandemic sessions (115). A full presentation of the clinical outcomes reported in included studies is shown in Table 6.

View this table:
[Table 6:](http://medrxiv.org/content/early/2021/07/06/2021.07.05.21260018/T6)

Table 6: 
Studies which reported clinical outcomes of telemental health

Although the quality of therapeutic relationships reported by studies was generally good, clinicians reported problems reading patients’ emotions e.g. (44) or feeling less connected to the service user compared to face-to-face sessions e.g. (57). Clinicians also reported difficulties regarding feeling and expressing empathy remotely. Other challenges to therapeutic relationships when using remote care included a lack of client engagement, possible misunderstandings due to lack of non-verbal signals, common context or not having a clear idea of patients’ physical state (alongside reduced privacy).

### Social outcomes

One study (82) compared social outcomes in a trial comparing telephone-only care to caregivers of older adults with neurocognitive disorder, with supplementary video care to both carers and service users. Findings indicated that those who received both telephone and video support had greater resilience, better cognitive functioning and a higher quality of life.

### Organisational and care delivery outcomes

Improved communication was noted between staff when using telemental health when compared to traditional face-to-face care, as the use of online file sharing or discussion platforms facilitated communication between staff e.g. (75, 106). The use of online methods also facilitated staff training and some staff reported that remote working resulted in a better work-life balance e.g. (60).

## Discussion

### Summary of findings

This review collated evidence regarding the implementation and outcomes of remote working in mental health services in the context of the COVID-19 pandemic. Most studies indicate a relatively high level of activity, suggesting that at least in the services studied in higher income countries, mental health care can be shifted to telemental health in a crisis. Services mainly reported using a mixture of phone and video calls, with both service users and clinicians varying in their preference for these modalities. There were some indicators of reduced numbers of missed appointments, potentially due to the greater convenience of remote care, which may make access to services easier for some service users.

There was reasonable acceptability across the studies, at least in conditions where the alternative may have been no contact with services at all. However, there were situations, where telemental health may be less acceptable, including for new patients, physical health aspects of care, and for service users without a private space at home to use for therapy. Telemental health also may not be as feasible for certain types of support, including support which needs a physical presence. There was also evidence that telemental health may not be feasible for some clinical presentations, including some service users with psychosis, learning difficulties or autism. Clinicians also reported a decrease in their ability to develop and maintain a strong therapeutic relationship with service users, due to being unable to pick up on non-verbal cues and a lack of connectedness. The acceptability levels found in this study are not dissimilar to previous studies e.g. (15), even though the participants in the current studies are less likely to be volunteering to pilot a new service and more likely to be using telemental health because they have no alternative.

Few formal investigations of how to improve implementation were identified in this review. However, some strategies for improving adoption/penetration/acceptability may include staff training, the use of telemental health champions, strategies for introducing service users to technology and providing some simple guidance on how to use it best, identifying situations or populations when telemental health is not a good idea and those where it might be better. There was also a lack of fidelity assessments when therapies had to be adapted to fit telemental health delivery formats, therefore little is known about the consequences of these adaptations.

Our interpretation of this pattern of findings is that the successful delivery in a pandemic of telemental health should not necessarily be seen as confirmation that people are happy with this mode of delivery long-term, as some of the identified problems may become more serious over time, and reports of being satisfied may have reflected awareness that at the time of the study, it was difficult to offer care by any other means. The longevity of these changes will ultimately turn not only on information technology, safety, and quality, but also on whether policy changes will support the reimbursements and regulatory adjustments implemented during the current crisis (30, 58).

### Implications for future research

There was a lack of reporting in included studies of trying to identify and reach those patients who are at increased risk of digital exclusion. The needs of those at risk of digital exclusion are still largely underreported in the literature and should be made a priority for future research. Studies also included little information regarding the cost-effectiveness of telemental health implementation. Further research is needed to explore the differences in cost (both to the service and service user) between face-to-face and telemental health care. Further research can also formally compare (rather than simply observe) different delivery support strategies that can improve the implementation and potentially also the clinical effectiveness of telemental health, including for specific conditions and service user groups.

Finally, there is scope to conduct big data studies to identify who is not accessing remote care or those at risk of disengaging, and potential comparisons for matched groups to try to compare effectiveness across a range of settings, as this could be done more quickly than in clinical trials whilst respecting patient preference.

### Strengths and limitations

The studies included in this review identified outcomes across different settings and healthcare systems, which may help findings generalise to different settings. This review also captured recent findings on the use of telemental health during the COVID-19 pandemic, allowing findings to be used to improve both existing and future models of remote mental health care.

However, it is also important to take some limitations into account when interpreting the findings from this review. Firstly, the results from quality assessment indicated that whilst around half of primary research studies and the majority of the service evaluations were high quality, around half of primary research studies were scored as moderate to low. This reflects the short nature of studies and often quick turnaround from data collection to publication. Some studies were also published in preprint form and therefore had not undergone peer review. The majority of studies used cross-sectional data, rather than more rigorous methods. Secondly, there was a lack of high quality quantitative evidence for the clinical effectiveness of telemental health care. Clinical effectiveness outcomes were only reported in 9/77 included studies, with some of these findings only based on qualitative evidence or a small number of service users. It is also important to note that the voices of those who dropped out of care may not be included.

The short time scale for data collection and assessment of changes in practice in included studies could also be viewed as a limitation of this research, as it is not clear if changes will be sustained over time or in other contexts (e.g. lower-income countries). Research was also not inclusive of those not accessing or using remote technologies, meaning there is a risk of those at risk of digital exclusion being forgotten when taking the findings of this review into consideration.

We also designed this review to be conducted rapidly to ensure results would be relevant and quickly available, therefore we chose to search four databases and not all studies were independently double-screened. However, we are confident that the rigour of our searches and inclusion of preprint servers meant that the papers included are representative of the literature on this topic.

## Conclusion

Telemental health was a largely effective method to enable continuation of mental health support during the COVID-19 pandemic. Whilst most reported outcomes were positive, telemental health was not feasible for all types of support and may not be acceptable to all service user groups. A blended approach combining face-to-face and telemental health care may be the most desirable service model for future care. The need remains for higher-quality evidence regarding the clinical effectiveness of telemental health and how uptake can be improved for groups at risk of digital exclusion.

## Supporting information

Supplementary Table 1 [[supplements/260018_file02.docx]](pending:yes)

Supplementary Table 2 [[supplements/260018_file03.docx]](pending:yes)

## Data Availability

N/A - systematic review

## Policy Research Unit

This paper presents independent research commissioned and funded by the National Institute for Health Research (NIHR) Policy Research Programme, conducted by the NIHR Policy Research Unit (PRU) in Mental Health. The views expressed are those of the authors and not necessarily those of the NIHR, the Department of Health and Social Care or its arm’s length bodies, or other government departments.

## Applied Research Collaboration South London and King’s Improvement Science

This study is supported by the National Institute for Health Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. NS and LG are members of King’s Improvement Science, which offers co-funding to the NIHR ARC South London and is funded by King’s Health Partners (Guy’s and St Thomas’ NHS Foundation Trust, King’s College Hospital NHS Foundation Trust, King’s College London and South London and Maudsley NHS Foundation Trust), and Guy’s and St Thomas’ Foundation. The views expressed are those of the author(s) and not necessarily those of the NIHR, the NHS, the charities, or the Department of Health and Social Care.

LG is funded by King’s Improvement Science, which is part of the NIHR ARC South London and comprises a specialist team of improvement scientists and senior researchers based at King’s College London. King’s Improvement Science is funded by King’s Health Partners (Guy’s and St Thomas’ NHS Foundation Trust, King’s College Hospital NHS Foundation Trust, King’s College London and South London and Maudsley NHS Foundation Trust), and the Guy’s and St Thomas’ Foundation.

## Conflicts of Interest

NS is the director of the London Safety and Training Solutions Ltd, which offers training in patient safety, implementation solutions and human factors to healthcare organisations and the pharmaceutical industry. The other authors have no conflicts of interest to declare.

## Acknowledgements

*   Received July 5, 2021.
*   Revision received July 5, 2021.
*   Accepted July 6, 2021.


*   © 2021, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/)

## References

1.  1.World Health Organization. The impact of COVID-19 on mental, neurological and substance use services: results of a rapid assessment. 2020.
    
    

2.  2.Pfefferbaum B, North CS. Mental health and the Covid-19 pandemic. New England Journal of Medicine. 2020;383(6):510–2.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F06%2F2021.07.05.21260018.atom) 

3.  3.Rajkumar RP. COVID-19 and mental health: A review of the existing literature. Asian journal of psychiatry. 2020;52:102066.
    
    

4.  4.Rains LS, Johnson S, Barnett P, Steare T, Needle JJ, Carr S, et al. Early impacts of the COVID-19 pandemic on mental health care and on people with mental health conditions: framework synthesis of international experiences and responses. Social psychiatry and psychiatric epidemiology. 2021;56(1):13–24.
    
    

5.  5.Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. The Lancet Psychiatry. 2020.
    
    

6.  6.Association AP. What are telehealth and telepsychology. 2017.
    
    

7.  7.Johnson S, Dalton-Locke C, San Juan NV, Foye U, Oram S, Papamichail A, et al. Impact on mental health care and on mental health service users of the COVID-19 pandemic: a mixed methods survey of UK mental health care staff. Social psychiatry and psychiatric epidemiology. 2021;56(1):25– 37.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F06%2F2021.07.05.21260018.atom) 

8.  8.Telebehavioural Health Institute. Telehealth bibliography..
    
    

9.  9.Bashshur RL, Shannon GW, Bashshur N, Yellowlees PM. The empirical evidence for telemedicine interventions in mental disorders. Telemedicine and e-Health. 2016;22(2):87–113.
    
    

10. 10.Salmoiraghi A, Hussain S. A systematic review of the use of telepsychiatry in acute settings. Journal of Psychiatric Practice®. 2015;21(5):389–93.
    
    

11. 11.Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemedicine and e-Health. 2013;19(6):444–54.
    
    

12. 12.Hubley S, Lynch SB, Schneck C, Thomas M, Shore J. Review of key telepsychiatry outcomes. World journal of psychiatry. 2016;6(2):269.
    
    

13. 13.Varker T, Brand RM, Ward J, Terhaag S, Phelps A. Efficacy of synchronous telepsychology interventions for people with anxiety, depression, posttraumatic stress disorder, and adjustment disorder: A rapid evidence assessment. Psychological Services. 2019;16(4):621.
    
    

14. 14.Fletcher TL, Hogan JB, Keegan F, Davis ML, Wassef M, Day S, et al. Recent advances in delivering mental health treatment via video to home. Current psychiatry reports. 2018;20(8):1–9.
    
    

15. 15.Barnett P, Goulding L, Casetta C, Jordan H, Rains LS, Steare T, et al. Remote working in mental health services: a rapid umbrella review of pre-COVID-19 literature. medRxiv. 2020.
    
    

16. 16.Gloff NE, LeNoue SR, Novins DK, Myers K. Telemental health for children and adolescents. International Review of Psychiatry. 2015;27(6):513–24.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3109/09540261.2015.1086322&link_type=DOI) 

17. 17.Seritan AL, Heiry M, Iosif A-M, Dodge M, Ostrem JL. Telepsychiatry for patients with movement disorders: a feasibility and patient satisfaction study. Journal of clinical movement disorders. 2019;6(1):1–8.
    
    

18. 18.Santesteban-Echarri O, Piskulic D, Nyman RK, Addington J. Telehealth interventions for schizophrenia-spectrum disorders and clinical high-risk for psychosis individuals: A scoping review. Journal of telemedicine and telecare. 2020;26(1-2):14–20.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1357633X18794100&link_type=DOI) 

19. 19.Gentry MT, Lapid MI, Clark MM, Rummans TA. Evidence for telehealth group-based treatment: A systematic review. Journal of telemedicine and telecare. 2019;25(6):327–42.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1357633X18775855&link_type=DOI) 

20. 20.Connolly SL, Miller CJ, Lindsay JA, Bauer MS. A systematic review of providers’ attitudes toward telemental health via videoconferencing. Clinical Psychology: Science and Practice. 2020;27(2):e12311.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/cpsp.12311&link_type=DOI) 

21. 21.San Juan NV, Shah P, Schlief M, Appleton R, Nyikavaranda P, Birken M, et al. Service user experiences and views regarding telemental health during the COVID-19 pandemic: a co-produced framework analysis. medRxiv. 2021.
    
    

22. 22.Stoll J, Müller JA, Trachsel M. Ethical issues in online psychotherapy: A narrative review. Frontiers in psychiatry. 2020;10:993.
    
    

23. 23.Tullio V, Perrone G, Bilotta C, Lanzarone A, Argo A. Psychological support and psychotherapy via digital devices in Covid-19 emergency time: Some critical issues. Medico-Legal Journal. 2020;88(2):73–6.
    
    

24. 24.Europe Economics. Regulatory approaches to telemedicine. 2020.
    
    

25. 25.Scott Kruse C, Karem P, Shifflett K, Vegi L, Ravi K, Brooks M. Evaluating barriers to adopting telemedicine worldwide: A systematic review. Journal of telemedicine and telecare. 2018;24(1):4–12.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F06%2F2021.07.05.21260018.atom) 

26. 26.Providers. AoMH. COVID-19 and Digital Access to Personalised Mental Health Services and Support 2020.
    
    

27. 27.Cowan KE, McKean AJ, Gentry MT, Hilty DM, editors. Barriers to use of telepsychiatry: clinicians as gatekeepers. Mayo Clinic Proceedings; 2019: Elsevier.
    
    

28. 28.Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of eMental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR mental health. 2018;5(1):e9769.
    
    

29. 29.Sharma A, Sasser T, Schoenfelder Gonzalez E, Vander Stoep A, Myers K. Implementation of home-based telemental health in a large child psychiatry department during the COVID-19 crisis. Journal of child and adolescent psychopharmacology. 2020;30(7):404–13.
    
    

30. 30.Connolly SL, Stolzmann KL, Heyworth L, Weaver KR, Bauer MS, Miller CJ. Rapid increase in telemental health within the Department of Veterans Affairs during the COVID-19 pandemic. Telemedicine and e-Health. 2021;27(4):454–8.
    
    

31. 31.Gentry MT, Puspitasari AJ, McKean AJ, Williams MD, Breitinger S, Geske JR, et al. Clinician satisfaction with rapid adoption and implementation of telehealth services during the COVID-19 pandemic. Telemedicine and e-Health. 2021.
    
    

32. 32.Nicholas J, Bell IH, Thompson A, Valentine L, Simsir P, Sheppard H, et al. Implementation lessons from the transition to telehealth during COVID-19: a survey of clinicians and young people from youth mental health services. Psychiatry Research. 2021;299:113848.
    
    

33. 33.Smith K, Ostinelli E, Macdonald O, Cipriani A. Covid-19 and telepsychiatry: development of evidence-based guidance for clinicians. JMIR mental health. 2020;7(8):e21108.
    
    

34. 34.Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009;6(7):e1000097.
    
    

35. 35.The Endnote Team. Endnote. Endnote X9 ed. Philadelphia, PA: Clarivate; 2013.
    
    

36. 36.Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Systematic Reviews. 2016;5(1):210.
    
    

37. 37.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science. 2009;4(1):1–15.
    
    

38. 38.Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(2):65–76.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s10488-010-0319-7&link_type=DOI) 

39. 39.Safaeinili N, Brown-Johnson C, Shaw JG, Mahoney M, Winget M. CFIR simplified: Pragmatic application of and adaptations to the Consolidated Framework for Implementation Research (CFIR) for evaluation of a patient-centered care transformation within a learning health system. Learning health systems. 2020;4(1):e10201.
    
    

40. 40.Thomas J, Graziosi S, Brunton J, Ghouze Z, O’Driscoll P, Bond M. EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis. Computer software] EPPI-Centre Software UCL Social Research Institute London https://eppiioe ac uk/cms/Default aspx. 2020.
    
    

41. 41.Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods appraisal tool (MMAT), version 2018. Registration of copyright. 2018;1148552:10.
    
    

42. 42.Tyndall J. AACODS Checklist. Flinders University. 2010.
    
    

43. 43.Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92.
    
    

44. 44.Aafjes-van Doorn K, Bekes V, Prout TA. Grappling with our therapeutic relationship and professional self-doubt during COVID-19: will we use video therapy again? Counselling Psychology Quarterly. 2020.
    
    

45. 45.Anton MT, Ridings LE, Gavrilova Y, Bravoco O, Ruggiero KJ, Davidson TM. Transitioning a technology-assisted stepped-care model for traumatic injury patients to a fully remote model in the age of COVID-19. Counselling Psychology Quarterly. 2020.
    
    

46. 46.Barney A, Buckelew S, Mesheriakova V, Raymond-Flesch M. The COVID-19 Pandemic and Rapid Implementation of Adolescent and Young Adult Telemedicine: Challenges and Opportunities for Innovation. J Adolesc Health.
    
    

47. 47.Bekes V, Aafjes-van Doorn K. Psychotherapists’ Attitudes Toward Online Therapy During the COVID-19 Pandemic. Journal of Psychotherapy Integration. 2020.
    
    

48. 48.Békés V, Aafjes-van Doorn K, Prout TA, Hoffman L. Stretching the Analytic Frame: Analytic Therapists’ Experiences with Remote Therapy During COVID-19. J Am Psychoanal Assoc. 2020.
    
    

49. 49.Benaque A, Gurruchaga MJ, Abdelnour C, Hernández I, Cañabate P, Alegret M, et al. Dementia Care in Times of COVID-19: Experience at Fundació ACE in Barcelona, Spain. J Alzheimers Dis. 2020.
    
    

50. 50.Berdullas Saunders S, Gesteira Santos C, Morán Rodríguez N, Fernández Hermida JR, Santolaya F, Sanz Fernández J, et al. The COVID-19 psychological helpline of the Spanish Ministry of Health and Spanish Psychological Association: characteristics and demand. Rev Esp Salud Publica. 2020.
    
    

51. 51.Rohan B, Jonathan H, Christian Dalton L, View OPVSJ, Sian O, Una F, et al. Impact of the COVID-19 pandemic on Older Adults Mental Health Services: a mixed methods study. MedRxiv. 2020.
    
    

52. 52.Bierbooms Jjpa van Haaren M, Ijsselsteijn WA, de Kort YAW, Feijt M, Bongers IMB. Integration of Online Treatment Into the “New Normal” in Mental Health Care in Post-COVID-19 Times: Exploratory Qualitative Study. Journal of Medical Internet Research. 2020.
    
    

53. 53.Boldrini T, Lomoriello AS, Del Corno F, Lingiardi V, Salcuni S. Psychotherapy During COVID-19: How the Clinical Practice of Italian Psychotherapists Changed During the Pandemic. Frontiers in Psychology. 11.
    
    

54. 54.Alexandra B, Alison M, Henry A, Daisy F. Impact of the Covid-19 pandemic on the mental health and wellbeing of adults with mental health conditions in the UK: A qualitative interview study. MedRxiv. 2020.
    
    

55. 55.Carpiniello B, Tusconi M, di Sciascio G, Zanalda E, di Giannantonio M. Mental health services in Italy during the COVID-19 pandemic. Psychiatry and Clinical Neurosciences. 74(8):442–3.
    
    

56. 56.Cheli S, Cavalletti V, Petrocchi N. An online compassion-focused crisis intervention during COVID-19 lockdown: a cases series on patients at high risk for psychosis. Psychosis-Psychological Social and Integrative Approaches.
    
    

57. 57.Chen JA, Chung WJ, Young SK, Tuttle MC, Collins MB, Darghouth SL, et al. COVID-19 and telepsychiatry: Early outpatient experiences and implications for the future. Gen Hosp Psychiatry. 66:89–95.
    
    

58. 58.Childs Amber W, Unger A, Li L. Rapid design and deployment of intensive outpatient, group-based psychiatric care using telehealth during coronavirus disease 2019 (COVID-19). Journal of the American Medical Informatics Association. 2020;27(7):1420–4.
    
    

59. 59.Colle R, Ait T, A EK, de Larminat D, Commery L, Boniface B, et al. Short-term acceptability by patients and psychiatrists of the turn to psychiatric teleconsultation in the context of the COVID-19 pandemic. Psychiatry Clin Neurosci. 74(8):443–4.
    
    

60. 60.Datta N ini, Derenne J, ers M, Lock James D. Telehealth transition in a comprehensive care unit for eating disorders: Challenges and long-term benefits. International Journal of Eating Disorders. 2020;53(11):1774–9.
    
    

61. 61.Dores AR, Geraldo A, Carvalho IP, Barbosa F. The Use of New Digital Information and Communication Technologies in Psychological Counseling during the COVID-19 Pandemic. Int J Environ Res Public Health. 17(20).
    
    

62. 62.Erekson DM, Bailey RJ, Cattani K, Fox ST, Goates-Jones MK. Responding to the Covid-19 pandemic at a university counseling center: administrative actions, client retention, and psychotherapy outcome. Counselling Psychology Quarterly.
    
    

63. 63.Feijt M, de Kort Y, Bongers I, Bierbooms J, Westerink J, Ijsselsteijn W. Mental Health Care Goes Online: Practitioners’ Experiences of Providing Mental Health Care During the COVID-19 Pandemic. Cyberpsychology Behavior and Social Networking.
    
    

64. 64.Fernandez A, a F, Munguia L, Mestre-Bach G, Steward T, Etx, et al. COVID Isolation Eating Scale (CIES): Analysis of the impact of confinement in eating disorders and obesity-A collaborative international study. European Eating Disorders Review. 28(6):871–83.
    
    

65. 65.Foye U,  Dalton-Locke Christian, Harju-Seppanen J, Lane R, Beams L, San J,  Norha Vera, et al. How has Covid-19 affected mental health nurses and the delivery of mental health nursing care in the UK? Results of a mixed methods study. MedRxiv. 2020.
    
    

66. 66.Gaddy S, Gallardo R, McCluskey S, Moore L, Peuser A, Rotert R, et al. COVID-19 and Music Therapists’ Employment, Service Delivery, Perceived Stress, and Hope: A Descriptive Study. Music Therapy Perspectives. 2020;38(2):157–66.
    
    

67. 67.Steven G, Ceri D, Jackie H, Patrick N, Rachel Rowan O, Prisha S, et al. Experiences of living with mental health problems during the COVID-19 pandemic in the UK: a coproduced, participatory qualitative interview study. MedRxiv. 2020.
    
    

68. 68.Gomet R, Souchet C, Mouellic Mathilde L, Maafa M, Labey A ra, et al. Addiction outpatient hospital reinvents itself during the Coronavirus outbreak. Alcoologie et Addictologie. 42(1):54–7.
    
    

69. 69.Graell M, Moron-Nozaleda MG, Camarneiro R, Villasenor A, Yanez S, Munoz R, et al. Children and adolescents with eating disorders during COVID-19 confinement: Difficulties and future challenges. European Eating Disorders Review. 28(6):864–70.
    
    

70. 70.Grover S eep, Mishra E, Chakrabarti S, Mehra A, Sahoo S. Telephonic monitoring of patients on clozapine in the resource-poor setting during the COVID-19 pandemic. Elsevier B.V.; 2020. p. 489– 90.
    
    

71. 71.Grover S, Mehra A, Sahoo S, Avasthi A, Tripathi A, D’Souza A, et al. Impact of COVID-19 pandemic and lockdown on the state of mental health services in the private sector in India. Indian Journal of Psychiatry. 62(5):488–93.
    
    

72. 72.Grover S, Mehra A, Sahoo S, Avasthi A, Tripathi A, D’Souza A, et al. State of mental health services in various training centers in India during the lockdown and COVID-19 pandemic. Indian Journal of Psychiatry. 62(4):363–9.
    
    

73. 73.Haxhihamza K, Arsova S, Bajraktarov S, Kalpak G, Stefanovski B, Novotni A, et al. Patient Satisfaction with Use of Telemedicine in University Clinic of Psychiatry: Skopje, North Macedonia During COVID-19 Pandemic. Telemedicine and E-Health.
    
    

74. 74.He Z, Chen J, Pan K, Yue Y, Cheung T, Yuan Y, et al. The development of the ‘COVID-19 Psychological Resilience Model’ and its efficacy during the COVID-19 pandemic in China. Int J Biol Sci. 2020;16(15):2828–34.
    
    

75. 75.Hom MA, Weiss RB, Millman ZB, Christensen K, Lewis EJ, Cho S, et al. Development of a Virtual Partial Hospital Program for an Acute Psychiatric Population: Lessons Learned and Future Directions for Telepsychotherapy. Journal of Psychotherapy Integration. 30(2):366–82.
    
    

76. 76.Humer E, Pieh CS, Kuska M, Barke A, Doering BK, Gossmann K, et al. Provision of Psychotherapy during the COVID-19 Pandemic among Czech, German and Slovak Psychotherapists. International Journal of Environmental Research and Public Health. 17(13).
    
    

77. 77.Humer E, Stippl P, Pieh C, Schimböck W, Probst T. Psychotherapy via the Internet: What Programs Do Psychotherapists Use, How Well-Informed Do They Feel, and What Are Their Wishes for Continuous Education? Int J Environ Res Public Health. 17(21).
    
    

78. 78.Izakova L, Breznoscakova D, ova K, Valkucakova V, Bezakova G, Suvada J. What mental health experts in Slovakia are learning from COVID-19 pandemic? Indian Journal of Psychiatry. 62(9):459–66.
    
    

79. 79.Jurcik T, Jarvis GE, Doric JZ, Krasavtseva Y, Yaltonskaya A, Ogiwara K, et al. Adapting mental health services to the COVID-19 pandemic: reflections from professionals in four countries. Counselling Psychology Quarterly.
    
    

80. 80.Khanna R, Murnane T, Kumar S, Rolfe T, Dimitrieski S, McKeown M, et al. Making working from home work: reflections on adapting to change. Australas Psychiatry. 28(5):504–7.
    
    

81. 81.Kopec K, Janney CA, Johnson B, Spykerman K, Ryskamp B, Achtyes ED. Rapid Transition to Telehealth in a Community Mental Health Service Provider During the COVID-19 Pandemic. Prim Care Companion CNS Disord. 22(5).
    
    

82. 82.Lai FH, Yan EW, Yu KK, Tsui WS, Chan DT, Yee BK. The Protective Impact of Telemedicine on Persons With Dementia and Their Caregivers During the COVID-19 Pandemic. Am J Geriatr Psychiatry. 28(11):1175–84.
    
    

83. 83.Lakeman R, Crighton J. The Impact of Social Distancing on People with Borderline Personality Disorder: The Views of Dialectical Behavioural Therapists. Issues in Mental Health Nursing.
    
    

84. 84.Lin XB, Swift J, Cheng Y, An Q, Liang H, Wang YS, et al. The Psychological Hotline Services Quality Survey during the Pandemic of COVID-19 in Mainland China. International Journal of Mental Health Promotion. 2020;22(3):109–13.
    
    

85. 85.Looi JC, Allison S, Bastiampillai T, Pring W. Private practice metropolitan telepsychiatry in smaller Australian jurisdictions during the COVID-19 pandemic: preliminary analysis of the introduction of new Medicare Benefits Schedule items. Australas Psychiatry. 28(6):639–43.
    
    

86. 86.Looi JC, Allison S, Bastiampillai T, Pring W. Private practice metropolitan telepsychiatry in larger Australian states during the COVID-19 pandemic: an analysis of the first 2 months of new MBS telehealth item psychiatrist services. Australas Psychiatry. 28(6):644–8.
    
    

87. 87.Lynch DA, Medalia A, Saperstein A. The Design, Implementation, and Acceptability of a Telehealth Comprehensive Recovery Service for People With Complex Psychosis Living in NYC During the COVID-19 Crisis. Frontiers in Psychiatry. 11.
    
    

88. 88.McBeath AG, du P, Bager-Charleson S. The challenges and experiences of psychotherapists working remotely during the coronavirus pandemic. Counselling & Psychotherapy Research. 20(3):394–405.
    
    

89. 89.Medalia A,  Lynch David  A, Herl s T. Telehealth conversion of serious mental illness recovery services during the COVID-19 crisis. Psychiatric Services. 71(8):872.
    
    

90. 90.Miu AS, Vo HT, Palka JM, Glowacki CR, Robinson RJ. Teletherapy with serious mental illness populations during COVID-19: telehealth conversion and engagement. Counselling Psychology Quarterly.
    
    

91. 91.Olwill C, Mc Nally D, Douglas L. Psychiatrist experience of remote consultations by telephone in an outpatient psychiatric department during the covid-19 pandemic. Irish Journal of Psychological Medicine. No Pagination Specified.
    
    

92. 92.Rashmi P, Jessica I, Aimee B, Matthew B, Hitesh S, Megan P, et al. Impact of the COVID-19 pandemic on remote mental healthcare and prescribing in psychiatry. MedRxiv. 2020.
    
    

93. 93.Peralta E, Taveras M. Effectiveness of teleconsultation use in access to mental health services during the coronavirus disease 2019 pandemic in the Dominican Republic. Indian Journal of Psychiatry. 62(9):492–4.
    
    

94. 94.Pierce Bradford S, Perrin Paul B, Tyler Carmen M, McKee Grace B, Watson Jack D. The COVID-19 telepsychology revolution: A national study of pandemic-based changes in U.S. mental health care delivery. American Psychologist.No Pagination Specified.
    
    

95. 95.Probst T, Stippl P, Pieh C. Changes in Provision of Psychotherapy in the Early Weeks of the COVID-19 Lockdown in Austria. International Journal of Environmental Research and Public Health. 17(11).
    
    

96. 96.Reilly SE, Zane KL, McCuddy WT, Soulliard ZA, Scarisbrick DM, Miller LE, et al. Mental Health Practitioners’ Immediate Practical Response During the COVID-19 Pandemic: Observational Questionnaire Study. Jmir Mental Health. 7(9).
    
    

97. 97.Roach P, Zwiers A, Cox E, Fischer K, Charlton A, Josephson CB, et al. Understanding the impact of the COVID-19 pandemic on well-being and virtual care for people living with dementia and care partners living in the community. MedRxiv. 2020.
    
    

98. 98.Roncero C, García-Ullán L, de la Iglesia-Larrad JI, Martín C, Andrés P, Ojeda A, et al. The response of the mental health network of the Salamanca area to the COVID-19 pandemic: The role of the telemedicine. Psychiatry Res. 291:113252.
    
    

99. 99.Rosen Craig S, Morl Leslie A, Glassman Lisa H, Marx Brian P, Weaver K, et al. Virtual mental health care in the Veterans Health Administration’s immediate response to coronavirus disease-19. American Psychologist. No Pagination Specified.
    
    

100.100.Sasangohar F, Bradshaw MR, Carlson MM, Flack JN, Fowler JC Freel, et al. Adapting an Outpatient Psychiatric Clinic to Telehealth During the COVID-19 Pandemic: A Practice Perspective. J Med Internet Res. 22(10):e22523.
    
    

101.101.Scharff A, Breiner CE, Ueno LF, Underwood SB, Merritt EC, Welch LM, et al. Shifting a training clinic to teletherapy during the COVID-19 pandemic: a trainee perspective. Counselling Psychology Quarterly.
    
    

102.102.Schlegl S, Meule A, Favreau M, Voderholzer U. Bulimia nervosa in times of theCOVID-19 pandemic-Results from an online survey of former inpatients. European Eating Disorders Review. 28(6):847–54.
    
    

103.103.Sciarrino NA, Myers US, Wangelin BC. When chaos is the norm: How some veterans with PTSD are continuing to engage in trauma-focused treatments during the COVID-19 pandemic. Psychol Trauma. 12:S69–s70.
    
    

104.104.Sequeira A, Alozie A, Fasteau M, Lopez AK, Sy J, Turner KA, et al. Transitioning to virtual programming amidst COVID-19 outbreak. Counselling Psychology Quarterly.
    
    

105.105.Severe J, Tang R, Horbatch F, Onishchenko R, Naini V, Blazek MC. Factors Influencing Patients’ Initial Decisions Regarding Telepsychiatry Participation During the COVID-19 Pandemic: Telephone-Based Survey. JMIR Form Res 2020.
    
    

106.106.Sheehan R, Dalton-Locke C, Ali A, Totsika V, San Juan NV, Hassiotis A. Mental healthcare and service user impact of the COVID-19 pandemic: results of a UK survey of staff working with people with intellectual disability and developmental disorders. MedRxiv. 2020.
    
    

107.107.Sklar M, Reeder K, Carandang K, Ehrhart MG, Aarons GA. An Observational Study of the Impact of COVID-19 and the Transition to Telehealth on Community Mental Health Center Providers. 2020.
    
    

108.108.Jet DT, Hunna JW, Laura MT, Stina B, Rachael EF, Casey MM, et al. Early Impact of COVID-19 on Individuals with Eating Disorders: A survey of ∼1000 Individuals in the United States and the Netherlands. MedRxiv. 2020.
    
    

109.109.Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett ML, Huskamp HA. Suddenly Becoming a “Virtual Doctor”: Experiences of Psychiatrists Transitioning to Telemedicine During the COVID-19 Pandemic. Psychiatr Serv. 71(11):1143–50.
    
    

110.110.Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett M, Huskamp HA. Treatment of opioid use disorder during COVID-19: Experiences of clinicians transitioning to telemedicine. J Subst Abuse Treat. 118:108124.
    
    

111.111.van Dijk SDM, Bouman R, Folmer EH, den Held RC, Warringa JE, Marijnissen RM, et al. (Vi)-rushed Into Online Group Schema Therapy Based Day-Treatment for Older Adults by the COVID-19 Outbreak in the Netherlands. Am J Geriatr Psychiatry. 28(9):983–8.
    
    

112.112.Wang XB, Gordon RM, Snyder EW. Comparing Chinese and US practitioners’ attitudes towards teletherapy during the COVID-19 pandemic. Asia-Pacific Psychiatry.
    
    

113.113.Wilson CA, Dalton-Locke C, Johnson S, Simpson A, Oram S, Howard LM. Challenges and opportunities of the COVID-19 pandemic for perinatal mental health care: a mixed methods study of mental health care staff. MedRxiv. 2020.
    
    

114.114.Wood HJ, Gannon JM, Chengappa KNR, Sarpal DK. Group teletherapy for first-episode psychosis: Piloting its integration with coordinated specialty care during the COVID-19 pandemic. Psychology and Psychotherapy-Theory Research and Practice.
    
    

115.115.Helen W, Michael L, Vladeta A-G, Erich S, Susan Y, Pascal B, et al. Treatment provision for adults with ADHD during the COVID-19 pandemic: An exploratory study on patient and therapist experience with on-site sessions using face masks vs. telepsychiatric sessions. MedRxiv. 2020.
    
    

116.116.Yellowlees P, Nakagawa K, Pakyurek M, Hanson A, Elder J, Kales HC. Rapid Conversion of an Outpatient Psychiatric Clinic to a 100% Virtual Telepsychiatry Clinic in Response to COVID-19. Psychiatr Serv. 71(7):749–52.
    
    

117.117.Zulfic Z, Liu D, Lloyd C, Rowan J, Schubert KO. Is telepsychiatry care a realistic option for community mental health services during the COVID-19 pandemic? Australian and New Zealand Journal of Psychiatry.