Abstract
Objective The aim of this research was to create a scale to assess the competency of therapists who conduct group cognitive behavioral therapy (G-CBT), which can serve as a tool to aid the continued training of therapists.
Methods Three stepped studies were conducted. Study 1: Through literature review and experts’ consensus process, essential skills for G-CBT were articulated and were categorized according to the criteria of the Cognitive Therapy Scale, a well-established rating scale to evaluate clinicians’ skills in individual cognitive behavioral therapy. The list of those skills was organized into a rating scale. Study 2: Behavioral anchors were added to each skill and were classified by the levels of difficulty (beginner, intermediate, and advanced levels), based on the rating by G-CBT experts. Study 3: Interrater reliability and validity of the rating scale were examined in a sample of forty-one videotaped G-CBT sessions of actual clinical sessions and educational role-plays.
Results A twelve-item Group Cognitive Therapy Scale was developed. It consists of eleven items that have been adopted from the original Cognitive Therapy Scale, with the addition of a new item called “Intervention Using Relationships with Other Participants.” This scale showed excellent internal consistency (Cronbach’s alpha: 0.95), satisfactory inter-rater reliability (interclass correlation coefficients: 0.65 - 0.88), and high predictive validity.
Conclusion A novel rating scale to evaluate therapists’ competency in G-CBT was developed and successfully validated.
Introduction
Group cognitive behavioral therapy (G-CBT) is a type of cognitive behavioral therapy (CBT) that is implemented in a group format. A group format embodies social and emotional benefits for the participants of sharing experiences with others (1). G-CBT is generally considered cost-effective compared to individual sessions (1, 2), although its effect size is somewhat lower than individual CBT. Therefore, G-CBT programs are often considered as a part of low-intensity interventions, which is provided prior to more intense interventions such as individual psychotherapy and pharmacotherapy (3). G-CBT also has been implemented outside of the medical field, for instance in the judiciary and industrial fields (4).
Although G-CBT is categorized as a low-intensity intervention, it does not mean that G-CBT is easier for therapists to conduct than individual CBT. Providing group therapy requires therapists to have skills to facilitate interactions and communication among the members of the group, to focus on the therapeutic group processes, and to find solutions to the problems that arise in the group (5, 6). Therefore, therapists of G-CBT need to have such skills in addition to the skills that are required in individual CBT.
A few manuals and competence assessment scales for group psychotherapies have been developed as the aiding tools for therapist training (7, 8, 9, 10). However, quality control methods have not been established in the field of G-CBT and awaits the development of a standardized assessment scale to define and measure therapist competence. Once developed, the scale would not only encourage self-reflection among therapists, but it would also serve as a tool for evaluating the effectiveness of training and ensuring the quality of treatment (11, 12, 13, 14).
A few scales that measure therapists’ competence in G-CBT have been developed. Hepner et al. (15) developed an adherence and competence rating scale for G-CBT for depression. The scale includes items specific to group therapy (group dynamics, group motivation, group participation, etc.). Wong (16) created a quality assessment checklist for rehabilitation group therapists of people with acquired brain injury working on their memory skills. This checklist covers competencies regarding group discussion facilitation, communication skills, interpersonal competence, and session structure.
However, these tools have limitations in the following ways: 1) they are specific to people with depression, substance dependence, or acquired brain injury and does not include skills that could apply to G-CBT for other conditions, and 2) the scales do not contain specific examples of therapist behavior, which limits the reliability of the rating. Therefore, there is a need to establish a more reliable rating scale to assess therapists’ competency for G-CBT that can be are independent of specific disorders.
The objectives of this study were to develop and validate a rating scale to evaluate therapists’ competency in conducting G-CBT. We placed emphasis on developing a scale that assesses the underlying skills common to a range of disorders and intervention methods and on developing a checklist of illustrative therapist behaviors to improve the reliability of the rating. We implemented the following processes, which are outlined in detail in the sections below. Study 1: Articulating essential skills for G-CBT by conducting literature review and obtaining experts’ consensus; Study 2: Providing behavioral anchors to each of the scale items and categorizing them by the levels of difficulty (beginner-, intermediate- and advanced-levels); and Study 3: Examining internal consistency, inter-rater reliability, and predictive validity using G-CBT video samples.
Study 1
The objective of this first study was to articulate essential clinical skills of therapists who conduct G-CBT.
Methods
The following three steps were taken - literature search, organizing of group therapist skills and examining face- and content-validities.
Literature search
Articles included in the review were English publications regarding clinical competence to conduct face-to-face group therapy for adults with mental health problems. Publications regarding individual, family or online therapies were excluded. We searched peer-reviewed articles published from January 1980 to October 2020, using PsychInfo, Scopus, and PubMed databases using the broad strategy of including any of the following competence-related terms: “therapeutic factor,” “therapeutic competence,” “clinical skill,” or “clinical competence,” in combination with one or more of the following group therapy terms: “group psychotherapy,” “group format,” or “group therapy”.
A two-stage process for selecting relevant articles was employed by two independent reviewers (MM and MN). First, they screened titles and abstracts of all searched articles. Second, they reviewed full copies of screened articles and assessed for eligibility. When there was discrepancy, discussion was made until reached agreement.
Organization of group therapist skills
To create a rating scale for G-CBT, the required skills for G-CBT which were elicited by the literature search were organized by mapping onto the framework of an existing rating scale, the Cognitive Therapy Scale (CTS). The CTS is a well-established scale to assess therapists’ competence in individual CBT. The CTS (17) and its revised edition (18) comprise eleven essential skills for CBT (agenda setting, feedback, understanding, interpersonal effectiveness, collaboration, pacing and efficient use of time, guided discovery, focusing on key cognition or behaviors, strategy for change, application of cognitive-behavioral techniques, and homework). Each item has clear goals, and a therapist’s skills are rated according to the degree of achievement of each goal on a 7-point scale (from 0 = poor to 6 = excellent), which is evaluated based on video- or audio-recordings or direct observation of actual CBT sessions. This scale has been used as the standard for therapist qualification in clinical trials and clinical practices (19, 20) and has been used in many accrediting bodies in CBT, such as the Beck Institute (https://beckinstitute.org/) and the Academy of Cognitive and Behavioral Therapies (https://www.academyofct.org/).
Four clinicians of different disciplines (a psychiatrist, a physician in psychosomatic medicine, a psychotherapist, and a clinical psychologist) with expertise in G-CBT had four focused-group discussions (six hours each) to re-categorize the extracted competencies according to the framework of the CTS. We endeavored to articulate the above essential skills in the descriptions of observable behavior. Using the CTS format, we described the essential skills under the following aspects: 1) objectives of each skill, 2) desirable therapist skills, and 3) rating criteria for each clinical skill item. Further, we added 4) a behavioral checklist for each item, so that it can serve as an objectively-measurable standard of behavior of therapists. We devised the scale so that it can be used for various mental problems and in a wide range of settings (e.g. clinical, educational, industrial, or judicial settings and stress management for healthy individuals).
We paraphrased the description in the original CTS for the group-therapy context. The descriptions in the original CTS that can be applied to G-CBT were adopted as they were, and a few new descriptions that are unique to G-CBT were added. In addition to eleven items of the original CTS, a new item “Intervention Using Relationships with Other Participants” was created, in order to accommodate skills that are specific to G-CBT. We provided each of the twelve items with 1) the objective and 2) desirable therapist strategies in group therapy. In addition, we created 3) behavioral checklists, in the scope of better illustrating the “desirable therapist strategies” as an observable behavior. Through this procedure, a prototype of the twelve-item group CTS (G-CTS) was developed.
Face and content validities
To assess face validity of the scale, a panel of ten G-CBT experts outside the research team rated the appropriateness and importance of each G-CTS item using a five-point rating scale as follows: 1: Not important; 2: Slightly important; 3: Somewhat important; 4: Important; and 5: Very important. The members of the panel were board members of the Japanese Association of Cognitive Behavioral Group Therapy, with more than ten years of experience in G-CBT and consisted of psychiatrists, nurses, licensed mental health workers, occupational therapists, and clinical psychologists from different fields (medical, welfare, industrial, and educational).
The impact score (IS) for each item was calculated using the following formula,
where “Frequency” is the number of experts rated the item as 4 or 5, and “Importance” is the mean score of the item. The items with the impact score of 1.5 or more were considered appropriate for the scale [21, 22].
To assess content validity, the content validity ratio (CVR) and the content validity index (CVI) was calculated.
The CVR indicates whether important and correct items are included in the scale. The CVR is calculated by the following formula;
where N is the total number of experts and N is the total number of experts rated the intended item as essential. The CVR of 0.62 and more are considered appropriate [23]. The same expert panel evaluated the essentiality of the items on the following four-point scale: 4: Essential; 3: Essential but needs modification; 2: Relevant but not essential, and 1: Not essential.
The CVI was calculated to determine whether the items appropriately measure clinical skills of therapists in G-CBT. The same 10 experts rated the simplicity, relevancy, or specificity, and clarity of each item on a four-point rating scale. The number of experts who rated an item as 3 or 4 was divided by the total number of experts to calculate the CVI of that item. Items with a CVI of more than 0.90 were kept, while items with a CVI of 0.80–0.89 were revised. In addition, the items were modified based on comments in the free-text section.
Intra-class correlation coefficients and standard deviations for each value were calculated.
Results
Literature search
Our literature search yielded the following 148 papers after excluding duplications: thirty-eight (PubMed), seventeen (Web of Science), fifty-seven (Scopus), and thirty-six abstracts/titles (PsycInfo). Thirteen of these 148 papers were included in the review, based on the consensus review by two of our authors. Further, five relevant books were identified by manual search (24, 6, 25, 26, 27). Seventeen group therapist skills were extracted from this review (Table 1).
Organization of group therapist skills
Table 2 shows the projection of extracted seventeen therapeutic skills onto the items of the CTS. A new twelfth item of “Intervention Using Relationships with Other Participants” was created in order to accommodate skills that are specific to G-CBT. The creation of this item was also suggested by the panel of experts who evaluated face and content validity of the scale. This item represents therapists’ skill of being aware of and utilizing the interactions of the participants (group dynamics) as a therapeutic technique. For example, a therapist may deliberately ask other participants to comment on their fellow participants, as a way of bringing insight to a single participant. Table 3 presents an example of an item of G-CTS.
Face and content Validity
The Tables 4–7 shows the IS, CVR and CVI for each item of the prototype G-CTS. All of the items had the IS of 1.5 or more, thus were kept remained in the scale. The mean ICC of CVR was .719 (95° confidence interval: .607-.809), which showed a substantial agreement rate. Three items of the with the CVR of less than 0.62 and six items with the CVI of 0.80−0.89 were revised. Nine items were revised based on narrative comments by the panel. The details of the revision is described in Tables 4-7. Finally, the G-CTS with 66 behavioral-checklist items were created.
Study 2
The objective of this second study was to classify G-CBT therapists’ desirable behaviors according to the degree of difficulty in implementation of these conducts (beginner, intermediate, and advanced levels), thereby establishing a rating system for the Group Cognitive behavioral Therapy Scale (G-CTS).
Methods
A survey was administered by the board members of the Japanese Association of Group Cognitive Behavioral Therapy. The clinical background of the respondents was diverse, consisting of fifteen clinical psychologists (42.9%), ten physicians specializing in psychiatry or psychosomatic medicine (28.8%), six psychiatric social workers (17.1%), three nurses (8.6%), and others (8.5%). Their areas of expertise were medical (n=26, 74.3%), welfare (n=6, 17.1%), education (n=3, 8.6%), industry (n=2, 5.7%), and judicial (n=2, 5.7%).
This survey asked the respondents to evaluate each point on the 66-item checklist by selecting one of the following five choices: 1) Beginner Level (Every G-CBT therapist is required to do this), 2) Intermediate Level (Skilled therapists are desired to do this), 3) Advanced Level (Skilled therapists may do this), 4) Too complex (It can be relevant with G-CBT, but is too difficult and even advanced-level therapists may not do this), and 5) Inappropriate (It is not relevant to G-CBT).
We classified each item into Beginner, Intermediate, and Advanced levels based on the response distribution and according to the following criteria:
The items with more than 10% of endorsement to “Advanced level” were categorized as “Advanced level”.
The items with <10% of endorsement to “Advanced level” and with more than 50% of endorsement to other specific difficulty level were categorized into that specific difficulty level.
The items with <10% of endorsement to “Advanced level” and with < 50% of endorsement to any other difficulty level were categorized as “Intermediate level”.
Results
The results of the survey are shown in Tables 8 to 12. Twenty three items were classified as Beginner Level, thirty one as Intermediate Level, and twelve as Advanced Level. No item fell into “Too complex” or “Inappropriate”.
Additionally, we reorganized the above checklist for group therapists’ desirable behaviors according to the Revised Cognitive Therapy Scale (CTS-R) rating system (18), which provides anchor points for evaluation on the seven-level Likert scale. We did so because both the level of competence in the execution of CBT techniques and adherence to CBT protocols are taken into consideration in the CTS-R. A score of 0 indicates non-adherence to a standard CBT protocol, and a score of 6 indicates extreme expertise even in difficult cases.
We added the following scoring rules based on the behavioral checklists:
Three points or more are given if a therapist satisfied all the Beginner-level items.
Four points or more are given if a therapist satisfies all the Beginner-level and Intermediate-level items.
Five points or more are given if a therapist satisfies all the items of both the Beginner-level and Intermediate-level items, in addition to some of the Advanced-level items (Table 13).
View this table:Table 13. Scoring for each item
Study 3
The objective of this third study was to investigate the reliability and validity of the Group Cognitive Therapy Scale (G-CTS). We used two groups of video samples - training sessions conducted by novice therapists (Beginner Group samples) and by skilled therapists (Advanced Group samples).
A past study that examined degree of proficiency of group CBT therapists’ skills yielded moderate effect size (Cohen’s d = 0.58) (18). Since our study sample comprised expert- and novice-therapists, we assumed that the effect size will be large (d = 0.8). Using the G*Power 3 program (28), we estimated our target sample size as 23 participants per arm to detect an effect size of 0.8 at an alpha error level of 0.05 and a beta error level of 0.2.
Methods
Samples
Two sets of G-CBT session videos were prepared (Beginner Group samples and Advanced Group samples). The Beginner Group videos were video-recordings of the roleplays of typical G-CBT sessions for mild depression, conducted by psychology graduate students as a way of training. The Advanced Group videos were video-recordings of actual clinical G-CBT sessions for patients with mild depression in actual clinical therapy, as the Beginner Group role-played with typical minor depression patients. The therapists in Advanced Group completed the basic G-CBT training course offered by the Association of Cognitive Behavioral Group Therapy in Japan (https://cbgt.org/) and had five or more years of experience in G-CBT (mean length of G-CBT experience = 10.47 years). We aimed to prepare 23 videos for each group, but for a practical reason, we were able to shoot only 18 videos for the Beginner Group.
Participants
Two clinicians with five or more years of experience in G-CBT who had undergone training in G-CTS (a two-hour lecture and an evaluation exercise) independently rated each video, based on the G-CTS.
Statistical analysis
We computed Cronbach’s alpha coefficient to examine the internal consistency of the G-CTS and interclass correlation coefficients to assess the inter-rater reliability. Since the data did not show normal distribution, we compared the total G-CTS scores of the Beginner Group and the Advanced Group sessions using the Mann-Whitney test, to examine predictive validity.
Results
Internal consistency
Cronbach’s alpha coefficients were 0.95[.93−.97] and 0.96[.95−.98] for each rater, confirming a high level of internal consistency.
Inter-item correlations
We computed correlation coefficients between each item, which ranged between .55 and .83, showing that the redundancy was not very high.
Inter-rater reliability
Interclass correlation coefficients for each item were satisfactory, ranging from 0.75 to 0.90. The interclass correlation coefficient for the total G-CTS score was 0.82 [.68−.90] (Table 14).
Predictive validity
The comparison between each G-CTS score of the Beginner Group and Advanced Group videos is shown in Table 15. All the scores of each item and the total were significantly higher in the Advanced Group than in the Beginner Group, with large effect size [.60−.88].
Discussion
The purpose of this study was to develop a scale for evaluating the quality of G-CBT and to investigate its validity and reliability.
We created a twelve-item scale for G-CBT with solid behavioral anchors by reforming the CTS, which is a well-established scale to evaluate the quality of individual CBT, and by adding specific skills for group psychotherapy. The most important feature of the G-CTS was that it increased rater agreement by describing many specific examples of desired therapist behavior.
We demonstrated that G-CTS has high internal reliability and high inter-rater reliability compared with existing instruments that measure the quality of individual CBT. For example, the interclass correlation coefficients of the CTS-R and Assessment of Core CBT Skills (ACCS) were 0.40-0.86 and 0.27-0.83, respectively (18, 29). The interclass correlation coefficients for the total scores of CTS-Rs were 0.63 (thirteen-item edition) and 0.57 (fourteen-item edition) (18). We conclude that our scale achieved an exceptionally high inter-rater reliability because it has clear behavioral anchors for rating. Further, the high predictive validity of the G-CTS was demonstrated by comparing the total scores and the scores for each item in the Beginner Group and the Advanced Group of G-CBT therapists.
The G-CTS behavioral checklist created in this study provides concrete guidelines that can be used by therapists to hone their skills in G-CBT. The G-CTS checklist and the rating scale offer a framework that can be utilized in teaching, supervision, research, and qualification of G-CBT therapists as well as in programs that facilitate education and training of new practitioners and contribute to the dissemination of G-CBT. Classifying therapist behaviors in this manner not only increases the convenience of administering the scale but also clearly indicates priorities for skill acquisition for beginner practitioners and supervisors of G-CBT.
There are several limitations to this study. First, the video sessions used in the assessment were exclusively of mild-depression patients, and the reliability and validity of the G-CTS was not examined for G-CBT in other disorders. Second, G-CTS targets the only leader-therapist and does not include co-leaders. Third, there were substantial differences in the videos of Beginner and Advanced Groups - role plays were filmed in the beginner videos, and clinical scenes were filmed in advanced videos – so it may be inappropriate to compare the two. Fourth, since the therapists’ years of experience and ages were relatively proportional, the video evaluators may have been able to predict the years of experience from the therapists’ physical appearances. Lastly and most importantly, the validation process was limited to internal consistency, inter-rater reliability and predictive validity. We were not able to conduct factor analysis due to small sample size. Further verification with larger sample is needed.
Despite these limitations, our study is noteworthy since we developed a novel rating scale for G-CBT that is not specific to a certain disorder. The scale has solid behavioral anchors, which led to high reliability of the scale.
Future research implications include the following: First, the reliability and validity of the G-CTS may need to be verified in samples other than depression. It is also necessary to establish an evaluation system that can accommodate the evaluation of co-therapists. Finally, since therapists’ desirable behavior may be different among different cultures, cultural adaptation of behavioral anchors of the G-CTS may be needed.
Data Availability
Data are available from the authors for researchers who meet the criteria for access to confidential data.
Author Contributions
M.N., M.M., M.O., H.K., and D.F. designed the study. M.N., M.M., and D.F. wrote the manuscript. Data analyses were performed by M.N. under supervision of H.K., D.F., and M.O. provided clinical advice. M.N. and M.M. provided their sessions’ video data. M.N. and M.M. reviewed the literature. M.N., M.M., and M.O. rated the videos. All authors contributed to the article and approved the submitted version.
Funding
This project was funded by the grants issued by the Japanese Ministry of Health, Labor and Welfare to DF (grant number: 21GC1015) and Meiji Yasuda Mental Health Foundation.
Declarations of Interest
The authors declare that this research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Ethics Approval and informed consent
This study was reviewed and approved by the Ethics Committee of the National Hospital Organization Hizen Psychiatric Medical Center (approval number: 28-7). The participants provided their written informed consent to participate in this study.
Acknowledgments
We thank Tsuyoshi Akiyama, and study group of group cognitive behavior therapy for their advice on scale creation and for their cooperation to our research.
References
