Catchii: empowering literature review screening in healthcare

Application design and selection of features

We designed Catchii to include all the features of importance in SR software defined in the literature screening tool comparison analysis [2]. This article rates the importance of features into three categories: mandatory, desirable and optional. The features are related to the technical functioning of the tool and include: support for importing references (M1), multiple users (M2), the ability to (re-)allocate references (M3), having distinct title-abstract (TiAb) and full-text phases (M4), including or excluding references (M5), resolving discrepancies (M6), exporting results (M7), having a stable release of the software (M8) and the level of customer support (M9). The desirable and optional features are not critical for a SR, but they can assist the process. Features such as support for non-Latin characters (D1), multiple different user roles (D2), randomising the order of references (D3), keyword highlighting (D4), attaching PDF (Portable Document Format) files (D5), attaching comments (D6), displaying project progress (D7), project auditing (D8), and being free to use (D9) were considered desirable, while reference labelling (O1), some form of machine learning/automation (O2) and flow-diagram creation (O3) were considered optional. An acceptable threshold was assigned for each feature (for example, a “distinct TiAb and full-text” feature has the following options: “no distinct phases”, “TiAb & full-text phase” and “user defined phases” and acceptable threshold is “TiAb & full-text phase”). In the following sections the Catchii implementation of each feature at acceptable threshold is discussed.

Implementation of features

Catchii is a systematic literature review tool that fully supports reference importing (M1). Users can easily import references in different formats, including files created by PubMed, Web of Science, OVID (e.g. PsycINFO, MEDLINE), Scopus, a general RIS (Research Information Systems) file created by online databases or reference manager tools, as well as a CSV (Comma Separated Values) file. For each uploaded set, a dataset label can be specified for the contained references (O1). Additionally, Catchii supports articles written using various characters (D1), including Latin, Cyrillic, Chinese, Japanese, Arabic and others.

Each user can perform screening for an unlimited number of reviews, and for each review, multiple users are supported (M2). Each reviewer is blind to others’ decisions until everyone has made their decision for a particular article. Multiple roles for reviewers (D2) are also supported, including a manager of the review (with full access to the review), a reviewer (with rights to make inclusion/exclusion decisions but cannot resolve discrepancies), a reviewer with the rights to also resolve discrepancies, and a reviewer who only has rights to review and resolve discrepancies (if there are any).

In the event that a reviewer drops out, references from that user can be re-allocated (M3) to a new user. Catchii has an easy system for adding, removing, and replacing reviewers from a review. A replacement reviewer can be added to the review at any time including before other reviewers have finished the title-abstract screening.

Catchii also has a built-in duplicate detection feature which compares the title or abstract of all articles published in a particular year by using the similar_text() function in the “PHP: Hypertext Preprocessor” language to calculate similarity in percentages between two text strings. Articles that have an abstract or title that is at least 80% similar are returned to the user for review. By default, all duplicate records except one are marked and can be collectively removed with one click. In the case of a missing abstract, only titles are compared. We consider this feature as an automation feature (O2), as users do not have to manually find duplicate records but can automate the task. However, from the Van der Mierden article it is unclear which automation features are included in this category [2].

Catchii also supports distinct TiAb and full-text phases (M4). In both phases, reviewers must decide whether to include or exclude a particular article (M5). In the full-text phase, a reason for exclusion can be recorded by the reviewer. Each phase is divided into two parts: in the first part, inclusion/exclusion decisions are made, and in the second part, the decisions are confirmed and discrepancies are resolved (M6). Technically, a decision for a disagreement between reviewers can be made by the manager of the review, a reviewer with rights to solve disagreements, or an additional reviewer who only has rights to solve disagreements and does not take part in the screening process. In the screening process, each reviewer can choose the order of records: either sort them by year, title, dataset label, or list them in a randomized order (D3).

To enhance the efficiency of title and abstract assessment, Catchii has incorporated a keyword highlighting feature (D4). In contrast to most systematic review tools, Catchii supports multiple groups of keywords and allows for the assignment of distinct colours to differentiate them. Conventional systematic review tools typically utilize two sets of keywords – inclusion and exclusion, which are highlighted in green and red, respectively. This approach can limit the acquisition of information and may cause difficulties for individuals with red-green colour blindness. By utilizing keyword groups, reviewers are able to assign different colours for inclusion or exclusion keywords. For example, when the inclusion keywords are “Sun,” “UV,” “cancer,” and “humans,” Catchii can assign distinct colours for each keyword, allowing for quick visual acquisition. We believe that this improves the assessment of abstracts and ultimately accelerates the screening process. The colours used are chosen from a range of the colour spectrum and are suitable for individuals with different forms of colour blindness. Additionally, reviewers may also have the ability to collect papers for other purposes during the screening process, which may not be included in the review but are still valuable to the reviewer. By specifying a keyword group to identify these papers, a reviewer can then add them as a favourite (an additional feature of Catchii) and exclude from the screening process, but later return to the list of favourite articles.

In the full-text phase, Catchii supports the attachment of PDFs (D5) both manually and through automatic download from publicly available resources on the Internet, such as PubMed Central. Uploaded PDFs will be accessible to all reviewers. If necessary, each reviewer can add comments (D6) to the records that can be made visible to other reviewers or kept private for the writer. A detailed project progress (D7) is displayed for each project/review, separated by phases and sub-phases. The progress of each reviewer can be also viewed by other co-reviewers on the review.

Upon completion of the screening, records can be fully exported (M7) into a reference manager software or as a CSV file. PDF files can be added to the exported references and opened in a reference manager software, making it easy to perform data extraction on the user’s computer. A PRISMA flowchart can be generated (O3) as an editable document, allowing reviewers to make any necessary corrections. Catchii has a built-in logging system where each decision and update in a review is recorded. For auditing purposes, the manager of the review can download an audit log (D8) to review the actions of each reviewer.

Finally, Catchii is a free-to-use tool (D9) with a stable release (M8) that is actively supported. There is a short documentation, a forum, and a direct support option for users (M9). Additionally, Catchii has other features not previously outlined in the requirement from Van der Mierden. Specifically, Catchii supports mobile devices, free-text data extraction, the ability to bookmark articles into a favourites list, and an offline screening mode that enables screening of records even without an internet connection, such as on airplanes or in areas with an unstable connection.

Duplicate detection

We evaluated the performance of duplicate detection of Covidence and Rayyan, tools which are one of the most popular and have been recommended [2, 8]. Both tools were compared to Catchii, in two ways. Firstly, we assessed their sensitivity by counting the number of duplicates detected. We conducted a search for records containing the term “7,8-DHF” in PubMed, Scopus, and Web of Science databases, and selected all records published between 2010 and 2020 (inclusive). A total of 390 records were downloaded (130 in PubMed, 125 in Scopus, and 135 in Web of Science) [11]. We manually reviewed the records and identified 246 duplicates in the combined dataset. Then, we imported the records into Rayyan, Covidence, and Catchii, and counted the number of duplicates detected by each tool.

Catchii missed four records out of 246 (98.4% sensitivity) - one with a different publication year and three records with a large block of text added to the abstract that resulted in a low similarity percentage, below the 80% threshold. Catchii showed higher sensitivity than Covidence, which missed eight records (94.3%), but had lower sensitivity than Rayyan, which found all duplicates (100%). However, while Rayyan did not miss any duplicates, their system requires manual resolution of duplicates, which is a time-consuming task. For example, resolving 100 records with one duplicate each would require 300 clicks (select the record, click on a link to resolve the duplicate and click to remove the chosen duplicate). In Covidence, duplicates are removed automatically after import, with the review of removed duplicates taking place afterwards. In Catchii, all duplicates can be removed with one click after reviewing them.

Secondly, we evaluated the duplicate detection capabilities of each tool. For this purpose, we downloaded six records and modified them manually to test various scenarios [11]. We tested case sensitivity by converting one record’s abstract and title to uppercase, altered the abstract (added a short copyright notice in the end), created duplicates with more than two instances of the same record, removed abstract of one record (titles were identical), removed abstract of one record and altered title (added “[review]” in the end), and the use of non-Latin characters (title and abstract both in Mandarin Chinese).

Of the three tools, only Catchii was able to accurately identify all types of duplicates (6/6). Both Rayyan and Covidence missed the duplicate that had a missing abstract and a small alteration in the title (5/6). The results of both experiments are presented in Table 1.

Feature comparison

The systematic review screening tools comparison article by Van der Mierden evaluated available software based on the number of features they offer in three categories [2]. We calculated the score for Catchii by taking into account the number of features with an acceptable threshold described previously and combining it with the results of the Van der Mierden article. Catchii received a score of 20 points (21, if duplicate detection and removal is considered automation), which is the same or higher than the highest-scoring tool, DistillerSR. Compared to Catchii, DistillerSR has one fewer desirable feature but one more (or the same) optional features. As shown in Table 2, among the top-scoring SR tools, Catchii is the only one that is completely free for the user.

Additionally, we calculated a score for the Systematic Review Accelerator (SRA) tool [12], which was published after Van der Mierden’s article and thus was not included in the comparison. The SRA toolbox is not an all-in-one screening application, but more a set of tools (Deduplicator, Screenaton and Disputatron) that can be used individually to perform deduplication, TiAb screening and resolving disagreements between reviewers (respectively), which can be evaluated based on their features. The following features with an acceptable threshold were found for the SRA: M1, M2, M3 (yes/no; unsure), M6, M7, M8, M9, D1, D4, D9, and O2 (Deduplicator), resulting in a total score of 10 or 11, and thus it was not included in the comparison table with the top five tools.