A target product profile for electronic clinical decision support algorithms combined with point-of-care diagnostic test results to support evidence-based decisions during patient consultations by health workers =================================================================================================================================================================================================================== * Karell G. Pellé * Clotilde Rambaud-Althaus * Valérie D’Acremont * Gretchen Moran * Rangarajan Sampath * Zach Katz * Francis Moussy * Garrett L. Mehl * Sabine Dittrich ## ABSTRACT Health workers in low-resource settings often lack the support and tools to follow evidence-based clinical recommendations for diagnosing, treating and managing sick patients. Digital technologies, by combining patient health information and point of care diagnostics with evidence-based clinical protocols, can help improve the quality of care, the rational use of resources (humans, diagnostics and medicines) and save patient lives. The development of a target product profile for electronic clinical decision support algorithms (CDSAs) aimed at guiding preventive or curative consultations, and that integrate diagnostic test results will help align developer and implementer processes and specifications with the needs of end-users, in terms of quality, safety, performance and operational functionality. To identify characteristics for a CDSA, experts from academia, research institutions, and industry as well as policy makers with expertise in diagnostic and CDSA development, and implementation in LMICs were convened. Experts discussed the critical characteristics of a draft TPP which was revised and finalised through a Delphi process to facilitate consensus building. Experts were in overwhelming agreement that, given that CDSAs provide patients’ management recommendations, the underlying clinical algorithms should be available in human readable format and evidence-based. Whenever possible, the algorithm output should take into account pre-test disease probabilities, diagnostic likelihood ratios of clinical or laboratory predictors and disease probability thresholds to test and to treat. Validation processes should at a minimum ensure the CDSA are implementing faithfully the evidence-based algorithm they are based on (internal validation through clinical association and analytical validation). Additionally, clinical validation, bringing practice evidence about the impact of the CDSA use on health outcomes, was recognized as a good to have. The CDSAs should be designed to fit within clinic workflows, connectivity challenges and high volume settings. Data collected through the tool should conform to local patient privacy regulations and international data standards. KEYWORDS * Target Product Profile * Clinical Guideline * Algorithm * Clinical Decision Support System * Diagnostics * Point of Care ## INTRODUCTION Health workers at primary health care (PHC) have the greatest challenge in ensuring appropriate care for their communities. One way to support health workers is through the provision of clinical guidance in the form of clinical practice guidelines or clinical decision support algorithms (CDSA). Clinical practice guidelines are a set of recommendations on diagnostic and treatment modalities based on systematic review of evidence and assessment of the benefits and harms for the patients. The World Health Organisation (WHO) developed several simple clinical decision guidelines to support health workers in low- and middle-income countries (LMICs) in the evaluation and management of clinical problems, with the ultimate goal of improving quality of care[1]. This includes the Integrated Management of Childhood Illness (IMCI) designed in the 1990s for common diseases affecting children younger than 5 years of age[2], which was later adapted for use at the community as the integrated Community Case Management guideline (iCCM)[3]. Other guidelines have been developed and either take an integrated management approach similar to IMCI, such as the Integrated Management of Adolescent and Adult Illness (IMAI[4]), or a disease-specific approach (i.e. WHO’s dengue guidelines for diagnosis, treatment, prevention and control[5]). While the global health community awaits updated and more comprehensive clinical algorithms, the Fourth Industrial Revolution[6] is seeing developing economies “leapfrogging”[7] mobile technologies across multiple sectors and including for health. As a result, a plethora of digital clinical decision support systems (CDSS) have been implemented in LMICs. CDSS are defined as “digitised job aids that combine individual’s health information with the health worker’s knowledge and clinical protocols to assists health workers in making diagnosis and treatment decisions”[8]. It is well recognized that adherence to standardized clinical practice guidelines is a measurement of quality of care[9,10], and that improving adherence to evidence-based guidelines is associated with better health outcomes[11]. By digitalizing clinical guidelines such as IMCI and iCCM, into electronic CDSAs in its simplest form, a decision tree, several groups have shown significant increase in health worker adherence to guidelines compared to routine care (paper-based IMCI)[12, 13,14]. However, based on IMCI/iCCM, today these CDSAs carry the same limitations than the main guidelines, and still lack point of care (POC) diagnostic tests that could improve the accuracy of diagnoses and treatment recommendations. The only diagnostic test widely used with IMCI/iCCM guidelines is a malaria rapid diagnostic test (mRDT) and while this inclusion was important in the context of “test-and-treat”[15], it is limited to ruling in or out one pathogen alone. Acute febrile illness can be caused by bacterial or viral respiratory infections, or seasonal, geographically localised pathogens, like dengue. Hence, the inclusion of additional simple wet and digital diagnostics into adapted algorithms can reduce antimicrobial over-prescriptions as well as patient outcome[16, 17]. With the growing development of CDSAs, especially related to the management of sick children in LMICs (**Supplemental File 1**), it is important to ensure that new tools meet the needs of the end-user and take into consideration the emerging landscape of diagnostic tools that might be included in new algorithms forming a toolkit consisting of CDSA and diagnostics. The aim of this work was to define a target product profile (TPP) to inform developers and implementers alike of the requirements for CDSAs aimed at guiding health workers during a preventive or curative consultation with any type of patient, and that have the ability to incorporate diagnostic test results, especially when diagnostics tests have been implemented in pre-existing guidelines. ## 1. METHODS ### 1.1. Draft Target Product Profile A draft TPP (version 0) for CDSAs combined with POC test results was developed (by: KP, SD, RS, ZK, GLM, FM) based on standard procedure at FIND and WHO, where characteristics are defined as “minimal” and “optimal” criteria (**Supplemental File 2**). “Minimal” is used to refer to the lowest acceptable output for a characteristic and “optimal” for the ideal target of a characteristic. Comments are used to explain or provide examples for the characteristics. As this TPP is for a multi-component toolkit, CDSA and POC diagnostics, the structure of the TPP was adapted to cover the different components and for the purpose of clarity. These sections included: Scope General, Scope Toolkit, Electronic Clinical Decision Algorithm, Point of Care Tools, App, Data and Procurement. ### 1.2. Expert Meeting To discuss selected aspects of the draft TPP (algorithm validation, performance, machine learning, diagnostic data, disease prediction, clinical workflow and application functionality) (**Supplemental File 2**), a meeting on this subject was convened by FIND and WHO involving 39 experts from academic institutions, industry, private and public sectors from 11 countries[18]. Experts were selected based on their experience in relevant areas of work directly related to digital health (clinical decision support or other digital tools targeting resource-challenges settings) and/or diagnostics (wet and digital). Following the expert meeting the TPP was updated based on expert feedback (version 1.0) (**Supplemental File 3**). ### 1.3. Delphi Process To facilitate consensus building for the TPP, we followed a Delphi-like process where the TPP version 1.0 was reviewed. For this, a three-part online survey was developed. The first part of the survey collected professional information for survey respondents as well as experience with clinical practice/research, diagnostic development, software development, implementation of healthcare programs in LMICs, and health information systems. The second part of the survey allowed for rating of both minimal and optimal criteria for all TPP characteristics. Agreement was scored on a Likert scale from 1-5, where 1 corresponds to fully disagree, 2-mostly disagree, 3-neither agree nor disagree, 4-mostly agree and 5-fully agree. Disagreement with a criterion was based on a rating from 1 to 3 and required a comment/suggestions by the survey respondent. To reach consensus, more than 75% of survey respondents should provide a rating of at least 4 (agreement) on the Likert scale. The third part of the survey provided an opportunity for respondents to share comments/feedback/ideas on the TPP if not covered in the characteristics section. The survey was developed with the online software SurveyGizmo (SurveyGizmo LLC, Boulder CO, USA). Survey analyses were performed using Microsoft Excel 2013 (Microsoft Corporation, Redmond, USA). #### Patient and public involvement statement At no point in this work were patients or the public involved in the design of this study or reporting and dissemination of the results. ## RESULTS ### 1. Expert Consensus on the Target Product Profile through a Delphi Process In the first round, the online survey was sent to 77 experts (including workshop participants) and 28 responded to the survey (response rate, 36.4%), with some respondents submitting a coordinated response for their institution. Twenty-five percent of respondents were from LMICs (7/28), however the majority of respondents, (22/28, 78%), had experience working in Africa and 57% (16/28) in South East Asia. More than 85% of respondents (24/28) had more than 5 years of experience in clinical practice and research, 50% (14/28) in health programme implementation in LMICs, 39% (11/28) in health information systems, 21% (6/28) in software development and 14% (4/28) in diagnostic product development **(Supplemental File 4)**. Most participants work in international organizations (7/28, 25%) and academic institutions (10/28, 35.7%). In the first round, at least 50% agreement was reached for all criteria for all characteristics and feedback was taken into consideration to develop TPP version 1.1 for the second round of the Delphi survey (**Supplemental File 5**). In this new version, the characteristic “Encounter” was removed as several participants expressed it was redundant with the characteristic “Workflow”. Therefore version 1.1 had 40 characteristics. A second online survey was sent to the 28 respondents from round 1. In this round, 23 participants responded to the survey (response rate, 82%). Most participants were from academia (7/23, 30%), non-profit international organizations (7/23, 30%) and had a similar profile than respondents from round 1 (**Supplemental File 4**). In this round of review, at least 75% of participants agreed with the minimal and optimal criteria for all TPP characteristics. On average, 91% of survey respondents agreed with the minimal and optimal criteria (**Figure 1**) and the TPP was finalized (version 2). ![Figure 1.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2019/10/29/19008433/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2019/10/29/19008433/F1) Figure 1. Target product profile second and final survey results ### 2. Final Target Product Profile #### Scope General and Scope Toolkit Components This TPP defines a toolkit composed of CDSAs and POC diagnostics intended to support evidence-based clinical decisions taken by health workers during a preventive or curative consultation, by capturing patient clinical and context-specific data, as well as diagnostic test results to provide recommendations on diagnosis and patient management (including preventive & curative treatment, counseling, etc.). The clinical algorithm integrates POC diagnostic results and is embedded in an App. POC diagnostics in this toolkit should be regulatory approved for use in the setting of implementation. The experts agreed that the CDSA should first define its target i.e. i) patients (e.g. age group), ii) medical problems (e.g. syndrome or disease), iii) end users (e.g. minimally trained health worker or trained clinician) and iv) level of care (e.g. community or health facility) for which the clinical recommendations are meant and relevant. These specifications should be clearly communicated to the end-user at the beginning of the algorithm. The health worker should immediately be warned if information she or he enters in the system does not fit with these specifications. In cases where an App proposes different algorithms for different use cases (i.e. antenatal care, management of childhood illness), the specifications shall be clearly available for each individual algorithm (**Table 1**). View this table: [Table 1:](http://medrxiv.org/content/early/2019/10/29/19008433/T1) Table 1: Minimal and optimal target product profile characteristics focused on the scope of the toolkit and toolkit component characteristics, as defined by expert consensus process. #### Clinical Decision Support Algorithms Regarding the algorithm medical content, the working group agreed that it must be evidence-based. Whether newly developed or adapted from existing guidelines, the algorithm medical content should be developed following international standards for the development or adaptation of evidence-based clinical guidelines, to ensure it implements the best available evidence. This evidence can originate from well-established clinical guidelines such as WHO/international/local clinical care guidelines, or from peer-reviewed articles, clinical practice and/or validation research and the level of evidence and the strength of recommendations in the target context should ideally be communicated. In line with this, recommendations should cause the least harm to the patient and to the community. For example, algorithms related to treatment and management of patient suffering from infectious diseases should also support antimicrobial stewardship to avoid unnecessary side effects for the patient and the development of resistance in the community and at global level. In addition, the algorithm should be validated both analytically and semantically to ensure that the algorithm output is accurate and reproducible, doesn’t deviate from expert content/evidence and that there are no interactions or conflicts in the logic. Further, important characteristics were content quality, machine learning (ML). With the rise of ML also in this domain of CDSAs to analyse the data captured through algorithms and then for example augment algorithm diagnostic or prognostic accuracy, workshop experts thought it important to address this in the TPP. It was agreed that an algorithm that recommends treatment decisions and management of patients should be at a minimum human interpretable, meaning one should understand how input data are processed into output data. Changes to the algorithm logic based on ML analyses should also be validated (**Table 2**). View this table: [Table 2:](http://medrxiv.org/content/early/2019/10/29/19008433/T2) Table 2: Minimal and optimal target product profile characteristics focused on the electronic clinical decision-support algorithm and point of care tools, as defined by expert consensus process. #### Point of Care Tools Experts agreed that the post-test probability or the likelihood of a disease being present, should be assessed based on the pre-test probability—the prevalence of the disease in the corresponding patient population—and the diagnostic positive/negative likelihood ratio—how good a test predicts the presence or absence of disease—of all clinical information i.e. signs, symptoms, and POC diagnostic test results (**Table 2**). Clinical diagnosis often lacks accuracy and therefore diagnostic tests that have the proven added value for patient management should be integrated within algorithms. These tests can be wet diagnostics such as pathogen-specific RDTs or hand-held devices such as haemoglobinometers, and do not exclude digital diagnostics such as digital thermometers. #### App An important characteristic in this section is System Validation. Today, there is no standard process to validate and assess performance accuracy of CDSAs. There is also a severe lack of evidence and peer-reviewed publications on both the development and the performance of these tools in LMICs. While some groups have performed randomised controlled trials (RCTs) to measure patient outcome and other parameters that impact patient health, such as the reduction of antibiotic over-prescription[16, 17], others have assessed the performance of their tool by measuring health worker adherence to guidelines[12,13,14]. The level of validation ultimately depends on the type of algorithm. The USFDA has provided some guidance to developers and implementers on the kind of evidence that needs to be provided[19]. The TPP criteria are inspired by this guidance. The minimal criteria guarantee that the App has a CDSA that is based on evidence (clinical association) and that input data is processed correctly (analytical validation). The optimal criteria include clinical validation in addition to clinical association and analytical validation. Through clinical validation, the App has been assessed for the intended outcome (i.e. improved clinical outcome, better quality of care, rational use of resources and reduction of antibiotic over-prescription, etc.) (**Table 3**). View this table: [Table 3:](http://medrxiv.org/content/early/2019/10/29/19008433/T3) Table 3: Minimal and optimal target product profile characteristics focused on App characteristics, as defined by expert consensus process. In May 2017, a new regulation (EU 2017/745) on medical devices, to which CDSA pertain by definition[20] was adopted by the European Commission, that will be applied starting spring 2020[21]. This regulation includes a reinforcement of the rules on the clinical evidence that should be provided by developers and manufacturers, at least if based in Europe. While waiting for LMICs to have their own regulations, this new regulation will certainly impact development and validation by European developers. Experts also called for Application Programming Interface (API) to allow other software to ‘talk to’ and interact with the tool through commonly known programming languages. The App should be protected by authorisation and authentication that as a minimum requirement adhere to usual technical standards, and optimally would adhere to HIE (Health Information Exchange) and/or Health Level 7 International[22] medical standards. It was noted that adherence to HL7 can be expensive, hence this was proposed as an optimal rather than a minimal requirement. The App should also be designed to integrate with patient care pathways and health worker workflows with limited disruption. The CDSA accordingly has pause and resume capability to allow time for diagnostic tests to be performed and results entered in the algorithm. POC diagnostic tests such as mRDTs can require up to 20 min for results to be read, depending on the manufacturer. #### Data Digital tools have the potential to amass enormous amount of data. With CDSA, patient personal data is captured, stored and potentially used for other operational (i.e. supply chain management) and algorithm development purposes. Many countries have personal data privacy and protection laws in place to protect citizens from the abuse of their personal information. In the European Union this is known as the General Data Protection Regulation[23]. However, there are still many countries around the world that do not have a national eHealth strategy or personal data legislation. This called for the need to address personal data handling with CDSA for security and accountability purposes. Experts agreed that the implementer, who would be considered as the data controller, is responsible for the data and should comply with local data policies and legislations. No specific legislation was mentioned as these differ by country and region. In addition, the historic of the processes that are applied to data as well as the origin of data should be documented (Data Provenance) and the App should function under secure connectivity to avoid loss and corruption of sensitive data, and mitigate cyber-attacks, whether data is at rest or in transmission **(Table 4)**. View this table: [Table 4:](http://medrxiv.org/content/early/2019/10/29/19008433/T4) Table 4: Minimal and optimal target product profile characteristics focused on data characteristics, as defined by expert consensus process. ## DISCUSSION In this work, our aim was to develop a TPP to enable the development of a toolkit that includes CDSAs that are based on evidence and designed to integrate results of diagnostics performed at POC to guide rational clinical decisions. The use of such toolkit would aid in strengthening healthcare particularly at the community and primary care echelons, and help address health workers difficulties to differentiate diseases and syndromes without diagnostic tests and the aid of formalised diagnostic processes that can be provided through algorithms. In terms of TPP characteristics, “Data” and “Validation” emerged as key topics of discussion. Data generated from CDSA require storage and connectivity and are therefore vulnerable to abuse, loss or unlawful purposes. Although many countries have enacted personal data protection legislation, many fall short on enforcing these and others are still drafting bills. For example, in the African region, in 2017, 17 (out of 54) African countries had put in place comprehensive personal data protection legislation[24]. Several regimes in the region are starting to draft legislation regarding processing and movement of personal data[25], however funding attrition and the lack of regulators cripple the process. That said, there are continent-wide initiatives that promise to set data protection framework such as EU’s GDPR. The African Network of Personal Data Protection Authorities (RAPDP), a cooperation of 8 African countries seeks to draft data protection laws, formulate opinions on specific issues and establish a consultative framework on data protection. There are also regional conventions for data protection and privacy that have been enacted such as the Southern African Development Community (SADC) Model Law[26] for the south and the Act on Personal Data Protection within ECOWAS (Economic Community of West African States) for the west[27]. Unlike their southern and western counterparts, east African nations have not adopted similar regional frameworks, which can set cross-border and data portability limitations. In addition, some countries national laws fall short in setting the required safeguards for data privacy breach and data portability and lack bodies such as Data Protection Authorities to enforce these legislations[24]. This is just the African scenario, but every country and region will have their own set of policies. Therefore, the healthcare programme and data controllers have an important responsibility to manage the programme in compliance with local laws or, where lacking, consult with government authorities on which regulation to abide by to foremost protect patient rights. Regulation of CDSS is also very contentious topic. Groups will have to watch this space, FDA and European Commission regulations, closely to determine whether their tool fits the definition of a Software as a Medical Device (SaMD) and what data they will need to generate for regulatory approval. In terms of “Validation”, the big question was how to ensure CDSAs are safe for use. Developers and implementers should aim for clinical association, analytical validation and clinical validation. However clinical validation can require significant amount of financial resources and time. As an example, e-POCT was developed in 2014 to incorporate the latest scientific evidence to expand the medical content of IMCI. Its safety and efficacy was determined in a RCT in Tanzanian outpatient clinics[17]. However, in 2019 the tool has yet been implemented at scale or evaluated in non-controlled settings. Efforts like e-POCT to optimise CDSA content and assess its impact on patient health outcomes are needed to grow the evidence for CDSA’s role in improving patient safety. But providing these tools to patient also demands that we put our efforts in studies to measure the effectiveness of CDSA while in the hands of the end-user and in the setting of intended use. Indeed, there may be factors when CDSAs are implemented as part of routine practice that affect the overall effectiveness and impact of the tool. This includes a lack of or poor health worker training, lack of connectivity, software malfunctions, poor design leading to low usability and acceptability of the tool by health workers. Cycles of evaluation and improvement can be sought to perfect these tools in intended settings. In addition, digital data obtained from CDSA validation can be used in turn to dynamically improve the tool itself faster than development improvements in classic wet diagnostics. CDSA also have the potential to serve as platforms to support the evaluation or development of new content or guidelines. These can be faster adapted to geographies to conform to local policies. The call for the development of PHC during the declaration of Astana revived the global commitment to provide quality health services to all[28]. Achieving universal health coverage will therefore require that digital health interventions are quality assured and evidence-based. Indeed, the WHO has begun developing frameworks to create a common digital language and synthetizing the evidence around emerging mobile-based digital health technologies through two recent publications[29, 88]. These resources are meant to guide the global health community in assessing digital interventions that will improve quality of care, meaning interventions that are safe, effective, timely, efficient, equitable and people-centred, and this include CDSAs[30]. The management of patients in low-resource settings with complex epidemiology is extremely challenging without clinical algorithms and accurate and portable rapid diagnostic tests. Consequences of suboptimal quality of care can go even beyond the patient’s health to include public health considerations such as the development of antimicrobial resistance due to the over-prescribing of medicines. The aim of the proposed TPP is to support developers and implementers of these toolkits aimed at guiding health workers throughout clinical assessment, to give them confidence in clinical decisions and actions, whether it is sending a patient to a hospital or not, prescribing or not prescribing antibiotics or recommending rehydration at home. ## Data Availability All data are included in supporting files. ## FUNDING This work was funded by the Fondation Botnar and supported by the Global Antimicrobial Resistance Innovation Fund (GAMRIF), a UK aid programme. The funders had no role in the study design, data collection and analysis, or preparation of the manuscript. ## ACKNOWLEDGEMENTS We would like to thank all the experts who participated in the FIND/WHO workshop to discuss the first draft of the TPP, as well as Rigveda Kadam, Wallace White and Mo Tobin (FIND) for discussions on TPP development. We would also like to thank the online reviewers who provided feedback to develop the final consensus-driven TPP characteristics. * Received October 8, 2019. * Revision received October 21, 2019. * Accepted October 29, 2019. * © 2019, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial 4.0 International), CC BY-NC 4.0, as described at [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/) ## REFERENCES 1. Tunçalp ⍰., Were WM, MacLennan C, et al. Quality of care for pregnant women and newborns—the WHO vision. BJOG 122(8):1045–9. doi: 10.1111/1471-0528.13451. Epub 2015 May 1. [http://onlinelibrary.wiley.com/doi/10.1111/1471-0528.13451/full](http://onlinelibrary.wiley.com/doi/10.1111/1471-0528.13451/full) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/1471-0528.13451&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25929823&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F10%2F29%2F19008433.atom) 2. World Health Organisation. Integrated Management of Childhood Illness (WHO) 2014 [May 2019]. Retrieved from [http://apps.who.int/iris/bitstream/10665/104772/16/9789241506823\_Chartbook_eng.pdf](http://apps.who.int/iris/bitstream/10665/104772/16/9789241506823_Chartbook_eng.pdf). 3. World Health Organisation. Integrated management of childhood illness: caring for newborns and children in the community. 2011. World Health Organization. [May 2019]. Retrieved from [http://www.who.int/iris/handle/10665/44398](http://www.who.int/iris/handle/10665/44398) 4. World Health Organisation. Integrated Management of Adolescent and Adult Illness (IMAI). [May 2019]. Retrieved from [http://www.who.int/3by5/publications/documents/imai/en/](http://www.who.int/3by5/publications/documents/imai/en/). 5. WHO and TDR. Dengue: Guidelines for diagnosis, treatment, prevention and control: new edition. 2009 (WHO/HTM/NTD/DEN/2009.1). [May 2019]. Retrieved from [https://www.who.int/tdr/publications/training-guideline-publications/dengue-diagnosis-treatment/en/](https://www.who.int/tdr/publications/training-guideline-publications/dengue-diagnosis-treatment/en/) 6. Klaus Schwab (2015). The Fourth Industrial Revolution: what it means, how to respond. [May 2019]. Retrieved from [https://www.foreignaffairs.com/articles/2015-12-12/fourth-industrial-revolution](https://www.foreignaffairs.com/articles/2015-12-12/fourth-industrial-revolution). 7. Goldemberg José. Leapfrog energy technologies. Energy Policy 1998; 26(10):729–741. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0301-4215(98)00025-1&link_type=DOI) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000076354000001&link_type=ISI) 8. World Health Organisation guideline: recommendations on digital interventions for health system strengthening. Executive summary. Geneva: World Health Organization; 2019. (WHO/RHR/19.8). Licence: CC BY-NC-SA 3.0 IGO. 9. Darling G. The impact of clinical practice guideline and clinical trial on treatment decisions. Surg Oncol. 2002 Dec; 11. 10. Armstrong Schellenberg J, Bryce J, de Savigny D, et al. The effect of Integrated Management of Childhood Illness on observed quality of care of under-fives in rural Tanzania. Health Policy Plan. 2004;19 11. Rowe AK, Labadie G, Jackson D, et al. Improving health worker performance: an ongoing challenge for meeting the sustainable development goals. BMJ 2018;362:k2813. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjIvanVsMjdfNy9rMjgxMyI7czo0OiJhdG9tIjtzOjM5OiIvbWVkcnhpdi9lYXJseS8yMDE5LzEwLzI5LzE5MDA4NDMzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 12. Mitchell M, Hedt-Gauthier BL, Msellemu D, et al. Using electronic technology to improve clinical care - results from a before-after cluster trial to evaluate assessment and classification of sick children according to Integrated Management of Childhood Illness (IMCI) protocol in Tanzania. BMC Med Inform Decis Mak. 2013;13:95. Published 2013 Aug 27. doi:10.1186/1472-6947-13-95 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6947-13-95&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23981292&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F10%2F29%2F19008433.atom) 13. Bernasconi A, Crabbé F, Raab M, et al. Can the use of digital algorithms improve quality care? An example from Afghanistan. PLOS ONE 13(11): e0207233.[https://doi.org/10.1371/journal.pone.0207233](https://doi.org/10.1371/journal.pone.0207233) 14. Sarrassat S, Blanchet K, Lewis J et al. Integrated eDiagnosis Approach (IeDA) for the management of illness in under-five children at the primary health care level in Burkina Faso: Findings from a stepped-wedge cluster randomised trial. 15. World Health Organization (2010). Guidelines for the Treatment of Malaria. 2nd edition. Geneva, Switzerland: World Health Organization. 16. Shao AF, Rambaud-Althaus C, Samaka J, et al. New Algorithm for Managing Childhood Illness Using Mobile Technology (ALMANACH): A Controlled Non-Inferiority Study on Clinical Outcome and Antibiotic Use in Tanzania. PLoS One. 2015 Jul 10;10(7):e0132316. doi: 10.1371/journal.pone.0132316. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0132316.&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26161535&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F10%2F29%2F19008433.atom) 17. Keitel K, Kagoro F, Samaka J, et al. A novel electronic algorithm using host biomarker point-of-care tests for the management of febrile illnesses in Tanzanian children (e-POCT): A randomized, controlled non-inferiority trial. PLoS Med. 2017 Oct 23;14(10):e1002411. doi: 10.1371/journal.pmed.1002411. eCollection 2017 Oct. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pmed.1002411&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F10%2F29%2F19008433.atom) 18. FIND and WHO. Electronic clinical decision support algorithms_ Workshop report. 2018. Available from: [https://www.finddx.org/mal-fev/ecdm-toolbox/](https://www.finddx.org/mal-fev/ecdm-toolbox/) 19. U.S. Food & Drug Administration. “Software as a Medical Device (SAMD): Clinical Evaluation. Guidance for Industry and Food and Drug Administration Staff”. December 8, 2017. [October 2018]. Retrieved from [https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM524904.pdf](https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM524904.pdf) 20. European Commission. Guidance document Medical Devices – Scope, field of application, definition – Qualification and Classification of standalone software – MEDDEV 2.1/6. [May 2019]. Retrieved from [http://ec.europa.eu/DocsRoom/documents/17921/attachments/1/translations](http://ec.europa.eu/DocsRoom/documents/17921/attachments/1/translations) 21. European Commission. (2017) Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance)Text with EEA relevance. [July 2019] Retrieved from [http://data.europa.eu/eli/reg/2017/745/2017-05-05](http://data.europa.eu/eli/reg/2017/745/2017-05-05) 22. HL7 International. Retrieved from [https://www.hl7.org/](https://www.hl7.org/) 23. General Data Protection Regulation (GDPR). [May 2019]. Retrieved from [https://gdpr-info.eu/](https://gdpr-info.eu/) 24. Cynthia Rich. A look at New Trends in 2017 : Privacy laws in Africa and the Near East (16) 6 Blomberg BNA. [May 2019]. Retrieved from [https://media2.mofo.com/documents/170911-privacy-africa.pdf](https://media2.mofo.com/documents/170911-privacy-africa.pdf) 25. UNCTAD. Data Protection and Privacy Legislation Worldwide. [May 2019]. Retrieved from [https://unctad.org/en/Pages/DTL/STI\_and\_ICTs/ICT4D-Legislation/eCom-Data-Protection-Laws.aspx](https://unctad.org/en/Pages/DTL/STI_and_ICTs/ICT4D-Legislation/eCom-Data-Protection-Laws.aspx). Accessed May 14 2019. 26. ITU. (2013) Establishment of harmonized policies for the ICT market in the ACP Countries. Data protection: Southern African Development Community (SADC) Model Law. [May 2019]. Retrieved from [https://www.itu.int/en/ITU-D/Projects/ITU-EC-ACP/HIPSSA/Documents/FINAL%20DOCUMENTS/FINAL%20DOCS%20ENGLISH/sadc\_model\_law\_data\_protection.pdf](https://www.itu.int/en/ITU-D/Projects/ITU-EC-ACP/HIPSSA/Documents/FINAL%20DOCUMENTS/FINAL%20DOCS%20ENGLISH/sadc\_model_law_data_protection.pdf) 27. ECOWAS (2010). [May 2019]. Supplementary Act A/SA.1/01/10 on Personal Data Protection within ECOWAS. [https://www.statewatch.org/news/2013/mar/ecowas-dp-act.pdf](https://www.statewatch.org/news/2013/mar/ecowas-dp-act.pdf) 28. The Lancet. The Astana Declaration: the future of primary health care? Lancet 2018;392(10156):1369. doi.org/10.1016/S0140-6736(18)32478- 29. Classification of digital health interventions. Geneva: World Health Organization; 2018 (WHO/RHR/18.06). Licence: CC BY-NC-SA 3.0 IGO 30. World Health Organisation. Quality of Care: A Process for Making Strategic Choices in Health Systems. Geneva: WHO, 2006. [July 2019]. Retrieved from [https://apps.who.int/iris/handle/10665/43470](https://apps.who.int/iris/handle/10665/43470)