The process of developing and piloting a tool in the Maldives and Zimbabwe for assessing disability inclusion in health systems performance

2.1 The Missing Billion Framework for a disability-inclusive health system

The Missing Billion framework of a disability-inclusive health system was put forward by two authors (HK and PH), based upon the WHO’s building blocks framework for “strengthening health systems to improve health outcomes”(WHO, 2007), and the Primary Health Care Performance Initiative (PHCPI) Conceptual Framework (Veillard et al., 2017). These frameworks were chosen because they are widely used and were considered potentially appropriate to assess the performance of the health system in terms of disability-inclusion. The WHO’s Building Blocks framework describes six building blocks needed for a strong health system: service delivery; health workforce; information; medical products, vaccines and technologies; financing; leadership and governance (WHO, 2007). The PHCPI framework uses many of the same components as the Building Blocks framework, but emphasises the relationship between, and hierarchy of, the different components (e,g, health financing influences the health workforce and access to drugs and supplies) (PHCPI, 2018; Veillard et al., 2017). It also introduces a greater focus on outputs and outcomes and the “Social determinants and context”. The frameworks were adapted to focus on people with disabilities, by including a further focus on the patient perspective given the specific and additional difficulties that people with disabilities face in accessing healthcare (WHO, 2022), including additional components in relation to disability-specific aspects of the health system (e.g. access to assistive technology - AT, existence of laws protection the right to healthcare of people with disabilities), and reducing some of the details on other components (e.g. PHCPI supportive management: formal training).

The Missing Billion Disability-Inclusive Health System Framework includes 4 system-level components and 5 service delivery components (2 on the demand side, 3 on the supply side) (Figure 1, Table 1). The goal of the framework is to reduce the life expectancy gap experienced by people with disabilities by improving the performance of health systems for people with disabilities. Improving performance with respect to disability-inclusion in these 9 components is hypothesised to improve the output of this system (i.e. “Effective service coverage”) and thereby lead to the ultimate outcome of improved “health status” for people with disabilities.

• Download figure
• Open in new tab

Figure 1: The Missing Billion Health System Framework

The framework was presented to a range of experts through a series of individual meetings, including governmental stakeholders (Georgia, Ireland, Malawi, Ukraine, Poland), UN officials, health systems experts (PHCPI and Ariadne Lab), academics specialising in disability studies, and disability rights organizations (European Disability Forum). The sample included people with disabilities. These key informants were asked about the relevance of the nine components for creating disability-inclusive health systems, and suggestions were sought for improvements.

2.2 Development of indicators for a health system performance assessment tool on disability inclusion health system

Two authors (an epidemiologist – HK – and a global health system expert - PH) developed specific indicators to measure the level of disability-inclusion for each framework component.

First, we reviewed indicators proposed to monitor the implementation of article 25 (health) of the UN Convention on the Rights of Persons with Disabilities (UN, 2020), PHCPI and UHC (Veillard et al., 2017; WHO, 2017). We also reviewed a range of toolkits relevant to health system strengthening for people with disabilities to identify possible application and note good practice examples: Strategizing health in the 21^st century: a handbook (WHO, 2016), Toolkit frontline health (unpublished), Essential Management Package for Strengthening Physical Rehabilitation Centers (MSH, 2015), Autism toolkit for GPs (no longer available online), Learning Disability Annual Health Check Toolkit (NHS, 2021) and the rapid Assistive Technology Assessment (rATA) (WHO, 2019).

Next, we formulated 3-5 indicators per framework component; we aimed to increase the feasibility of data collection through limiting the number of indicators. We focussed on developing indicators where data would be available, based on our previous experience and internet searches (e.g. of policy documents).

The preliminary indicator set was reviewed by a lead academic specialising in collecting data on disability and a UN representative working on disability. They gave specific feedback on the perceived relevance of the indicator, wording of the indicators and feasibility of collecting the data. We also asked additional key informants about specific areas of assessment and/or sources of information within the context of the indicators. The indicators were adapted, based on the feedback obtained. In total, 34 indicators were developed to measure performance across the 9 health system components, as well as 14 for the output/outcome components.

2.3 Pilot-test in Maldives and Zimbabwe

Research partners were selected to pilot test the application of the indicator set in the Maldives (2020) and Zimbabwe (2021) to assess its feasibility. These settings were chosen as they are contrasting in nature and the context of ongoing research of our group, so that there were research partnerships and activities in place. The Maldivian research partner (SH) is a policy specialist, who led a national disability study in the country, which included a national survey, qualitative fieldwork and policy analysis. She also works closely with stakeholders from the MoH and disability groups. The Zimbabwean research partners (TK and TS) are academics who work closely within the Zimbabwean health, policy and legal system. In both settings, the health system performance assessment was undertaken within the context of a broader research study on disability.

The research partners aimed to collect data for all the indicators through reviewing policy documents and internet search of grey literature (e.g. NGO or OPD reports, national policy/strategy reports, laws, ministry of health reports), searching scientific publications and interviewing 3-5 key stakeholders per country (e.g. MoH representatives). They noted whether the data was available, and if so what it showed for each of the indicators, on a standardised excel sheet. They also reported which indicators could not be completed and why, and which indicators need to be re-worded or otherwise clarified, or eliminated. Overall, the researchers provided their qualitative impression on the feasibility of undertaking the assessment. The results were discussed as a group, including consideration of how the indicator set needed to be adapted.

2.4 Indicator set and assessment approach refinement

Individual interviews were held with seven health system stakeholders (MoH representatives, NGOs, academics) from a range of countries (South Africa, Bangladesh, Nigeria, Malawi, Indonesia, Uganda, India). The key informants were asked about the perceived value of the indicator set, potential users, challenges to implementation and strategies for encouraging the uptake of findings. A workshop was then held with three international stakeholders and three human-centred design experts to agree recommendations on the use of the indicator set and its scale up.

2.5 Ethics

Ethical approval for the study was granted from the Medical Research Council of Zimbabwe (MRCZ), the Maldives National Bureau of Statistics, the National Health Research Committee at the Maldives MoH and the Institutional Review Board of London School of Hygiene & Tropical Medicine, UK.

3.1 Framework indicators

Overall, the research partners in the Maldives and Zimbabwe reported that it was feasible to collect most of the indicators, and that the majority were clearly defined and important. The specific changes suggested for the indicators are described.

3.3. Feedback on indicator set application

The primary objectives of the indicator set were determined by consensus and include to: 1) collate data on the level of disability inclusion in the health system to set a benchmark, 2) identify ways in which the health system could be improved to become more disability-inclusive, and 3) generate monitoring data on change in disability inclusion through repeated use. It was agreed that the application of the indicator set should be led by the MoH, or by a consultant (e.g. NGO) working in collaboration with the MoH. People with disabilities should be active partners in this process. Additional users of the indicator set could include researchers to undertake situational analyses or to identify indicators to use in studies, disability programs to monitor inclusion and disability rights groups to identify gaps and advocate for action.

Key identified challenges were that a disability focal person may be lacking at the MoH, resulting in limited drive of the process. Furthermore, MoH partners may consider that disability is the responsibility of other departments (e.g. Ministry of Social Welfare), and indeed a plan for disability-inclusive health may need the involvement of a range of Ministries. The assessment may need to include other key partners, such as the Ministry of Finance or Social Welfare. Another concern was that the MoH may be reluctant to reveal poor levels of disability inclusion and thus be reticent to undertake the assessment using the indicator set. Education and advocacy are therefore important to demonstrate the importance of disability-inclusive health, including the financial benefits.

In terms of logistics, it was anticipated that the timeline for the assessment would be 2-3 months. Provision of tools (e.g. formatted excel worksheets) and good practice examples of use would assist with scale up of the use of the indicator set. It was advised that the assessment should commence with an inception workshop attended by key partners (e.g. MoH, OPDs, relevant NGOs, healthcare workers, representatives from other ministries). After the assessment, there should be a sharing and strategic planning workshop to agree key recommendations and actions. This workshop should be attended by MoH, people with disabilities and other key audiences (e.g. relevant NGOs and OPDs who can support action and hold the MoH to account). Developing and sharing examples of disability strategies and good practice could help in plan development to implement the agreed recommendations and actions. Furthermore, there should be clear plans for costing and accountability (e.g. monitoring of implementation).

A website was created describing the assessment purpose, process and indicators: https://www.themissingbillion.org/assessment-toolkit.

4.1 Key findings

There is growing awareness of the importance of promoting disability-inclusive health systems, as they will help to achieve the right to healthcare for people with disabilities (UN, 2006), and improve their health outcomes (Kuper & Heydt, 2019; WHO, 2011, 2022). Disability-inclusive health systems are also likely to be more resilient and work better for all (e.g. minority language speakers, people with temporary impairments), and are expected be cost-saving (WHO, 2022). Currently, few global actors focus on disability-inclusion in their programmes. Key reasons are a lack of awareness of the issue, little knowledge of where the gaps are in their activities, and a lack of clarity on the priority areas for intervention. The Missing Billion Disability-Inclusive Health System framework supports a structured approach to assessing the inclusion of people with disabilities holistically by health systems, with specific indicators related to different components. This assessment approach was considered by international key informants to be logical and comprehensive.

The pilot data from the Maldives and Zimbabwe demonstrated that it was feasible to collect indicators related to the framework components to indicate the level of disability-inclusion in the health system. The indicators were viewed as mostly useful and relevant, with some requiring refinement, usually to improve clarity. Moreover, the assessment revealed areas where the health system in the Maldives and Zimbabwe was making progress in terms of disability inclusion (e.g. in terms of legal framework) and other areas where there were large gaps (e.g. leadership on disability) or lack of data (e.g. accessibility, outputs and outcomes).

4.2. Implications for policy and practice

The key potential value of the indicator set is to collate data on the level of disability inclusion in the health system to set a benchmark and help to identify ways in which the health system could be improved to be more disability-inclusive. It will also allow international comparison to reveal good practice (Bitton et al., 2017; Fekri et al., 2018), and produce data on change in disability inclusion through repeated use.

Existing tools (e.g. PHCPI) may show overall health system performance, but not whether achievements are equitable and reach all segments of the population unless special attention is provided. As indicated already, people with disabilities will need additional focus as they face additional barriers to accessing services, which is in violation of their rights as set out in UNCRPD. The strength of the indicator set is therefore that it fills an important gap, as there are no other comprehensive health system performance assessment tools that have a specific focus on disability-inclusion. Indeed, it has already been used to inform guidance on World Bank health care investments to promote disability-inclusion (World Bank, 2022), as well as a WHO Euro policy brief on disability-inclusive health systems (WHO, 2021b). Furthermore, the indicator set fulfils many of the criteria of a ‘good’ health system performance assessment tool; it was developed in a structured and inclusive way, based upon a conceptual framework, addresses an important policy issue, and presents clear constructs and indicators (Tashobya et al., 2014).

A key concern is that there is not yet a structure in place to promote scale-up of the assessment using the indicator set (e.g. incorporation in World Bank or WHO strategy) (Tashobya et al., 2014). The success of the indicator set is therefore reliant on buy-in and implementation by individual MoHs, working in collaboration with people with disabilities (e.g. OPDs) (Tashobya et al., 2014). However, currently many MoHs lack capacity, are under-funded and perceives disability as a low priority. Advocacy is therefore needed to raise the priority of this issue, or the availability of seed funding or technical assistance. Incorporating the indicator set with existing tools may also encourage uptake, whether general health systems assessments (e.g. PHCPI, UHC monitoring initiatives), or disability-relevant assessments (e.g. WHO Systematic Assessment of Rehabilitation Situation) (Kleinitz et al., 2022), and we attempted to produce an indicator set which was complementary to these approaches. Furthermore, countries may be cautious about undertaking such an assessment if they perceive their country to be under-performing or if the resources and political will are not available to be able to respond to identified gaps. Care will therefore need to be taken on how to describe the purpose of the assessment, and ensure that this is viewed as an opportunity for measurement and improvement, rather than judgment. The indicator set may need to be tailored to different settings, including additional indicators viewed to be relevant or important for that setting (Ratcliffe et al., 2019). Another issue is that donors are generally not yet sufficiently committed to disability inclusion to encourage programmes or governments to make health systems disability-inclusive. This situation may be changing; The FCDO disability inclusion and rights strategy 2022 to 2030 makes commitments to “Achieving inclusive health for all” (FCDO, 2022), and the World Bank has recently published guidance on creating disability-inclusive health care systems (World Bank, 2022).

There are several limitations to the indicator set which need consideration. The pilot study in the Maldives and Zimbabwe showed that there was a lack of available data for several of the indicators, although this gap may be filled if the assessment is led by the MoH as they will have access to unpublished data. Indeed, the lack of involvement of the MoH in the pilot study is an important limitation. The indicator set focuses on measures within the health system, to drive action by health actors. Yet, several important actors for disability-inclusive health are outside of the health system, such as disability allowance providers and programmes to provide AT (Hameed et al., 2022b). Social determinants of health are important, perhaps particularly for people with disabilities, yet lie outside the boundary of the health system. The indicator set also does not focus on broader societal goals that are important for people with disabilities, such as increasing employment. We also did not undertake psychometric assessment of the indicator set. The assessment process was undertaken within the context of an ongoing research study in both settings. Consequently, we could not measure the time taken for the assessment alone. Moreover, this set-up may mean that the assessment appeared more feasible than it would in other settings, potentially limiting generalizability. Feasibility of assessment implementation was considered qualitatively, rather than through the use of objective measures. Finally, the Missing Billion Disability-Inclusive Framework was produced through adaptation from two existing frameworks, and not developed through new conceptual or theoretical work.

4.3. Future work

The indicator set needs to be implemented in more countries to allow a fuller assessment of feasibility (Bowen et al., 2009), including with objective measures of feasibility. Wider use will also allow finalisation of the indicator set, for instance, determining whether additional indices are needed, more response levels required, and composite indices should be created (e.g. for the outcomes and outputs).

Further consideration is also required of what is needed for the assessment to guide policy and programmatic improvement. A key priority is to encourage ownership and involvement by the MoH, and to generate and share good practice in the application of the indicator set. Consideration of how the international data is collated and displayed is also important, potentially through data visualization tools such as used by PHCPI (PHCPI, 2018). International comparison can generate sensitivity, as it implies judgement, and may also require a mechanism to review and validate results before they are displayed. The indicator set can also be adapted to assess disability-inclusion in specific aspects of care, such as in HIV programmes.

4.4. Conclusions

Health systems have neglected the inclusion of people with disabilities, leading to a violation of their rights and worse health access and outcomes. This new assessment approach can help to identify key issues and guide action, and thereby may ultimately improve health systems for all.