Evidence from 617 laboratories in 47 countries for SLMTA-driven improvement in quality management systems

Background The Strengthening Laboratory Management Toward Accreditation (SLMTA) programme is a large-scale effort to improve the quality of laboratories in resource-limited countries. Objectives This study sought to evaluate the first four years (2010–2013) of SLMTA implementation. Methods Country-level data were submitted by SLMTA programme leads and compiled globally. Performance was measured before (baseline) and after (exit) SLMTA implementation using an audit checklist which results in a percentage score and a rating of zero to five stars. Some laboratories continued to monitor performance in post-exit surveillance audits. We evaluated score improvements using two-tailed t-tests for equal variances and estimated the number of tests performed by SLMTA laboratories based on star level. Results SLMTA was implemented in 617 laboratories in 47 countries in Africa, the Caribbean, Latin America and Southeast Asia. At the baseline audit, the laboratories scored an average of 39% on the checklist and 84% of them were rated below one star. As of December 2013, 302 laboratories had completed the SLMTA programme; mean checklist scores increased from 39% at baseline to 64% at exit (p < 0.001) over an average 16-month programme duration. Ninety-two laboratories conducted a surveillance audit at a median of 11 months after their exit audit; 62% further increased their performance. Six SLMTA laboratories have achieved accreditation status. In total, the 617 SLMTA laboratories conduct an estimated 111 million tests annually. Only 16% of these tests were conducted by laboratories with at least one star at baseline, which increased to 68% of tests after SLMTA training. Thus, approximately 23 million tests are conducted annually by laboratories previously at zero stars that now have one to five stars; this number is projected to increase to 58 million when currently-enrolled laboratories complete the programme. Conclusion SLMTA has transformed the laboratory landscape in resource-limited countries worldwide and has the potential to make a substantial and sustainable impact on the quality of laboratory testing and patient care.


Introduction
Substantial resources have been invested in management training and development to help scale up and improve the quality of health services in order to reach the United Nation's Millennium Development Goals. 1 However, evidence is scarce in terms of the impact that has actually been achieved. An international meeting convened by the World Health Organization (WHO) in 2005 on strengthening leadership and management in low-income countries concluded that programme evaluation often focuses on the number of trainees and pre-/post-knowledge-based tests; evaluation of programme impact on managers' daily work and job outcomes is more the exception than the norm. 2 Strengthening Laboratory Management Toward Accreditation (SLMTA) is a large-scale effort aimed at improving the quality of laboratory services and patient care in resource-limited settings by developing competent laboratory managers. SLMTA provides an innovative training curriculum on implementing practical Quality Management Systems (QMS) using existing resources with built-in accountability and evaluation that focuses on result-oriented outcome measures. 3 Launched in 2009 in Kigali, Rwanda, this programme seeks to engage laboratories in continuous quality improvement and to accelerate their preparations toward accreditation to international standards. 4,5 International Organization for Standardization (ISO) 15189 requirements using the WHO Regional Office for Africa's (WHO AFRO) accreditation preparedness checklist. The findings from this unprecedented study shed light on the widespread success of an innovative programme that has empowered laboratory teams throughout the developing world to strive for continuous quality improvement and work toward accreditation, despite limited resources.

Research methods and design SLMTA implementation and evaluation
The methodology of the SLMTA programme has been described previously. 3,5 Briefly, SLMTA is a competencybased programme that uses a series of short courses and work-based learning projects to effect rapid and measurable laboratory improvement for better patient care through enhanced management skills and implementation of practical quality management systems. The SLMTA training programme is based on a series of three workshops. After each workshop, participants implement improvement projects supported by regular supervisory visits or on-site mentoring. Laboratories are evaluated using WHO AFRO's Stepwise Laboratory Quality Improvement Process Towards Accreditation (SLIPTA) checklist, 6 which includes 111 items divided into 12 sections that represent the 12 Quality System Essentials (QSEs). 7 In order to assess progress made by the laboratories, audits are conducted using the SLIPTA checklist at the beginning (baseline) and at the end (exit) of the SLMTA programme. Many countries also conduct intermediate audits to help guide programme implementation, as well as surveillance audits after exiting the programme so as to monitor continued improvement and assess sustainability. After an audit, laboratories receive a score which determines their star rating -from zero to five, with < 55% corresponding to zero stars, 55% -64% one star, 65% -74% two stars, 75% -84% three stars, 85% -94% four stars and ≥ 95% five stars.
When a laboratory achieves a five-star rating, it may be encouraged to seek accreditation.

Data analysis
Programme data up to December 2013 were collected from all countries implementing SLMTA. Variables included year of implementation, number and types of laboratories in each cohort (i.e., enrolment in the same training round), number of people trained, audit scores and approximate number of tests conducted by each laboratory. Data were collated and analysed in Microsoft ® Excel 2013. Descriptive statistics (percentages, medians, ranges) were calculated. Statistical significance of improvements were assessed using 2-tailed t-tests for equal variances (f-tests showed equal variances at p > 0.1); comparison of improvements across laboratory types was assessed using one-way analysis of variance (ANOVA).
Given the wide variations in the time between the baseline audit and the first SLMTA workshop, programme length was defined as the time from the first SLMTA workshop to the exit audit. In some countries, results for large national reference laboratories were reported by department rather than for the laboratory as a whole. For consistency, we aggregated the department scores into a single score per laboratory, using median values across laboratory departments. We estimated the number of laboratory tests conducted in SLMTA laboratories based on country reports for 2012; missing data were imputed using averages by laboratory type.

Programme spread
Since its introduction in 2009, SLMTA has been implemented in 47 countries worldwide, including 23 countries in Africa, 12 in the Caribbean Region, 10 in Central and South America, and two in Southeast Asia ( Figure 1, Table 1).        As of December 2013, 65 SLMTA cohorts had been initiated, with one to 55 laboratories per cohort. Thirty were first cohorts within countries, including three regional cohorts which encompassed multiple countries, whilst the remaining 35 were subsequent cohorts in 23 countries as they expanded the programme (Table 1).
A total of 1923 people from 617 laboratories were trained. Thirty-seven of these laboratories (6%) were re-enrolled in a subsequent SLMTA cohort. Eighteen per cent of the 617 laboratories were at the national level, 27% at regional or provincial levels, 38% at district or primary levels, 10% belonged to non-governmental, faith-based or private organisations, 5% were military laboratories and 2% were blood banks. Nearly all (98%) of these laboratories provide HIV-related services, such as HIV diagnosis, treatment monitoring, opportunistic infection or tuberculosis testing, and blood bank testing. Of these laboratories, 302 (49%) completed the SLMTA programme and conducted an exit audit, whilst the remaining 315 (51%) were still going through the programme at the time of this analysis.

Audit results
At the baseline audit, the mean score for all 617 SLMTAenrolled laboratories was 39% (median 37%) and 84% received zero stars (i.e., score < 55%) on the SLIPTA five-star scale (Table 1). For the 302 laboratories that had completed the programme and conducted an exit audit, mean scores increased from 39% at baseline to 64% at exit (p < 0.001). Whilst 85% had received zero stars at baseline, only 30% remained at zero stars at exit (Figure 2).
The average improvement from baseline to exit audit was 25 percentage points over a mean programme duration of 16 months. Sixty-eight per cent of laboratories improved by at least one star, including 22% that increased by three or more stars. Although 32% of the laboratories did not increase their stars, 23% of these more than doubled their scores from baseline to exit. Improvements tended to be higher for laboratories with lower baseline scores. Those with a starting score below 25% had an average increase of 37 percentage points, whilst those with a starting score of 65% or higher had an average increase of nine percentage points (r = -0.49, p < 0.001) ( Table 2).
Baseline and exit scores and improvements were similar between the different types of laboratories (p > 0.05) (Figure 3). Amongst laboratories that had completed the SLMTA programme, the 132 laboratories implementing SLMTA in the first year (2010) had the same mean improvement as the 170 laboratories implementing SLMTA in 2011 to 2013 (24 percentage points).

Sustainability
Ninety-two laboratories had conducted post-SLMTA surveillance audits, at a median of 11 months post-exit audit (range five to 28 months). Of these laboratories, 62% further increased their score, including 34% whose score increased by > 10 percentage points post-exit audit ( Figure 4). Of the national-level laboratories that conducted post-SLMTA audits (n = 19), 79% improved their scores, whilst only 56% of district-level laboratories (n = 27) further increased their   scores (p = 0.03). Overall, there was regression toward the mean, as laboratories that received lower scores at exit were more likely to improve their scores at the post-SLMTA audit (r = -0.48, p < 0.001).
As of September 2014, six SLMTA laboratories have been accredited to international standards (Table 3). These laboratories had baseline scores approximately equal to all SLMTA laboratories, with a median score of 38% (range 29% to 75%) as compared to a median of 37% for all SLMTA laboratories; one of the laboratories started at three SLIPTA stars, whilst the other five laboratories began at zero stars. At the exit audit, the six laboratories had a median score of 90%, with an increase of 52 percentage points, as compared to the international average of 64% and 25 percentage points. The median time from SLMTA initiation to accreditation was 31.5 months (range 17-54). In March 2013, the Kenya HIV National Reference Laboratory became the first SLMTA laboratory (and the first public medical laboratory in Kenya) to earn accreditation. 8 The Hai Duong Preventive Medicine Center in Vietnam had the lowest baseline score of the six laboratories at 29%, but achieved the greatest improvement from baseline to exit (58 percentage points); it took the laboratory only 19 months from initiation of SLMTA to accreditation.

SLMTA implications for laboratory testing
In total, the 617 SLMTA laboratories conducted an estimated 111 million tests in 2012. Approximately 43.5 million of these tests were conducted by the 302 laboratories that had completed the SLMTA programme by the time of this analysis. Only 16% of these tests were conducted by laboratories with at least one quality star prior to SLMTA implementation (3% had ≥ 3 stars); after SLMTA training 68% of tests were performed by laboratories with at least one quality star (28% had ≥ 3 stars). This translates to approximately 23 million tests conducted by laboratories that previously had zero stars and now have one or more stars. Furthermore, within the group with zero stars prior to SLMTA implementation, more than one in three tests were performed by laboratories scoring below 35% on the SLIPTA checklist; after SLMTA completion this proportion    decreased to one in 50 ( Figure 5). Assuming similar quality improvements for the 315 currently-enrolled laboratories, we project the total number of tests conducted by laboratories with one or more stars that had previously been at zero stars to rise to 58 million by the end of 2015.

Discussion
Since its launch in 2009, SLMTA has achieved wide global coverage and demonstrated measurable improvement using a standardised checklist, transforming the laboratory landscape in many resource-limited countries worldwide. The results have been remarkable, with audit scores increasing 25 percentage points, and two-thirds of the laboratories that completed the SLMTA programme improving by at least one quality star level. Twenty-three countries have implemented a second cohort of training and 23 million tests conducted by laboratories previously lacking any quality management system are now being conducted by laboratories with at least a basic level of quality in place. The SLMTA programme has the potential to make a substantial and sustainable impact on the quality of laboratory testing and, therefore, patient care.
As of August 2014, 102 laboratories from 15 countries in sub-Saharan Africa had received an official WHO AFRO SLIPTA audit by the African Society for Laboratory Medicine (ASLM) (Maruta T, 2014, personal communication, August 31); ninety-seven (95%) of these laboratories had gone through the SLMTA programme. It is evident that SLMTA has emerged as a vital strategy in accelerating laboratory preparedness for accreditation in the developing world and in facilitating the fulfillment of ASLM's 2020 Vision, Goal 2: 'by 2020, enroll 2500 laboratories in the WHO SLIPTA quality improvement programme and enable 250 laboratories to achieve accreditation by international standards'. 9 To date, the SLMTA programme has been primarily funded by the US President's Emergency Plan for AIDS Relief (PEPFAR) through CDC and implementation has been limited primarily to PEPFAR-supported countries. Although HIV-focused, these resources have been leveraged to build wide-spread capacity and strengthen the overall laboratory system. 10  This is the first study to examine the existing level of laboratory quality on a broad international scale. The mean baseline audit score of the 617 SLMTA-enrolled laboratories was 39%, well below the 55% required to attain even the lowest quality level on the SLIPTA scale, and only 16% of the laboratories met or surpassed the one-star level. These results paint a bleak picture of the current quality of laboratories in much of the developing world. Whilst the issue of the lack of quality amongst laboratories in developing countries has been acknowledged in numerous publications 4,11,12,13,14,15,16,17 and a series of policy statements, 18,19,20,21 the current study begins to quantify the problem and sheds some light on a potential solution.
The SLMTA curriculum was designed to be closely aligned to ISO 15189; however, it has proven to be suitable to support other standards as well. For example, a SLMTAenrolled laboratory in Vietnam that tests environmental and food samples was accredited to ISO 17025. Two per cent of SLMTA sites are blood banks or blood-transfusion centres, which use different standards and have their own accreditation programmes. In Cameroon, inspired by the transformation achieved by its SLMTA-enrolled laboratory, a hospital adopted SLMTA in order to embark on its own quality improvement journey. 22 The need for improved quality is so widespread that others have called for the expansion of SLMTA into a programme that helps guide quality improvement in entire hospital and public health systems. 23 2 Stars 20%  According to a May 2013 survey, only two governmentowned medical laboratories in the 48 sub-Saharan African countries outside of South Africa were accredited. 11 In 2013 to 2014, three more government-owned laboratories in the region (all located in Kenya) were accredited after SLMTA implementation (Table 3). In the same timeframe, two SLMTA laboratories outside of the sub-Saharan region have achieved accreditation, whilst several others are awaiting accreditation assessments. These are only the first of what we anticipate will be many SLMTA laboratories reaching accreditation as the programme matures, since preparing for accreditation is a multi-year process. However, it is not feasible, either economically or programmatically, for all laboratories to pursue accreditation to international standards. As countries develop laboratory strategic plans, they will need to assess the options and develop realistic goals. The Ministry of Health in Uganda, for example, has set a national target of three stars for general hospital laboratories and five stars for national and regional reference laboratories (Lali W, 2014, personal communication, August 31). Regardless of the ultimate goal, the SLMTA programme will provide the tools necessary to guide laboratories in the continuous quality improvement process designed to achieve better patient care.
It is outside the scope of this study to examine how factors such as cohort size, programme length, laboratory type, mentorship model (amount, quality and type) and additional training impact the performance of the programme. However, detailed data are being collected as part of the SLMTA programme, including audit scores for each of the 12 QSEs and laboratory indicators such as turnaround time, specimen rejection rates, equipment downtime, proficiency testing, customer satisfaction and cost. These data are a potential gold mine of information that could be harnessed to identify causative factors of success and to fine-tune the programme with evidence-based strategies for continued improvement. An electronic-tool that will facilitate collection, management, aggregation, analysis and reporting of SLMTA programmatic data on a global level is currently under development by CDC and its partners.

Limitations of the study
These study results should be interpreted in light of several limitations. Firstly, half of the enrolled laboratories have not yet completed the programme and therefore have no exit audit data. As the programme matures, a higher proportion of the participating laboratories will have full data, reducing the risk of bias. Similarly, whilst we found that most laboratories that conducted surveillance audits continued to improve over a time-span of anywhere from five to 28 months after the SLMTA programme ended, the programme is too young to assess long-term sustainability.
Another concern is the quality of the audits, especially in the initial years of the SLMTA programme before formal training and certification of auditors. Whilst the SLIPTA checklist was designed to help standardise audit scoring, some variability may remain; assessment of intra-and interauditor variability is needed.
This was an observational programmatic study. Many factors were not controlled, such as programme duration, size of cohorts, mentorship model and additional training provided to enrolled laboratories. Future studies that compare these factors, as well as results of SLMTA laboratories with non-SLMTA laboratories are needed in order to separate the impact of SLMTA implementation from natural improvement in laboratory quality over the duration of the programme. In addition, estimates of the number of laboratory tests were based on 353 (57%) laboratories that submitted information; missing data were imputed using average volume by laboratory type, but some uncertainty remains.
Ultimately, a thorough evaluation of the SLMTA programme will require assessment of programme impact on patient care and health outcomes. Systematic evaluation of key indicators is needed, as well as targeted evaluations of programme cost-benefit, the impact of quality improvement on testing error rates and the association of quality services with patient outcomes, so as to determine public and personal health implications.

Conclusion
Few management and leadership development programmes have been implemented on such a large scale with resultsoriented outcome measures. With data collected from 617 laboratories in 47 countries throughout Africa, the Caribbean, Latin America and Southeast Asia, SLMTA is truly a global effort. It has demonstrated its ability to transform the laboratory landscape in resource-limited countries worldwide. Evidence from this study suggests that the SLMTA programme has the potential to make a substantial and sustainable impact on the quality of laboratory testing and patient care.