Performance evaluation of tuberculosis smear microscopists working at rechecking laboratories in Ethiopia

Background Tuberculosis is an infectious disease caused by the bacillus Mycobacterium tuberculosis. According to the Ethiopian Federal Ministry of Health’s 2013–2014 report, the tuberculosis case detection rate was 53.7%, which was below the target of 81% set for that year. Objective This study assessed the performance of tuberculosis smear microscopists at external quality assessment rechecking laboratories in Ethiopia. Methods A cross-sectional study was conducted at 81 laboratories from April to July 2015. Panel slides were prepared and validated at the National Tuberculosis Reference Laboratory. The validated panel slides were used to evaluate the performance of microscopists at these laboratories compared with readers from the reference laboratory. Results A total of 389 external quality assessment rechecking laboratory microscopists participated in the study, of which 268 (68.9%) worked at hospitals, 241 (62%) had more than five years of work experience, 201 (51.7%) held Bachelors degrees, and 319 (82%) reported tuberculosis smear microscopy training. Overall, 324 (83.3%) participants scored ≥ 80%. Sensitivity for detecting tuberculosis bacilli was 84.5% and specificity was 93.1%. The overall percent agreement between participants and reference readers was 87.1 (kappa=0.72). All 10 slides were correctly read (i.e., scored 100%) by 80 (20.6%) participants, 156 (40.1%) scored 90% – 95%, 88 (22.6%) scored 80% – 85% and 65 (16.7%) scored below 80%. There were 806 (20.7%) total errors, with 143 (3.7%) major and 663 (17%) minor errors. Conclusion The overall performance of participants in reading the slides showed good agreement with the reference readers. Most errors were minor, and the ability to detect tuberculosis bacilli can be improved through building the capacity of professionals.


Introduction
Tuberculosis is an infectious disease caused by the bacillus Mycobacterium tuberculosis. It remains a major global health problem, responsible for ill health among millions of people each year. 1,2,3 It is the second leading cause of death among all infectious diseases worldwide after HIV. 1,3 According to the 2014 World Health Organization Global Tuberculosis Report, there were nine million new tuberculosis cases and 1.5 million tuberculosis deaths (1.1 million among HIVnegative people and 0.4 million among HIV-positive people) in 2013. 1 One quarter of global cases and deaths occurred in the African Region, 1 and Ethiopia ranked 10th in tuberculosis incidence among 22 high-burden countries. 1,4 According to Ethiopian Ministry of Health reports for 2012-2013 and 2013-2014, the targeted tuberculosis case detection rates were 82.7% for the 2012-2013 period and 81% for 2013-2014. However, the case detection rates achieved were 58.9% in 2012-2013 and 53.7% in 2013-2014, which were well below these targets. 5,6,7 A low case detection rate is often associated with a lack of effective programme awareness, lack of active cough identification and lack of quality-assured routine diagnosis (such as sputum quality, reagent quality, knowledge, and capacity of professionals). In Ethiopia, factors that are associated with low case detection rates have not been well studied. However, they are likely to be associated with these factors. Therefore, the present study dealt with the performance of tuberculosis smear microscopists as one factor affecting quality-assured routine diagnosis.
In most low-and middle-income countries, smear microscopy remains the foundation of tuberculosis diagnosis, despite its relatively low sensitivity. Microscopy has also remained essential to monitoring of tuberculosis treatment. A microscopy network with adequate http://www.ajlmonline.org Open Access population coverage and high quality performance is therefore critical. Bright-field sputum smear microscopy (i.e., conventional Ziehl-Neelsen staining) is widely available, simple to perform, inexpensive, and requires simple laboratory facilities. 1,8 Thus, one national tuberculosis control strategy recommended by the World Health Organization is to pursue expansion and enhancement of high-quality directly-observed treatment, short-course chemotherapy through early case detection and diagnosis at quality-assured laboratories. 9 Quality assurance of microscopy remains a critical activity of all laboratory networks, and a comprehensive external quality assessment (EQA) programme that includes on-site evaluation, random blinded rechecking, and panel testing should be implemented. 1,8,10 EQA programmes are needed to ensure that smears are performed and stained properly, results are interpreted correctly and all microscopy centres achieve an acceptable level of performance. Effective EQA programmes require dedicated and qualified staff for rechecking of smears. The implementation of EQA for microscopy has the advantage, not only of strengthening laboratory networks, but of improving diagnostic quality. 11 The Ethiopian Federal Ministry of Health and the Ethiopian Public Health Institute decentralised EQA programmes to regional reference laboratories and have guided the regions to decentralise further into sub-regional laboratories and EQA rechecking laboratories. This decision was made with the assumption that all microscopy centres in the various regions would have a chance to participate in EQA programmes and that improved EQA coverage could be achieved. The mandate for conducting a rechecking programme was given to the EQA rechecking laboratories by Regional Health Bureaus. Following the endorsement of the Regional Health Bureaus, EQA rechecking laboratories have the right to perform tuberculosis EQA blind rechecking by collecting slides from the microscopy centres in their catchment areas. The aim of this study was to produce baseline data about the performance of the tuberculosis rechecking laboratories and the microscopists who work there.

Ethical considerations
Leftover samples were collected anonymously from federal hospitals for panel preparation. All information about each tuberculosis EQA rechecking laboratory was kept confidential and used only for the purposes of this study and for the improvement of acid-fast bacilli (AFB) microscopy.

Study design
A cross-sectional study was conducted from April to July 2015 at tuberculosis EQA rechecking laboratories in Ethiopia. Validated panel slides were used to assess the performance of microscopists working in the laboratories. The study was conducted at 12 (100%) regional laboratories, three (75%) sub-regional laboratories, 46 (61%) hospital laboratories and 20 (59%) health centre laboratories among the 125 EQA rechecking laboratories in Ethiopia.
Sputum samples for panel preparation were collected from federal hospitals anonymously, and panel slides were prepared in the National Tuberculosis Reference Laboratory. Each dilution panel was validated by six different readers following World Health Organization guidelines (reference readers). 10 A set of 10 validated slides was distributed to participating laboratories to assess the reading and interpretation proficiency of smear microscopists; 50-70 minutes were allowed to complete the reading. 10,12 The panel composition and bacilli load were: one grade 3+ slide, one grade 2+ slide, two grade 1+ slides, three 1-9 AFB/100 field slides and three negative slides. 10 The results were expressed as correct, minor error or major error. Major errors were classified as high false positive, if a negative smear was misread as grade 1+ to 3+ positive, or as high false negative, if a grade 1+ to 3+ positive smear was misread as negative (Table 1). Minor errors were classified as a quantification error, when there was a difference of more than one grade in the reading of positive smear between the examinee and the reference readers, a low false positive, when a negative smear was misread as scanty (1-9 AFB/100x field), or as low false negative, when a scanty slide (1-9 AFB/100x field) was misread as negative. 10,12,13  Each slide was worth 10 points. The total possible score was 100 points (for 10 slides), and based on national and World Health Organization guidelines, a passing score was considered to be ≥ 80%. 10,12 Committing major errors, like a high false positive or high false negative, was worth zero points, whereas minor errors, like low false positive, low false negative and quantification errors, were worth five points. 8

Study participants
A total of 389 microscopists (2 to 13 microscopists per rechecking laboratory) from 81 tuberculosis EQA rechecking laboratories participated in the study, of whom 263 (67.6%) were men and 126 (32.4%) were women ( Table 2). Most of the study participants worked in hospital laboratories (n = 268, 68.9%); 241 (62%) participants had more than five years of work experience in tuberculosis smear microscopy services, 201 (51.7%) held a Bachelors degree, and 319 (82%) had gone through tuberculosis smear microscopy in-service training.

Panel testing
Among all 389 participants, 324 (83.3%) scored ≥ 80% (passing) ( Table 3). When stratified by place of work, more participants working in hospitals (n = 231, 86.2%) achieved a passing score (≥ 80 %) than participants who worked in other types of facilities. On the other hand, 21/23 (91.3%) microscopists with less than two years of work experience scored ≥ 80%, which was higher than microscopists with more than two years of work experience. The proportion of participants who scored ≥ 80% was higher among holders of a Masters degree compared with participants with other educational backgrounds, and the proportion of participants who scored ≥ 80% was slightly higher among participants who had not had tuberculosis smear microscopy in-service training. In general, there were no statistically-significant associations between the performance of participants in tuberculosis bacilli detection and their sex, work experience, educational background, place of work or tuberculosis smear microscopy in-service training.
A total of 3890 validated slides were read by study participants ( Table 4). The overall sensitivity for detecting tuberculosis bacilli was 84.5% and overall specificity was 93.1%. The overall percent agreement of participants with the reference readers was 87.1 (kappa=0.72). The percent agreement of participants working in health centres with the reference readers was 83.1% (kappa=0.64), which was slightly lower than participants working in hospitals or regional laboratories. The negative predictive values were quite low for participants working in all health facilities.
Of the 389 participants, 80 (20.6%) participants correctly read all 10 slides and scored 100% (Figure 1). A total of 156 (40.1%) scored 90% -95%, which means they committed one major error or two minor errors. A total of 88 (22.6%) participants scored 80% -85%, which means they committed three to four minor errors or two major errors or one major and one minor error or one major and two minor errors. Finally, 65 (16.7%) participants scored below 80%, which means they had more than four minor errors or two major errors or one major and two minor errors.

Discussion
This cross-sectional study evaluated the performance of tuberculosis smear microscopists working at EQA rechecking laboratories and the status of the respective laboratories. In this study, the overall agreement of participants with reference readers for reading the validated slides was 87.1% (kappa=0.72), which was good agreement based on kappa statistics. 14 However, lower agreement was observed when  17,18 Thus, performance in our study was slightly lower than in similar studies conducted in other parts of Ethiopia. This may have been due to the large number of laboratories and/or laboratory professionals included in our study, which was more of a nationwide study with wider representation. This may have made our study more prone to lower performance.
In general, agreement in reading between participants and reference readers was slightly lower than in similar studies conducted in other countries. Our finding was lower than studies done in India (98% agreement) 19 and Tanzania 20 (89.2% agreement). However, agreement in our study was

Concordance with reference readers
Health centre Hospital Regional laboratory Total concordance †, Reference readers were microscopists at Ethiopia's National Tuberculosis Reference Laboratory. higher than studies done in Ghana (73%) 21 and the Democratic Republic of Congo (74%). 22 These differences may be attributable to differences in the composition of the panel slides, as we prepared a second degree of difficulty in our slides (three scanty and three negative slides). 10 In our study, the overall sensitivity was 84.5% and specificity was 93.1%. The study in Hawassa, Ethiopia showed higher sensitivity (91.97%), but lower specificity (80.0%). 16 On the other hand, both sensitivity (96.5%) and specificity (96.4%) were higher in the West Amhara, Ethiopia report. 18 Similarly, the study conducted in East Amhara, Ethiopia showed higher sensitivity (88.4%) and specificity (99.3%). 17 Both sensitivity and specificity were 96.8% in a Southern Ethiopia finding, which was higher than our study. 15 Sensitivity (88.5%) and specificity (100%) were also higher in the study conducted in Tanzania. 20 In our study, the lower sensitivity indicates that there were high false-negative rates (patients with tuberculosis bacilli misdiagnosed as negative). The consequence of this low sensitivity is that tuberculosis patients are not treated, which results in ongoing disease, disease transmission or death.
The study in Hawassa, Ethiopia reported that 13.6% of microscopists correctly read all panel slides, which was slightly lower than our finding, and 86.4% committed at least one error in reading 10 slides, which was significantly higher than our finding. 16 In the study done in India, 95% of readers reported no errors, 19 demonstrating far greater proficiency than the present study. Although the majority of our participants (83.3%) had an acceptable performance score (≥ 80%), we consider this a weak achievement, since study participants were from facilities with a responsibility to recheck other health institutions' slides and provide support to them. Considering this responsibility, microscopists at these facilities should have scored better than the current findings.
In the present study, there were more low false negatives than quantification errors. False readings in Southern Ethiopia (3.2%), East Amhara (1.6%) and West Amhara (3.5%), Ethiopia were lower than our finding. 15,17,18 On the other hand, the percentage of errors in Hawassa, Ethiopia (29.75%) was higher than ours. 16 In addition, the Hawassa study had fewer major errors (2.22%), but more minor errors (27.5%) than our study. 16 Quantification errors were the biggest contributor to overall errors in the Hawassa study, whereas in the present study, low false negatives were the most frequent errors. 16 Fewer errors were observed in India, where quantification errors were the most frequent and no high false positives were reported. 19 In a similar study conducted in Mexico, quantification errors were frequent (12.3%), followed by low false negatives (5.7%). 23 A study conducted in the Democratic Republic of Congo also reported frequent low false-negative errors. 22 In another study conducted in Taiwan, low false-positive errors were much higher (28.6%) than in the present study. 24 False-negative errors could lead to failure to detect persons with infectious tuberculosis, who could continue to spread the disease in their communities; false positives could lead to unnecessary anxiety, exposure of patients to unwanted side effects of medications, and unnecessary expenditure. 25 While lower rates of minor errors are acceptable due to the inherent problems of AFB smear microscopy services, major errors are unacceptable. Among minor errors, low falsepositive and low false-negative errors both have a significant impact on patient management and tuberculosis control programmes, whereas quantification errors have no impact on patient management. Hence, improving the competency of professionals through training, implementation of strong EQA programmes, supportive supervision and mentoring are critically important to reduce or avoid these types of errors and to maximise case detection rates. 10 Limitations This study has a few limitations, which should be considered when interpreting our results. First, unstained slides were not sent to participating laboratories. Thus, the quality of the reagents used for routine AFB microscopy was not assessed. Additionally, information on the age of the participants was not collected in the demographic information. Therefore, we could not evaluate the effects of age variability on the performance of the study participants.

Conclusion
The overall performance of the tuberculosis EQA rechecking laboratories in reading AFB slides showed good agreement with the reference readers (87.1%). Overall, 20.7% of slides were misread, and most errors were minor. Nevertheless, these errors are alarming, and our findings are a clarion call to tuberculosis control programmes to give needed support to EQA programmes. A large number of minor errors were noted; thus, continuous mentoring and supportive