OBJECTIVES:

This first-ever country-level study assesses the implementation of the Helping Babies Breathe (HBB) program in 15 of Tanzania’s mainland regions by measuring coverage, adoption and retention of provider skills, acceptability among providers, and barriers and challenges to at-scale implementation.

METHODS:

Longitudinal facility-level follow-up visits assessed provider resuscitation knowledge and skills in using objective structured clinical examinations and readiness of facilities to resuscitate newborns, in terms of birth attendants trained and essential equipment available and functional. Focus group discussions were held with providers to determine the acceptability, challenges, and barriers to implementation of the HBB program.

RESULTS:

Immediately after HBB training, 87.1% of providers passed the objective structured clinical examination. This number dropped to 79.4% at 4 to 6 weeks and 55.8% at 4 to 6 months (P < .001). Noting this fall-off in skills, the program implemented structured on-the-job training and supportive supervisory visits, which were associated with an improvement in skill retention. At long-term follow-up, >90% of facilities had bag-mask devices available to all beds in the labor and delivery ward, and 96% were functional. Overall, providers were highly satisfied with the HBB program but thought that the 1-day training used in Tanzania was too short, so they would welcome additional training and follow-up visits to reinforce skills.

CONCLUSIONS:

The HBB program in Tanzania has gained acceptability and shown success in equipping providers with neonatal resuscitation knowledge, skills, and supplies. However, assessing the program’s impact on neonatal mortality has proven challenging.

What's Known on This Subject:

A pilot study of the Helping Babies Breathe program, a newborn resuscitation training program designed for birth attendants, conducted in referral-level facilities in Tanzania showed a 47% reduction in all-cause newborn mortality occurring in the first 24 hours of life.

What This Study Adds:

We present results from the 3-year national rollout of the Helping Babies Breathe program in Tanzania. The program was successful in equipping providers with life-saving newborn resuscitation skills and equipment; however, assessing impact on mortality requires greater efforts.

Neonatal mortality accounts for almost half of all deaths among children <5 years old, and nearly all these deaths occur in low- and middle-income countries.1 Birth asphyxia, the inability to start or maintain normal breathing at birth, is a leading cause of neonatal mortality in these settings, where many births occur without the attendance of a provider capable of resuscitating an asphyxiated newborn.2,3 Therefore, programs to prevent asphyxia-related neonatal mortality, such as Helping Babies Breathe (HBB), have focused on increasing birth attendants’ skills in newborn resuscitation.

The American Academy of Pediatrics and partners launched the HBB program in 2009, and it is now being used in >77 countries worldwide.4 An important pilot HBB study, conducted in 8 referral hospitals in Tanzania, showed a remarkable 47% reduction in all-cause newborn mortality within the first 24 hours of life.5 Based on these encouraging findings, the Tanzania Ministry of Health and Social Welfare (MOHSW) prioritized national expansion of HBB. The Children’s Investment Fund Foundation (CIFF) supported the MOHSW and implementing partner, Jhpiego, in the rollout of HBB to 15 of Tanzania’s mainland regions.6,7 Additionally, CIFF contracted a Harvard-based evaluation team to independently evaluate the impact and effectiveness of the HBB implementation.

This article presents findings from this first large-scale implementation of HBB in Tanzania. The goals of this study were to evaluate program coverage, determine the impact of HBB training on provider skills over time through testing on newborn mannequins, understand provider-level acceptability and perceptions of HBB in Tanzania, and identify the barriers and challenges to implementing HBB at scale.

The details of program implementation have been discussed elsewhere,6,8 and a summary is provided here. The CIFF-supported HBB program was rolled out in 15 of Tanzania’s 25 mainland regions from May 2013 through December 2014. The HBB course was adapted from the American Academy of Pediatrics’ HBB curriculum.9 

National and district trainers were chosen from a pool of HBB master trainers who were trained in 2009 by the MOHSW. All trainers received refresher trainings and then began a regional rollout of HBB courses to health care providers (Fig 1). Master trainers from the MOHSW and implementing partner traveled to each region and assisted regional trainers in conducting the local trainings. The local trainings were held at or near regional and district hospitals, with providers from lower-level facilities joining providers at these facilities for the trainings. Health care providers from labor and delivery wards and obstetric theaters were eligible to attend the HBB course. The maximum number of eligible providers from each facility level was 20 providers per hospital, 8 providers per health center, and 3 providers per dispensary. A trainer-to-trainee ratio of 1:6 was maintained for trainings. After the training, all health facilities were provided with HBB equipment and training materials, as well as instructions on how to properly use and clean them. Providers who attended the course were expected to return to their facilities and use the training materials to provide on-the-job training to colleagues who could not attend the course, although these colleagues would not be considered formally HBB trained.

FIGURE 1

Schematic overview of the HBB training program and its evaluation via a region-by-region approach across 3 illustrative regions in Tanzania.

FIGURE 1

Schematic overview of the HBB training program and its evaluation via a region-by-region approach across 3 illustrative regions in Tanzania.

An objective structured clinical examination (OSCE), previously validated and reported by our group, was used longitudinally to assess provider-level adoption and retention of HBB knowledge and skills.6 The OSCE was administered by a national or district trainer to each provider individually. During the OSCE, providers used a mannequin to show skills in 13 tasks relating to newborn resuscitation. Scores could range from 0 to 23, with passing scores starting at 16. Trained providers were first assessed immediately upon completion of the HBB course and again at 4 to 6 weeks after training. Any observed skills gaps were addressed after each participant’s scoring.

In February 2014, partners attended the HBB program’s first annual program review, during which there was noted concern over a drop in OSCE scores between immediately after training and the 4- to 6-week follow-up visits. Additionally, when trained providers returned to their facilities, there was limited self-initiated practice. Therefore, program partners developed specific interim interventions to address these concerns. Beginning in May 2014, Jhpiego equipped providers with a structured on-the-job training tool, which guided group practice of HBB skills by using a newborn mannequin. Supportive supervision visits were also instituted between local HBB trainers and providers at neighboring facilities who were HBB trained.

CIFF contracted an external evaluation team from Harvard Medical School and Massachusetts General Hospital to conduct an independent and longitudinal evaluation of the HBB training program in Tanzania. The goals of the evaluation included measuring changes in knowledge and skills of providers at targeted health facilities and providing the implementing partner with timely, actionable feedback on program impact. To achieve these goals, the evaluation team conducted unannounced facility-level follow-up visits at 4 to 6 weeks and 4 to 6 months after training to assess provider skills and perceptions, document available equipment, review records, assess program impact on asphyxia-related neonatal mortality, identify barriers and facilitators to scale-up, and undertake a costing analysis by using validated, field-tested tools (Table 1). We attempted facility-level collection of neonatal outcome data from the primary source registers among a sample of facilities, but as other country-level studies have found, historical data were not reliable. Some of these results are reported elsewhere, and the current analysis will focus on findings from the OSCEs, facility checklists, and focus group discussions.7,10 

TABLE 1

Methods Used to Evaluate the Impact of the HBB Program in Tanzania

MethodsPurposeKey IndicatorsOutcome
OSCEs Measure provider-level knowledge and skills • Percentage of providers passing the OSCE 1004 OSCEs were conducted. 
Facility and equipment checklists Measure coverage and equipment availability • Availability and functionality of bag-mask devices and penguin suction devices 335 facility checklists were completed. 
Comparing HBB training lists with delivery room registers Determine training coverage and proportion of deliveries conducted by HBB-trained providers • Percentage of birth attendants trained in HBB 335 facility registers were reviewed. 
• Percentage of deliveries attended by an HBB-trained provider in the past 1 and 3 mo 
Focus group discussions Identify provider-reported program strengths, weaknesses, barriers, recommendations, and elements important for scale-up • Likes, dislikes, barriers, and recommendations for the HBB program 222 focus group discussions were conducted, involving 599 participants. 
Perinatal death audits To understand circumstances surrounding recent newborn deaths and stillbirths, including contributing factors, clinical management, and opportunities for improvement • Clinical presentation 110 perinatal death audits were conducted across 15 regions. 
• Clinical management 
• Causes of death 
Cost analysis Quantify cost of national rollout and maintenance of the HBB program in Tanzania • Cost of implementing HBB program per facility Projected total costs for rollout to the 25 mainland regions would be $4 million, with an additional $5.6 million for another 5 y of program support after rollout (Chaudhury et al10). 
• Projected cost of national rollout of HBB 
Mortality assessment Measure the impact of the HBB program on birth asphyxia-related neonatal mortality • Neonatal mortality rate Various mortality measurement approaches were considered but ruled out, including implementing a parallel data collection system or using nonintervention control periods, regions, or facilities to not receive HBB. Instead, we attempted using existing historical health information systems, but pre-HBB facility-level mortality data quality were too limited for comparison. 
• Stillbirth rate 
• Frequency of neonatal deaths resulting from birth asphyxia 
MethodsPurposeKey IndicatorsOutcome
OSCEs Measure provider-level knowledge and skills • Percentage of providers passing the OSCE 1004 OSCEs were conducted. 
Facility and equipment checklists Measure coverage and equipment availability • Availability and functionality of bag-mask devices and penguin suction devices 335 facility checklists were completed. 
Comparing HBB training lists with delivery room registers Determine training coverage and proportion of deliveries conducted by HBB-trained providers • Percentage of birth attendants trained in HBB 335 facility registers were reviewed. 
• Percentage of deliveries attended by an HBB-trained provider in the past 1 and 3 mo 
Focus group discussions Identify provider-reported program strengths, weaknesses, barriers, recommendations, and elements important for scale-up • Likes, dislikes, barriers, and recommendations for the HBB program 222 focus group discussions were conducted, involving 599 participants. 
Perinatal death audits To understand circumstances surrounding recent newborn deaths and stillbirths, including contributing factors, clinical management, and opportunities for improvement • Clinical presentation 110 perinatal death audits were conducted across 15 regions. 
• Clinical management 
• Causes of death 
Cost analysis Quantify cost of national rollout and maintenance of the HBB program in Tanzania • Cost of implementing HBB program per facility Projected total costs for rollout to the 25 mainland regions would be $4 million, with an additional $5.6 million for another 5 y of program support after rollout (Chaudhury et al10). 
• Projected cost of national rollout of HBB 
Mortality assessment Measure the impact of the HBB program on birth asphyxia-related neonatal mortality • Neonatal mortality rate Various mortality measurement approaches were considered but ruled out, including implementing a parallel data collection system or using nonintervention control periods, regions, or facilities to not receive HBB. Instead, we attempted using existing historical health information systems, but pre-HBB facility-level mortality data quality were too limited for comparison. 
• Stillbirth rate 
• Frequency of neonatal deaths resulting from birth asphyxia 

Trained evaluation team members conducted OSCEs on all HBB-trained staff present at a facility during the 4- to 6-week and 4- to 6-month facility follow-up visits. The sampling method was a stratified, purposive sample of facilities within all 15 districts receiving the CIFF-supported HBB program. Facilities were chosen to ensure that regional and district hospitals (n = 33), health centers (n = 35), and dispensaries (n = 163) were all represented.

Data analyses included descriptive and inferential analyses, including frequencies, means, and univariate and multivariate logistic regressions. The mean OSCE scores at the 4- to 6-week follow-up visits conducted by the implementing partner and by the evaluation team were compared via paired t tests and a significance level of P < .05. All quantitative statistical analyses were done in Stata version 12 (Stata Corp, College Station, TX).

For analysis of the focus group discussions, an inductive thematic approach was applied to the written discussion notes.11 Investigators used NVivo 10 software (QSR International, Victoria, Australia) for data management and theme generation, along with regular peer consultations to increase interpretive rigor.

This study was approved by the institutional review board at Massachusetts General Hospital (Boston, MA), the National Institute for Medical Research (Dar es Salaam, Tanzania), and the Ministry of Health and Social Welfare (Dar es Salaam, Tanzania). Informed verbal consent was obtained from all study participants.

There were 13 169 providers who attended the HBB training. Two (0.02%) of these providers were removed from the analysis because they had incomplete OSCE data in which ≥1 of the OSCE questions had a missing value. Of the 13 167 providers included in the analysis, the majority were nurses or nurse-midwives (51.5%) and providers working at dispensaries (54.9%) (Table 2).

TABLE 2

Health Cadre and Facility Level of Providers Attending HBB Courses

Health CadreHospitalHealth CenterDispensaryOtherTotal (%)
Medical doctor 70 23 101 (0.8) 
Assistant medical officer 154 116 43 317 (2.4) 
Clinical officer 177 277 1119 1576 (12.0) 
Assistant clinical officer 11 32 314 357 (2.7) 
Nurse or nurse-midwife 2194 1607 2964 18 6783 (51.5) 
Medical assistant 558 679 2709 3951 (30.0) 
Other health worker 14 11 57 82 (0.6) 
Total (%) 3178 (24.1) 2730 (20.7) 7229 (54.9) 30 (0.2) 13 167 (100) 
Health CadreHospitalHealth CenterDispensaryOtherTotal (%)
Medical doctor 70 23 101 (0.8) 
Assistant medical officer 154 116 43 317 (2.4) 
Clinical officer 177 277 1119 1576 (12.0) 
Assistant clinical officer 11 32 314 357 (2.7) 
Nurse or nurse-midwife 2194 1607 2964 18 6783 (51.5) 
Medical assistant 558 679 2709 3951 (30.0) 
Other health worker 14 11 57 82 (0.6) 
Total (%) 3178 (24.1) 2730 (20.7) 7229 (54.9) 30 (0.2) 13 167 (100) 

At the 4- to 6-week follow-up visits, the evaluation team found that the percentage of birth attendants who received formal HBB training varied by region from 50.8% to 88.0% (Table 3). By the 4- to 6-month follow-up visits, all regions except Tanga and Dar es Salaam saw a decline in the percentage of formally HBB-trained birth attendants (range 40.4%–71.3%). The percentage of deliveries attended by a formally trained HBB provider in the past month ranged from 50.3% to 97.8% at 4 to 6 weeks and from 23.6% to 90.1% at 4 to 6 months.

TABLE 3

HBB Training Coverage of Birth Attendants and Deliveries

RegionPercentage of Birth Attendants Who Have Completed Formal HBB TrainingPercentage of Deliveries by Formally Trained HBB Providers in Past 1 MoPercentage of Deliveries by Formally Trained HBB Providers in Past 3 Mo
4–6 Wk After Training4–6 Mo After Training4–6 Wk After Training4–6 Mo After Training4–6 Wk After Training4–6 Mo After Training
Pwani 64.7 63.5 55.6 56.6 19.4 52.3 
Lindi 84.0 70.6 75.1 67.6 29.7 66.7 
Dar es Salaam 53.7 56.5 56.7 65.8 18.9 19.3 
Morogoro 67.1 50.7 56.7 23.6 24.6 40.7 
Iringa 88.0 71.3 75.5 51.3 30.3 57.2 
Ruvuma 62.3 40.4 64.8 40.0 54.6 24.9 
Mbeya 64.3 45.3 67.0 50.8 24.5 48.6 
Manyara 75.3 63.8 81.9 90.1 26.3 64.4 
Arusha 63.5 55.8 59.3 55.3 21.4 52.3 
Kilimanjaro 79.2 54.9 70.8 61.1 23.7 49.8 
Tanga 67.0 69.2 50.3 84.7 13.4 26.6 
Singida 86.5 69.6 97.8 44.0 28.3 45.1 
Kigoma 68.4 64.0 66.1 54.8 19.7 63.2 
Kagera 50.8 45.5 74.2 37.7 25.0 38.9 
Mara 54.1 45.5 62.3 50.3 18.0 52.8 
Total 68.7 57.1 62.9 51.5 22.7 38.0 
RegionPercentage of Birth Attendants Who Have Completed Formal HBB TrainingPercentage of Deliveries by Formally Trained HBB Providers in Past 1 MoPercentage of Deliveries by Formally Trained HBB Providers in Past 3 Mo
4–6 Wk After Training4–6 Mo After Training4–6 Wk After Training4–6 Mo After Training4–6 Wk After Training4–6 Mo After Training
Pwani 64.7 63.5 55.6 56.6 19.4 52.3 
Lindi 84.0 70.6 75.1 67.6 29.7 66.7 
Dar es Salaam 53.7 56.5 56.7 65.8 18.9 19.3 
Morogoro 67.1 50.7 56.7 23.6 24.6 40.7 
Iringa 88.0 71.3 75.5 51.3 30.3 57.2 
Ruvuma 62.3 40.4 64.8 40.0 54.6 24.9 
Mbeya 64.3 45.3 67.0 50.8 24.5 48.6 
Manyara 75.3 63.8 81.9 90.1 26.3 64.4 
Arusha 63.5 55.8 59.3 55.3 21.4 52.3 
Kilimanjaro 79.2 54.9 70.8 61.1 23.7 49.8 
Tanga 67.0 69.2 50.3 84.7 13.4 26.6 
Singida 86.5 69.6 97.8 44.0 28.3 45.1 
Kigoma 68.4 64.0 66.1 54.8 19.7 63.2 
Kagera 50.8 45.5 74.2 37.7 25.0 38.9 
Mara 54.1 45.5 62.3 50.3 18.0 52.8 
Total 68.7 57.1 62.9 51.5 22.7 38.0 

Jhpiego conducted 22 512 OSCEs as part of HBB implementation, of which 22 454 have complete OSCE data: 13 167 (58.6%) immediately after HBB training and 9287 (41.4%) at the 4- to 6-week follow-up visits. The evaluation team conducted among a stratified sample of facilities an additional 1004 OSCEs, of which 984 have complete OSCE data: 726 (73.8%) at the 4- to 6-week follow-up visits and 258 (26.2%) at the 4- to 6-month follow-up visits.

There were 681 providers that had OSCE assessments at 4 to 6 weeks by both the implementing partner and the evaluation team. A paired t test of this subset of data showed a statistically significant difference between the mean OSCE score of those conducted by the implementing partner (mean 16.4; SD 4.5) and the mean OSCE score of those conducted by the evaluation team (mean 16.8; SD 4.3) (t = −3.061; P = .002). Although the difference is statistically significant, the absolute difference (0.4 points out of 23 points) helps confirm a previously determined high interrater reliability and the ability to compare OSCE scoring across partners.6 

The mean OSCE score was highest (mean 18.9; SD 3.3), and the greatest percentage of providers passed the OSCE (87.1%) immediately after HBB training, compared with the follow-up visits (Table 4). The mean OSCE score dropped at the 4- to 6-week follow-up visits conducted by the implementing partner (mean 17.9, SD 4.1; 79.4% passing) and by the evaluation team (mean 16.8, SD 4.2; 68.3% passing). By the 4- to 6-month visits, the mean OSCE score had decreased to 15.8 (SD 4.7), with 55.8% of providers passing the OSCE. However, there was a decrease in fall-off of scores over the course of the program that was associated with the introduction of interim interventions (Fig 2).

TABLE 4

HBB OSCE Results by Region and Time Since Training

RegionImplementing PartnerEvaluation Team
Immediately After HBB4–6 Wk After HBB4–6 Wk After HBB4–6 Mo After HBB
Mean (SD)% PassingMean (SD)% PassingMean (SD)% PassingMean (SD)% Passing
Pwani 19.4 (3.3) 87.5 16.9 (4.4) 68.1 17.4 (3.3) 79.4 12.7 (5.4) 45.5 
Lindi 18.9 (3.2) 86.0 15.3 (4.6) 60.6 14.6 (3.7) 42.1 13.0 (4.6) 28.6 
Dar es Salaam 17.7 (4.0) 74.5 15.9 (4.2) 57.9 17.0 (3.9) 68.8 17.1 (3.1) 65.4 
Morogoro 18.2 (2.7) 86.8 16.8 (4.4) 71.2 15.7 (4.0) 65.5 14.1 (4.7) 36.8 
Iringa 19.0 (3.2) 86.0 17.4 (3.6) 77.5 16.3 (3.8) 64.7 16.4 (4.8) 60.0 
Ruvuma 18.7 (3.4) 84.3 17.5 (4.1) 75.8 16.8 (4.3) 71.0 19.4 (3.6) 88.9 
Mbeya 18.7 (3.4) 85.3 17.3 (4.2) 75.3 16.3 (4.3) 48.6 16.8 (4.8) 64.3 
Manyara 18.2 (3.8) 80.6 18.0 (4.1) 79.2 17.5 (4.4) 80.0 15.6 (3.7) 54.5 
Arusha 19.4 (3.0) 90.6 18.6 (3.8) 85.6 17.3 (3.7) 73.8 16.0 (3.8) 50.0 
Kilimanjaro 19.0 (3.2) 86.4 18.6 (3.5) 86.6 16.8 (3.8) 64.4 16.3 (3.9) 55.6 
Tanga 19.1 (3.2) 89.1 18.9 (3.6) 88.8 17.3 (4.2) 70.0 17.5 (4.2) 72.7 
Singida 19.5 (2.8) 93.6 18.7 (3.7) 87.2 16.7 (4.7) 67.6 14.6 (5.9) 38.5 
Kigoma 19.2 (3.3) 88.8 18.4 (4.2) 83.3 17.0 (5.5) 73.4 14.8 (5.7) 47.1 
Kagera 19.8 (2.9) 92.6 20.0 (3.1) 93.2 20.1 (3.3) 88.9 18.9 (2.3) 100.0 
Mara 19.7 (2.8) 93.8 18.5 (3.8) 86.8 16.6 (4.9) 76.9 16.8 (4.0) 63.6 
Total 18.9 (3.3) 87.1 17.9 (4.1) 79.4 16.8 (4.2) 68.3 15.8 (4.7) 55.8 
RegionImplementing PartnerEvaluation Team
Immediately After HBB4–6 Wk After HBB4–6 Wk After HBB4–6 Mo After HBB
Mean (SD)% PassingMean (SD)% PassingMean (SD)% PassingMean (SD)% Passing
Pwani 19.4 (3.3) 87.5 16.9 (4.4) 68.1 17.4 (3.3) 79.4 12.7 (5.4) 45.5 
Lindi 18.9 (3.2) 86.0 15.3 (4.6) 60.6 14.6 (3.7) 42.1 13.0 (4.6) 28.6 
Dar es Salaam 17.7 (4.0) 74.5 15.9 (4.2) 57.9 17.0 (3.9) 68.8 17.1 (3.1) 65.4 
Morogoro 18.2 (2.7) 86.8 16.8 (4.4) 71.2 15.7 (4.0) 65.5 14.1 (4.7) 36.8 
Iringa 19.0 (3.2) 86.0 17.4 (3.6) 77.5 16.3 (3.8) 64.7 16.4 (4.8) 60.0 
Ruvuma 18.7 (3.4) 84.3 17.5 (4.1) 75.8 16.8 (4.3) 71.0 19.4 (3.6) 88.9 
Mbeya 18.7 (3.4) 85.3 17.3 (4.2) 75.3 16.3 (4.3) 48.6 16.8 (4.8) 64.3 
Manyara 18.2 (3.8) 80.6 18.0 (4.1) 79.2 17.5 (4.4) 80.0 15.6 (3.7) 54.5 
Arusha 19.4 (3.0) 90.6 18.6 (3.8) 85.6 17.3 (3.7) 73.8 16.0 (3.8) 50.0 
Kilimanjaro 19.0 (3.2) 86.4 18.6 (3.5) 86.6 16.8 (3.8) 64.4 16.3 (3.9) 55.6 
Tanga 19.1 (3.2) 89.1 18.9 (3.6) 88.8 17.3 (4.2) 70.0 17.5 (4.2) 72.7 
Singida 19.5 (2.8) 93.6 18.7 (3.7) 87.2 16.7 (4.7) 67.6 14.6 (5.9) 38.5 
Kigoma 19.2 (3.3) 88.8 18.4 (4.2) 83.3 17.0 (5.5) 73.4 14.8 (5.7) 47.1 
Kagera 19.8 (2.9) 92.6 20.0 (3.1) 93.2 20.1 (3.3) 88.9 18.9 (2.3) 100.0 
Mara 19.7 (2.8) 93.8 18.5 (3.8) 86.8 16.6 (4.9) 76.9 16.8 (4.0) 63.6 
Total 18.9 (3.3) 87.1 17.9 (4.1) 79.4 16.8 (4.2) 68.3 15.8 (4.7) 55.8 
FIGURE 2

Implementation data on percentage of providers passing the HBB OSCE, by region and time since training. Regions are listed in the chronological order in which training occurred, beginning in May 2013 through November 2014.

FIGURE 2

Implementation data on percentage of providers passing the HBB OSCE, by region and time since training. Regions are listed in the chronological order in which training occurred, beginning in May 2013 through November 2014.

Among the OSCE tasks that providers most often completed correctly were washing hands or using alcohol rub and then putting on gloves, drying the newborn, and improving ventilation with ≥1 technique (Table 5). Among the tasks that providers most often did not complete correctly were preparing an area for ventilation, checking the equipment before delivery, and stimulating breathing by rubbing the newborn’s back.

TABLE 5

Percentage of Providers Correctly Completing Each OSCE Task, by Time Since Training

OSCE TaskImplementing PartnerEvaluation Team
Immediately After Training, % Correct4–6 Wk After Training, % Correct4–6 Wk After Training, % Correct4–6 Mo After Training, % Correct
Washes hands or uses alcohol rub, puts on gloves 91.6 92.0 88.6 77.5 
Prepares an area for ventilation and checks equipment 71.2 81.3 69.2 53.9 
Dries infant thoroughly 94.8 92.3 92.4 88.3 
Removes wet cloth and replaces with a dry cloth 89.2 81.0 67.9 56.2 
Positions head and clears airway by using a penguin sucker 87.7 80.4 81.1 79.5 
Stimulates breathing by rubbing the back 68.1 54.6 45.4 48.3 
Cuts cord and places infant so that it can be ventilated 81.0 81.7 80.4 74.8 
Keeps infant warm during resuscitation 87.7 83.2 79.8 68.6 
Starts ventilation within 1 min of birth 85.2 82.2 75.0 67.1 
Ventilates at 40 breaths per min 80.4 75.8 73.6 70.9 
Looks at the exposed chest for chest movement 84.8 82.5 85.7 84.9 
Improves ventilation with ≥1 of the following 88.6 84.9 81.8 80.2 
 Repositions head, opens mouth slightly, and clears secretions 
 Reapplies mask 
 Squeezes bag harder 
 Recognizes infant is doing well and stops ventilation 81.3 80.1 87.6 90.3 
OSCE TaskImplementing PartnerEvaluation Team
Immediately After Training, % Correct4–6 Wk After Training, % Correct4–6 Wk After Training, % Correct4–6 Mo After Training, % Correct
Washes hands or uses alcohol rub, puts on gloves 91.6 92.0 88.6 77.5 
Prepares an area for ventilation and checks equipment 71.2 81.3 69.2 53.9 
Dries infant thoroughly 94.8 92.3 92.4 88.3 
Removes wet cloth and replaces with a dry cloth 89.2 81.0 67.9 56.2 
Positions head and clears airway by using a penguin sucker 87.7 80.4 81.1 79.5 
Stimulates breathing by rubbing the back 68.1 54.6 45.4 48.3 
Cuts cord and places infant so that it can be ventilated 81.0 81.7 80.4 74.8 
Keeps infant warm during resuscitation 87.7 83.2 79.8 68.6 
Starts ventilation within 1 min of birth 85.2 82.2 75.0 67.1 
Ventilates at 40 breaths per min 80.4 75.8 73.6 70.9 
Looks at the exposed chest for chest movement 84.8 82.5 85.7 84.9 
Improves ventilation with ≥1 of the following 88.6 84.9 81.8 80.2 
 Repositions head, opens mouth slightly, and clears secretions 
 Reapplies mask 
 Squeezes bag harder 
 Recognizes infant is doing well and stops ventilation 81.3 80.1 87.6 90.3 

In unadjusted logistic regression, the odds of passing the OSCE significantly decreased with time since HBB training (4–6 week odds ratio [OR] 0.515, P < .001; 4–6 month OR 0.143, P < .001) (Table 6). Working at a dispensary, compared with at a hospital, decreased the odds of passing the OSCE (OR 0.716, P < .001), and medical assistants, compared with medical doctors, were less likely to pass the OSCE (OR 0.433, P = .002).

TABLE 6

Unadjusted and Adjusted ORs For Passing the HBB OSCE, by Time Since Training, Facility Level, Health Cadre, and Region

VariablesUnadjusted OR (95% CI)PAdjusted OR (95% CI)P
Time since HBB training     
 Implementing partner scoring immediately after HBB Reference — Reference — 
 Implementing partner scoring at 4–6 wk after HBB 0.515 (0.475–0.559) <.001 0.520 (0.479–0.563) <.001 
 Evaluation team scoring at 4–6 mo after HBB 0.143 (0.105–0.195) <.001 0.126 (0.093–0.173) <.001 
Facility level     
 Hospitals Reference — Reference — 
 Health centers 1.055 (0.922–1.206) .437 1.137 (0.989–1.306) .071 
 Dispensaries 0.716 (0.644–0.797) <.001 0.850 (0.758–0.953) .005 
 Other 0.760 (0.300–1.926) .563 0.692 (0.267–1.795) .449 
Health cadre     
 Medical doctors Reference — Reference — 
 Assistant medical officer 0.918 (0.500–1.685) .782 0.903 (0.482–1.693) .751 
 Clinical officer 0.769 (0.445–1.330) .347 0.831 (0.471–1.465) .522 
 Assistant clinical officer 0.898 (0.495–1.629) .723 1.011 (0.545–1.874) .973 
 Nurses or nurse midwives 1.021 (0.596–1.747) .941 1.089 (0.625–1.896) .764 
 Medical assistants 0.433 (0.253–0.743) .002 0.458 (0.262–0.801) .006 
 Other 0.706 (0.332–1.503) .367 0.763 (0.350–1.665) .497 
Region 1.000 (0.991–1.010) .960 1.009 (0.999–1.019) .079 
VariablesUnadjusted OR (95% CI)PAdjusted OR (95% CI)P
Time since HBB training     
 Implementing partner scoring immediately after HBB Reference — Reference — 
 Implementing partner scoring at 4–6 wk after HBB 0.515 (0.475–0.559) <.001 0.520 (0.479–0.563) <.001 
 Evaluation team scoring at 4–6 mo after HBB 0.143 (0.105–0.195) <.001 0.126 (0.093–0.173) <.001 
Facility level     
 Hospitals Reference — Reference — 
 Health centers 1.055 (0.922–1.206) .437 1.137 (0.989–1.306) .071 
 Dispensaries 0.716 (0.644–0.797) <.001 0.850 (0.758–0.953) .005 
 Other 0.760 (0.300–1.926) .563 0.692 (0.267–1.795) .449 
Health cadre     
 Medical doctors Reference — Reference — 
 Assistant medical officer 0.918 (0.500–1.685) .782 0.903 (0.482–1.693) .751 
 Clinical officer 0.769 (0.445–1.330) .347 0.831 (0.471–1.465) .522 
 Assistant clinical officer 0.898 (0.495–1.629) .723 1.011 (0.545–1.874) .973 
 Nurses or nurse midwives 1.021 (0.596–1.747) .941 1.089 (0.625–1.896) .764 
 Medical assistants 0.433 (0.253–0.743) .002 0.458 (0.262–0.801) .006 
 Other 0.706 (0.332–1.503) .367 0.763 (0.350–1.665) .497 
Region 1.000 (0.991–1.010) .960 1.009 (0.999–1.019) .079 

CI, confidence interval; —, not applicable (reference).

In multivariate logistic regression, time since HBB training, facility level, and health cadre were all independent predictors of passing the OSCE, whereas region did not predict passing the OSCE. Providers were less likely to pass the OSCE at the 4- to 6-week visits and at the 4- to 6-month visits (4–6 week OR 0.520, P < .001; 4–6 month OR 0.126, P < .001) compared with immediately after HBB training. Providers working at dispensaries were less likely to pass the OSCE than providers working at hospitals (OR 0.850, P = .005), and medical assistants were less likely to pass the OSCE compared with medical doctors (OR 0.458, P = .006).

Bag-mask devices and penguin suction devices were highly available and functional over time (Table 7). The percentage of bag-mask devices that were available but in a location other than immediately accessible in the labor and delivery ward or obstetric theater decreased from 12% to 0% from the 4- to 6-week to 4- to 6-month visits. Over the same time, the percentage of facilities where bag-mask devices were available in all newborn resuscitation areas increased from 84% to 92%.

TABLE 7

Availability and Functionality of Resuscitation Equipment, by Time

Resuscitation Equipment4–6 Wk After Training4–6 Mo After Training
Mean (Rangea) Number of Devices per FacilityPercentage of Facilities With EquipmentPercentage of Devices That Are FunctionalMean (Rangea) Number of Devices per FacilityPercentage of Facilities With EquipmentPercentage of Devices That Are Functional
Bag-mask devices 2.3 (1.4–3.3) 95 98.0 2.4 (1.4–5.5) 90 95.5 
Penguin suction devices 3.1 (2.6–5.2) 97 93.5 2.5 (1.1–5.4) 97 97.3 
HBB posters — 63 — — 76 — 
Resuscitation Equipment4–6 Wk After Training4–6 Mo After Training
Mean (Rangea) Number of Devices per FacilityPercentage of Facilities With EquipmentPercentage of Devices That Are FunctionalMean (Rangea) Number of Devices per FacilityPercentage of Facilities With EquipmentPercentage of Devices That Are Functional
Bag-mask devices 2.3 (1.4–3.3) 95 98.0 2.4 (1.4–5.5) 90 95.5 
Penguin suction devices 3.1 (2.6–5.2) 97 93.5 2.5 (1.1–5.4) 97 97.3 
HBB posters — 63 — — 76 — 

—, not applicable.

a

Range of mean number of devices per facility by region.

Over the course of the 3-year program, >200 focus group discussions were held with providers who received HBB training. Overall, the feedback was highly positive (Table 8). Providers reported feeling more confident and better equipped to resuscitate newborns. Birth attendants reported that the HBB equipment simplified resuscitation. Providers considered the supportive supervision visits and the follow-up visits critical for skill retention. They also appreciated the hands-on training on mannequins. However, they reported some dislikes and barriers, including that some providers still struggled to use the bag-mask device and thought that the 1-day training was too short to properly cover all the material. Additionally, providers thought that the financial incentives for training were too small and that intrafacility rotation of trained attendants limited the impact of the training.

TABLE 8

Themes From Focus Group Discussions With HBB-Trained Providers

Themes
Likes Dislikes 
• Providers largely pleased with the HBB program in Tanzania as a whole. • Fewer reported program dislikes than likes. 
• Improved provider-level newborn resuscitation knowledge, skills, and confidence. • Perceived small per diems during training. 
• Simple HBB equipment and poster. • Training too short. 
• Follow-up visits are good refresher. • A lot covered in 1 d, and sometimes not enough time for hands-on practice. 
• Practical hands-on training and ability to practice at their facility is useful. • Some HBB attendees were not birth attendants, and some birth attendants were not able to attend the trainings. 
HBB trainee manual and handouts useful for future reference. 
Barriers Recommendations 
• Regular rotation of HBB-trained providers away from labor ward. • Additional training desired. 
• Not enough time to practice HBB at work because of staff shortages and large patient volumes. • Train the staff who were not able to attend original training. 
• Limited space for newborn resuscitation in some smaller facilities and operating theaters. • Frequent follow-up visits and supportive supervision. 
• Many providers still struggling with using bag-mask device. • Integrate HBB training into medical and nursing preservice curricula. 
Themes
Likes Dislikes 
• Providers largely pleased with the HBB program in Tanzania as a whole. • Fewer reported program dislikes than likes. 
• Improved provider-level newborn resuscitation knowledge, skills, and confidence. • Perceived small per diems during training. 
• Simple HBB equipment and poster. • Training too short. 
• Follow-up visits are good refresher. • A lot covered in 1 d, and sometimes not enough time for hands-on practice. 
• Practical hands-on training and ability to practice at their facility is useful. • Some HBB attendees were not birth attendants, and some birth attendants were not able to attend the trainings. 
HBB trainee manual and handouts useful for future reference. 
Barriers Recommendations 
• Regular rotation of HBB-trained providers away from labor ward. • Additional training desired. 
• Not enough time to practice HBB at work because of staff shortages and large patient volumes. • Train the staff who were not able to attend original training. 
• Limited space for newborn resuscitation in some smaller facilities and operating theaters. • Frequent follow-up visits and supportive supervision. 
• Many providers still struggling with using bag-mask device. • Integrate HBB training into medical and nursing preservice curricula. 

This study adds valuable information on implementing and sustaining the HBB program, because it is one of the first and most extensive assessments of HBB at scale. We analyzed 3 years of monitoring and evaluation data from >13 000 health care providers in 15 of Tanzania’s mainland regions. Although the evaluation used several methods to assess the success of the program, including a cost analysis and qualitative interviews, the current study focuses largely on the assessment of providers’ resuscitation skills and equipment availability.

The initial success of HBB on developing provider-level resuscitation skills is evidenced by 87.1% of providers passing the HBB OSCE after training. However, to determine skill retention, we also assessed HBB skills over time. During the follow-up visits led by the implementing partner, the percentage of providers with passing OSCE scores decreased by 7.7% at 4 to 6 weeks and by 31.3% at 4 to 6 months. Early identification of this decline in skills prompted the development of interim interventions, which included structured tools to facilitate on-the-job HBB practice and supportive supervision by trained maternity ward leaders and district-level trainers. Similar to findings in a recent review of newborn resuscitation training approaches, we saw an improvement in HBB skill retention with the implementation of these interventions. Another remarkable accomplishment of the program was that the percentage of providers who passed the OSCE improved as the program matured (Fig 2). However, the odds of a provider passing the OSCE differed by the facility level in which they worked and their health cadre. The odds of passing the OSCE were 15% lower among providers working at dispensaries compared with hospitals. Furthermore, compared with medical doctors, medical assistants were less likely to pass the OSCE. However, medical assistants are usually unskilled birth attendants and perform very few deliveries. Skilled birth attendants such as nurse-midwives and clinical officers, who perform most of the deliveries, showed no significant difference in odds of passing the OSCE compared with medical doctors.

In addition to improving providers’ resuscitation skills, other successes of the HBB program in Tanzania were creating critical attention for newborn health and achieving widespread program coverage and implementation in an affordable way. In each region, by 4 to 6 weeks after implementation of the program, 68.7% of providers working in the maternity wards and obstetric theaters had received the HBB course. Coverage dropped to 57.1% at 4 to 6 months, probably because of attrition, new hires, and the common practice of interdepartmental staff rotation. Although the majority of providers were still trained in HBB at long-term follow-up visits, the decline in trained providers indicates the need for continued trainings. Coverage of resuscitation equipment, such as bag-mask devices and penguin suction devices, was high at >90% of facilities. Equipment also proved to be durable, in that almost all devices were still functional at the longer-term follow-up visits. The cost to train providers and equip the facilities was ∼$600 USD per health facility, equating to a national rollout cost of ∼$4 million, making HBB affordable in low-resource regions.10 A recent study examining HBB costs per lives saved in a rural missionary hospital in Tanzania found implementation costs of $233 USD per life saved ($4.21 USD per life year gained) and program maintenance costs of $80 USD per life saved ($1.44 USD per life year gained).12 

There were challenges to at-scale implementation of the HBB program. Sustaining high coverage of trained providers was difficult, as discussed. Although attempts were made to measure program impact on neonatal mortality by using historic and existing health information systems, reliably quantifying this impact was not possible. Measuring mortality in resource-limited settings is challenging because deaths often go undocumented, especially before a program is implemented. Subsequently, with program implementation comes better documentation, which can initially lead to a perceived increase in mortality. HBB training also corrects providers’ often-misunderstood definitions of perinatal outcomes, which can lead to an apparent increase in newborn mortality (eg, previously miscategorized “stillbirths” are correctly categorized as newborn deaths). Options to help overcome some of these data challenges include parallel data collection systems, sentinel panel of districts or facilities, active surveillance, and composite indicators (eg, perinatal death). But these options come with significant financial and logistical costs for country-level programs.

Recognizing the challenges of resources and clinical coverage during trainings, the American Academy of Pediatrics supports 1- or 2-day HBB trainings for providers.9 The Tanzania MOHSW chose a 1-day training approach because of the desire to expeditiously and cost-effectively train a large number of providers across large geographic regions with finite resources. This 1-day training was complemented by the mentioned interim interventions, specifically facility-based supportive supervision visits and structured on-the-job training intended to mitigate any negative impact from the shorter training course. The ideal duration of training remains unclear because data show that frequent, short trainings are more effective than long, infrequent trainings.13 Therefore, the ideal neonatal resuscitation training program for large-scale implementation in resource-limited settings may permit 1-day courses if they are followed by frequent refresher or formal in-facility mentorship programs. Consistent with this approach, Jhpiego is currently implementing a new facility-based clinical mentorship program to address concerns with skill retention. Nevertheless, it is still possible that the shorter, pragmatic 1-day training approach reduced the potential impact of the HBB program on provider skills. The program was implemented largely as a stand-alone vertical training program; more effectively embedding HBB into comprehensive newborn programming and preservice training would probably help ensure cost-effectiveness and sustainability.

The current study does have limitations. First, this country-level HBB implementation in Tanzania may not necessarily be generalizable to other resource-limited settings. However, its 3-year design, large and diverse sample, and implementation within 15 large regions significantly increase the study’s representativeness. Second, social desirability bias could have occurred during focus group discussions. The use of an independent and experienced evaluation team and the confidential nature of the discussions probably mitigated this bias. Third, it is unknown whether good performance on OSCEs translates into improved care in the clinical setting.14,15 Fourth, some providers were lost to follow-up through attrition or rotation to other departments. Lastly, as discussed, we were not able to accurately measure the program’s impact on neonatal mortality because of the poor quality and availability of data in historic health information systems. Additional program focus and resources will probably be necessary to accurately measure mortality impact in these resource- and data-limited settings.

The HBB program in Tanzania trained >13 000 providers with high rates of knowledge and skill adoption. Although there was notable fall-off in skills over time, early identification of this fall-off and implementation of interim interventions probably contributed to the decline in skill fall-off. Importantly, the HBB program was highly praised and accepted among providers working in labor and delivery wards, although they request additional training to the 1-day program. Although the program has gained acceptability and shown success in equipping providers with neonatal resuscitation knowledge and skills, the program’s impact on neonatal mortality at scale is unknown because of the difficulty in measuring impact by using existing data systems.

     
  • CIFF

    Children’s Investment Fund Foundation

  •  
  • HBB

    Helping Babies Breathe

  •  
  • MOHSW

    Ministry of Health and Social Welfare

  •  
  • OR

    odds ratio

  •  
  • OSCE

    objective structured clinical examination

Ms Arlington carried out the data analyses and drafted the manuscript; Drs Kairuki, Isangula, Meda, Thomas, and Temu, Mr Mponzi, and Dr Bishinga helped design and conduct the study, carried out the data collection and initial analyses, and reviewed and revised the manuscript; Drs Msemo and Azayo helped conceptualize and design the study and provided national leadership throughout the study; Dr Nelson conceptualized and designed the study, supervised the study, helped carry out data collection and initial analyses, and helped draft the manuscript; and all authors approved the final manuscript as submitted.

FUNDING: This program and its independent evaluation were funded by the Children’s Investment Fund Foundation (London, UK).

COMPANION PAPER: A companion to this paper can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2016-4172.

The authors acknowledge and thank for their critical assistance Tanzania’s national and district HBB trainers, Tanzania’s regional and district health officials, Tanzania’s HBB Steering Committee and HBB Data Committee, and Jhpiego-Tanzania’s in-country leadership and newborn resuscitation program staff.

1
Lawn
JE
,
Blencowe
H
,
Oza
S
, et al;
Lancet Every Newborn Study Group
.
Every Newborn: progress, priorities, and potential beyond survival.
Lancet
.
2014
;
384
(
9938
):
189
205
[PubMed]
2
Liu
L
,
Johnson
HL
,
Cousens
S
, et al;
Child Health Epidemiology Reference Group of WHO and UNICEF
.
Global, regional, and national causes of child mortality: an updated systematic analysis for 2010 with time trends since 2000.
Lancet
.
2012
;
379
(
9832
):
2151
2161
[PubMed]
3
World Health Organization
. Skilled attendants at birth. Global Health Observatory (GHO) data. Available at: www.who.int/gho/maternal_health/skilled_care/skilled_birth_attendance_text/en/. Accessed October 4, 2016
4
Kak
LP
,
Johnson
J
,
McPherson
R
, et al
. Helping Babies Breathe: lessons learned guiding the way forward. Available at: www.helpingbabiesbreathe.org/docs/HBB-Report-2010-2015.pdf. Accessed June 7, 2015
5
Msemo
G
,
Massawe
A
,
Mmbando
D
, et al
.
Newborn mortality and fresh stillbirth rates in Tanzania after helping babies breathe training.
Pediatrics
.
2013
;
131
(
2
). Available at: www.pediatrics.org/cgi/content/full/131/2/e353
[PubMed]
6
Reisman
J
,
Martineau
N
,
Kairuki
A
, et al
.
Validation of a novel tool for assessing newborn resuscitation skills among birth attendants trained by the Helping Babies Breathe program.
Int J Gynaecol Obstet
.
2015
;
131
(
2
):
196
200
[PubMed]
7
Isangula
KG
,
Kassick
ME
,
Kairuki
AK
, et al
.
Provider perceptions and experiences with the large-scale “Helping Babies Breathe” training program in Tanzania [published online ahead of print September 29, 2016].
Paediatr Int Child Health
.
doi: 10.1080/20469047.2016.1219119
8
Isangula
KG
,
Kassick
ME
,
Kairuki
AK
, et al
.
Provider experiences with the large-scale “Helping Babies Breathe” training programme in Tanzania.
Paediatr Int Child Health
.
2016
;
29
:
1
7
[PubMed]
9
American Academy of Pediatrics
. HBB implementation guide. 2011. Available at: https://www.aap.org/en-us/Documents/hbs_implementationguide_english.pdf
. Accessed March 19, 2017
10
Chaudhury
S
,
Arlington
L
,
Brenan
S
, et al
.
Cost analysis of large-scale implementation of the “Helping Babies Breathe” newborn resuscitation-training program in Tanzania.
BMC Health Serv Res
.
2016
;
16
(
1
):
681
[PubMed]
11
Braun
V
,
Clarke
V
.
Using thematic analysis in psychology.
Qual Res Psychol
.
2006
;
3
(
2
):
77
101
12
Vossius
C
,
Lotto
E
,
Lyanga
S
, et al
.
Cost-effectiveness of the “Helping Babies Breathe” program in a missionary hospital in rural Tanzania.
PLoS One
.
2014
;
9
(
7
):
e102080
[PubMed]
13
Mduma
E
,
Ersdal
H
,
Svensen
E
,
Kidanto
H
,
Auestad
B
,
Perlman
J
.
Frequent brief on-site simulation training and reduction in 24-h neonatal mortality—an educational intervention study.
Resuscitation
.
2015
;
93
:
1
7
[PubMed]
14
Ersdal
HL
,
Vossius
C
,
Bayo
E
, et al
.
A one-day “Helping Babies Breathe” course improves simulated performance but not clinical management of neonates.
Resuscitation
.
2013
;
84
(
10
):
1422
1427
[PubMed]
15
Reisman
J
,
Arlington
L
,
Jensen
L
,
Louis
H
,
Suarez-Rebling
D
,
Nelson
BD
.
Newborn resuscitation training in resource-limited settings: a systematic literature review.
Pediatrics
.
2016
;
138
(
2
):
e20154490
[PubMed]

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.