Preparedness for Pediatric Office Emergencies: A Multicenter, Simulation-Based Study

We conducted a multicenter, observational study over a 15-month period (December 2018 to March 2020), which included the following components:

1. Measurement of adherence to the AAP Policy Statement for pediatric office emergency preparedness by using an in-person survey.

2. Measurement of the quality of care for 2 simulated pediatric patients with emergencies.

Institutional review board approval was obtained from each collaborating site on the basis of each participating AMC’s requirements; the majority of reviews were deemed exempt.

Investigators from 9 pediatric AMCs each recruited a minimum of 2 pediatric primary care offices in their respective geographic regions. Offices were excluded if they provided subspecialty care or were physically connected to an ED or urgent care center. Urban and/or suburban setting was defined by an estimated EMS response time of <15 minutes on the basis of recorded EMS response times in previously categorized pediatric emergencies.¹ 

All lead investigators and research coordinators from the participating AMCs participated in online train-the-trainer sessions to ensure standardization of the study protocol execution. These sessions were conducted via Zoom (Zoom Video Communications, Inc, San Jose, CA) by the study principal investigators with each participating AMC. Each session lasted 90 minutes and involved reviewing the study protocol, each simulation scenario, performance and preparedness measurement checklists, and standardization of all data entry into a centralized database via Qualtrics (Qualtrics Inc, Provo, UT). The AMC team members included pediatric emergency physicians, pediatric critical care physicians, registered nurses, respiratory therapists, medics, and nurse practitioners. The script of these sessions is provided in Appendix 1 in the Supplemental Information.

The recruitment and selection of pediatric offices occurred through multiple methods, including AMC physician liaisons, personal connections, and phone calls and/or e-mails distributed to selected sites. Each pediatric office identified a champion to serve as the site contact who worked with the AMC team to coordinate all study phases.

Each AMC conducted an in-person site visit to each participating office and completed a pediatric emergency preparedness checklist-based tool. During this measurement, a trained member of the AMC study team completed a checklist for each office with the pediatric office champion. These 2 individuals directly identified all the items on the checklist (eg, locating each piece of equipment and reviewing policies and protocols). If the champion and study team were unsure or unable to locate the scored item during the measurement, no credit was given for that item in the tool.

The in situ simulation-based session was conducted to measure the quality of emergency care provided in these offices and to help identify target areas for improvement. Teams from each office were recruited for the simulations to mirror their typical team composition. These teams were composed of general pediatricians (1–2 physicians), advanced practice providers, registered nurses, medical assistants, and administrative staff. Participants were protected from clinical responsibilities during these simulations. Champions recruited providers at each site via an E-mail sent 1 month before the simulation.

All sessions were conducted in the actual office space to promote realism. Teams were required to find the appropriate resources, equipment, and medications within their office. However, these items were replaced by equipment and medications provided by the simulation team to prevent the participating office from incurring costs or using of their limited supplies.

Details of the simulation cases are summarized in our previously published work.¹⁵  Briefly, each simulation session consisted of 2 scenarios: a child aged 7 years presenting with asthma and a child aged 5 years presenting with seizure. A standardized and scripted orientation was used to introduce the project and the AMC team and described the format and expectations for the day. At each office, 1 or 2 teams participated. In offices with small numbers of staff, the same team of providers participated in both simulations. In larger offices, the staff were separated into 2 teams, with one caring for the patient and the other team observing. Both teams participated in the debriefings for each case. No incentives were given for participation in the simulated sessions. Other details of the simulation setup, the cases, and checklists are presented in Appendix 2A and 2B in the Supplemental Information.

Within 48 hours of completing the preparedness checklist and simulation-based measurements, each AMC team entered the collected data into a centralized data collection form in Qualtrics (Qualtrics Inc., Provo, Utah). These data were compiled into a database to collate data of all participating offices.

We recorded measures of office preparedness for pediatric emergencies at each participating office by using a checklist derived from the AAP Policy Statement. This checklist included equipment, supplies, medications, policies, and protocols. Items in this checklist are considered in the AAP guideline as either essential for all offices or strongly suggested for offices with EMS response times of >10 minutes.

1. Essential equipment and supplies checklist (20 items)

2. Policies and protocols checklist (9 items)

3. Strongly suggested equipment and medications checklist (32 items)

Items on all 3 checklists were not weighted, and a dichotomous response of yes or no was given on the basis of the availability of each item. Each checklist score was normalized to a 100-point scale. A total emergency preparedness score was calculated on the basis of the essential equipment and supplies checklist and the policies and protocols checklist. All sites’ demographics were also collected, including EMS response time, distance to the nearest ED, number of staff in the office (staff size), affiliation with an AMC, annual patient volume, and other demographics. Annual patient volume was divided into 4 quartiles.

These scenarios and checklists were created by content experts in pediatric emergency medicine and critical care by using evidence-based guidelines and best practices. Content validity was obtained by using a consensus-based approach among experts. Developed scenarios and checklists were piloted and iteratively adapted in simulations at independent sites that did not participate in this study.

All data were manually entered into Qualtrics and transferred into SPSS, v.27.0 (IBM SPSS Statistics, IBM Corporation), with which all statistical analyses were performed. For categorical variables, frequencies and percentages were calculated. For continuous variables, medians and interquartile ranges (IQRs) were calculated. Bivariate analyses were used to explore associations between practice characteristics and pediatric preparedness scores, which included independent t tests or 1-way analysis of variance tests for normal continuous data. Bivariate analyses were also used to describe the association between the pediatric preparedness checklist (eg, regular emergency drills and/or practice and EMS activation) and the simulation checklist by using χ² tests. We used additional bivariate analyses to explore associations between practice characteristics and simulation scores using Mann–Whitney U tests.

Finally, we used a generalized linear mixed model to model emergency preparedness scores as the dependent variables with a robust variance estimator to account for within-practice correlation to examine which variables explain higher emergency preparedness. Potential covariates in the model (eg, patient volume, staff size, AMC affiliation, and type of practice) were introduced if bivariate analyses were significant at P < .10. This model accounts for the nesting of teams within each site. Unstandardized β coefficients were reported.

Forty-two offices from 9 states participated in the study. Sixteen (38%) offices were recruited from the state of Indiana; 10 (24%) offices were recruited from the state of Maryland (Table 1). The median annual patient volume was 6000 patients, the median staff size was 17, and the median EMS response time was 5 minutes. The quartiles for the annual patient volume were quartile 1: ≤3919 patients; quartile 2: 3920 to 6000 patients; quartile 3: 6001 to 8819 patients; and quartile 4: ≥8820 patients. Fifteen (36%) of the offices were independent practices (ie, not part of a larger group).

A total of 48 teams participated in the simulation across 42 offices. There was a median of 6 members per team, and the median ratio of physicians to team members was 0.2 (IQR: 0.14–0.33) (Table 1).

The offices’ mean emergency preparedness score across the 42 offices was 74.7% (SD: 12.9). The mean essential equipment and supplies score was 82.2% (SD: 15.1). All participating offices had an oxygen source; pediatric oxygen masks; pediatric bag valve masks, nebulizers, and albuterol; a pulse oximeter; and blood pressure cuffs. The least available items were infant bag valve masks, cardiac arrest boards, and oral airways in 18%, 43%, and 47% of offices, respectively. The mean policies and protocols score was 57% (SD: 25.6). Only 33% of offices had policies for regular self-assessment, and only 43% conducted regular emergency drills (Table 2). The mean preparedness score for the additional equipment was 38% (SD: 28.3) (Supplemental Table 1).

Bivariate analyses revealed that several variables were associated with pediatric preparedness scores (Fig 1). Independent practices had lower pediatric preparedness scores compared with those that were part of a larger group (β = −11.89, 95% confidence interval [CI]: −19.33 to −4.45). Higher annual patient volume and larger total staff size were associated with higher scores (β = .001; 95% CI: 0.00 to 0.001; P = .017 and β = .51; 95% CI: 0.19 to 0.83; P = .002, respectively). AMC affiliation and the presence of learners were not associated with higher scores. Looking at a multivariable regression model, higher annual patient volume was no longer significantly associated with higher preparedness. Independent practices were associated with lower preparedness scores, whereas larger total staff size was associated with higher scores in the multivariable model (β = −10.52; 95% CI: −17.74 to −3.29; P = .005 and β = .41; 95% CI: 0.09 to 0.73; P = .014, respectively). The results of these analyses are in Table 3.

The median performance score of the asthma case was 63.6% (IQR: 43.2–81.2), whereas the median score of the seizure case was 69.2% (IQR: 46.2–80.8). Details of performance with the subcomponents of each case-based checklist are reported in Table 4. We stratified the simulation performance by practice characteristics in Supplemental Table 2.

We looked at simulation scores stratified by 2 of the checklist items, regular emergency drills and/or practice (essential checklist 6) and a standardized process of contacting EMS (essential checklist 7). The asthma simulation score was lower at sites that had policies for regular drills: 82% (IQR: 64–91) for those without a policy for regular drills versus 50% (IQR: 36–64) for those with a policy (P = .002). The difference was nonsignificant for the seizure scores: 69% (IQR: 62–85) vs 54% (17–77) (P = .302). Additionally, offices that had a standardized process of contacting EMS had a higher rate of activating EMS for the simulation cases (72% vs 47%; P = .014).

This study revealed variability in both pediatric emergency preparedness (adherence to the AAP Policy Statement) and the quality of emergency care measured by in situ simulations in a national sample of pediatric primary care offices. This is the first multicenter study to directly measure pediatric office emergency preparedness and quality of emergency care. These measurements provide the first step in improvement efforts aimed to ensure optimal care for children presenting to offices with emergencies. These data can be used to guide the development of interventions to improve emergency preparedness and care delivery in pediatric offices.

We found that nonindependent offices, with a larger staff size and with higher annual patient volume, had higher preparedness scores. However, on multivariable analysis, only larger staff size and nonindependent practices were significantly associated with higher preparedness scores. This higher preparedness could be secondary to additional staff to focus on this topic and additional resources available as a part of a larger system of practices. Larger staff size may correlate with higher patient volume and subsequently more exposure to pediatric patients, which could contribute to the higher preparedness score.

Despite the AAP Policy Statement being reaffirmed multiple times since its initial publication, pediatric office emergency preparedness remains highly variable. With this study, we add to the evidence reported in previous studies in which poor pediatric office preparedness was noted through self-reported surveys that are prone to bias. Notably, the in-person direct observation survey methods conducted in this study are less prone to biases.^7,16,17 

The mean preparedness score of essential equipment and supplies was 82%, reflecting a higher score compared with what has been reported in our previous pilot report of 64%.¹⁵  Although some equipment items are rarely used in everyday office-based clinical care, it is concerning that 82% of offices did not have an infant bag valve mask and would therefore need to wait for EMS arrival to administer life-saving ventilation to an infant. This highlights the need to have this equipment available and maintain the skills necessary to care for patients in respiratory distress, the most common emergency encountered in the office setting. A cardiac arrest board is another example of a potentially life-saving piece of equipment that was not available in the majority of offices, likely because of the extremely rare occurrence of cardiac arrests in the office setting. Lack of a board may lead to poor cardiopulmonary resuscitation quality before the arrival of EMS. The mean preparedness score for the additional equipment, noted as essential only if EMS response time was >10 min, was much lower (38%). This may again be attributed to its rare use in the office setting. In future work, researchers should explore the benefit of these items to potentially guide changes to the existing guidelines’ designation of essential equipment.

The mean preparedness score of policies and protocols was low at 57%, with common deficiencies in conducting a regular self-measurement and regular emergency drills and/or practice and having written protocols for emergency response. Despite the AAP recommendation of performing regular mock codes in the office, our findings were aligned with previously published surveys in which 20% to 40% presence of regular mock codes in offices was reported. This highlights major opportunities for future improvement through providing templates for standardized policies.^6,8,16 

Surprisingly, we did not find a correlation between office preparedness scores with simulation performance scores. This could be attributed to a small sample size or the fact that the presence of certain equipment and supplies does not necessarily translate to high-quality care. We noted that offices with policies for regular drills had lower asthma performance scores. This could be secondary to the poor quality of the simulation drills conducted by pediatric offices or the lack of rigorous validity of the simulation checklists used. We also noted that offices with a standardized process of contacting EMS had a higher rate of activating EMS during the simulations. This is an important finding because easy accessibility and contact of EMS will ensure timely transfer and definitive resuscitative care.

All participating sites received a customized preparedness report of office-based emergency preparedness and the quality of simulated care (Appendix 3 in the Supplemental Information). Additionally, all offices received clinical and educational resources and continued to collaborate with the academic medical centers to support improvement efforts. This collaborative model mirrors the components of our published emergency department (ED) readiness improvement collaborative situ simulations.^{10,11,15,18,19}  In our future work, we will focus on developing, implementing, and evaluating improvement interventions involving AMCs collaborating with regional offices.

Our study has a few limitations. Our recruitment method may have led to selection bias with the recruited office sites being more engaged in emergency preparedness, which may limit the generalizability of findings. We did not recruit any rural offices with low patient volume or offices that provided care to both children and adults. However, to mitigate this limitation, we recruited a spectrum of sites to represent the range of offices in the nation. Second, the emergency preparedness checklists we used have limited validity evidence, and the items are not weighted. However, these checklists are derived from an AAP Policy Statement and represent the best checklists available in the literature. Similarly, the simulated case checklists we used have limited validity evidence regarding internal structure and consequences. Lastly, we did not obtain interrater reliability of the checklist scoring because only 1 study personnel performed the on-site measurement. However, all lead investigators and research coordinators underwent a train-the-trainer session to ensure consistency and standardization.

This study revealed variability in the emergency preparedness and the quality of simulated emergency care provided in pediatric primary care offices. Essential life-saving equipment, such as an infant bag valve mask, was missing in most offices, highlighting the need for efforts to assess and improve pediatric office emergency preparedness. Many offices did not have emergency policies and procedures. Academic and community partnerships are a promising strategy to address these gaps in preparedness because they were already found to be effective in the ED setting. With this study, we inform future efforts and initiatives to work collaboratively to update the current policy statement for pediatric offices preparedness for emergencies and serves as a baseline for developing interventions to improve emergency preparedness and emergency care in the pediatric office.

We thank the members of the ImPACTS group, including Ellie Hamburger, MD, Kathleen Kadow, MD, and Kathy Prestidge from Children’s National Pediatricians and Associates; Kimberly Dawson, MSN, RN, CPEN, TCRN, EMT, Heather Sobolewski, MSN, RN-BC, CHSE, from Nemours/Alfred I. duPont Hospital for Children; and Theresa A. Walls, MD, MP H, from Center for Simulation, Advanced Education, and Innovation, Children’s Hospital of Philadelphia, Philadelphia, PA. We also thank the pediatric offices and their champions who facilitated the study and worked collaboratively with their pediatric AMC team; the members of the International Network for Simulation-based Pediatric Innovation, Research and Education who have helped to shape this project with their contributions; and the International Pediatric Simulation Society for providing the International Network for Simulation-based Pediatric Innovation, Research and Education with space at their annual meetings for our research group.

AAP

American Academy of Pediatrics

AMC

academic medical center

CI

confidence interval

ED

emergency department

EMS

emergency medical services

IQR

interquartile range