Choosing Wisely Campaign: Report Card and Achievable Benchmarks of Care for Children’s Hospitals

From 2014 to 2016, a multicenter working group from 4 children’s hospital members of CHA and analytic support specialists worked on a collaborative project. The PHIS database (CHA, Lenexa, KS) was queried to develop a report card for the SHM-CWC pediatric recommendations. This database includes clinical and daily billing data from 48 tertiary care children’s hospitals and accounts for ∼20% of all US pediatric hospitalizations every year. Participating hospitals submit de-identifying data. A joint effort between CHA and participating hospitals ensures the quality of the data submitted, as has been previously described.⁸  These data are subjected to a routine quality check with each submission and within each report. In total, 32 PHIS hospitals met the strict quality standards for inclusion of submitted data. The rest of the hospitals did not meet the strict data quality requirements from PHIS and were excluded from this study.

This research was exempt from approval by the institutional review board of Nicklaus Children’s Hospital.

The development of a hospital-specific report card required multiple conference calls among the authors and included the following steps:

1. assembly of content (all authors) and design of the report card (development experts);

2. selection and definition of measures, population inclusion, and exclusion criteria;

3. selection of ABCs as a method for establishing targets in the report card;

4. selection of the established CHA report card format for consistency in presentation and ease of use; and

5. education regarding content and purpose provided to users through Webcasts and newsletters.

Four of the 5 SHM-CWC pediatric recommendations were able to be translated into 5 quality metrics. The first recommendation (avoiding the use of CXR in asthma and bronchiolitis) was converted into 2 separate metrics, 1 for each condition. The fifth SHM-CWC pediatric recommendation, limiting the use of continuous pulse oximetry for acute respiratory illness unless the patient is receiving supplemental oxygen, could not be translated into a metric by using PHIS data because of inconsistent reporting of the use of these resources. Consequently, the report card has 5 metrics:

1. percentage of patients with uncomplicated asthma receiving CXR;

2. percentage of patients with uncomplicated bronchiolitis receiving CXR;

3. percentage of patients with uncomplicated bronchiolitis receiving bronchodilators;

4. percentage of patients with LRTI receiving systemic corticosteroids; and

5. percentage of patients with uncomplicated GER receiving acid suppressor therapy.

These metrics have the study population selected as the denominator and the number of patients who received the contraindicated therapy or intervention as a numerator. Thus, the percentages displayed on the report card represent nonadherence to the recommendations.

Children were eligible for inclusion if they were hospitalized (inpatient or observation status) between January 1, 2013 and September 31, 2015. The SHM-CWC pediatric recommendations refer to “uncomplicated” conditions. Some of the original recommendations do not specify age groups.¹  We defined different age groups for each condition as well as inclusion and exclusion criteria aimed at identifying only uncomplicated cases, as follows:

• Asthma: 3 to 17 years. Considering that the diagnosis of asthma in infants and children younger than 3 years old is challenging and it may overlap with other causes of wheezing. The upper limit of 17 years of age will capture most children with this condition.

• Bronchiolitis: ≥1 month to <1 year. This age group was selected to exclude newborn infants who tend to have a more severe disease and patients older than 1-year-old who may have other causes of recurrent wheezing that may require inhaled bronchodilators and CXR.

• LRTI: ≥1 month to <2 years. This age group closely matches the age group of the SHM-CWC pediatric recommendation on avoiding systemic corticosteroids. Newborn infants are excluded considering the likelihood of more severe disease. This group includes mostly cases of bronchiolitis and pneumonia who do not benefit from this intervention.

• GER: ≤1 year. This age group matches the age group of infants, for whom the specific SHM-CWC pediatric recommendation is intended. In this age group, the presence of physiologic, uncomplicated GER is more likely, and the routine use of acid suppressors is not recommended.

As inclusion criteria, the working group developed comprehensive lists of International Classification of Diseases, Ninth Revision (ICD-9) codes⁹  that identify bronchiolitis, asthma, LRTI and GER. Exclusion criteria comprised a detailed list of ICD-9 codes⁹  corresponding to multiple conditions and comorbidities that may complicate the course or treatment of these conditions. Additional exclusion criteria also included were length of stay (LOS) of more than 5 days, hospitalizations with charges for intensive care services, and extreme severity of illness (level 4) from the All Patient Refined Diagnosis Related Group (3M Corp, Wallingford, CT). This way, the cases selected for the study have a short LOS, no complications requiring intensive care, and no other chronic or comorbid conditions.

Additionally, to minimize misclassification, for each category of respiratory conditions, we excluded cases with any of the other conditions if they were listed as a comorbidity because they may require different management. For example, patients with an ICD-9 code indicating asthma are excluded from the population of patients with bronchiolitis and LRTI, and vice versa.

The resulting cohorts represent uniform populations of uncomplicated conditions, as explicitly targeted in the SHM-CWC pediatric recommendations.

The interventions and therapies not routinely recommended are CXR, bronchodilators, systemic corticosteroids, and acid suppressor therapy. Development experts define these interventions through a mapping process in which PHIS hospitals’ data are translated from their charge codes into clinical translation classification codes. These codes are a meaningful tool to make hospital billing comparable among institutions, and they are updated yearly.

Table 1 shows the age groups for each condition, a summary of the inclusion and exclusion criteria, and the 5 metrics selected. Readers can obtain the complete lists of inclusion and exclusion criteria and the most current group of clinical translation classification codes identifying interventions not recommended in the SHM-CWC pediatric recommendations by contacting Carla Hronek (carla.hronek@childrenshospitals.org). The detailed ICD-9 lists of inclusion and exclusion criteria for each condition and the lists of codes for procedures and interventions are available in the Supplemental Fig 4, which details the exclusion list for respiratory conditions).

We compiled the combined hospital performances on each of the quality metrics included in the report card. Categorical variables are summarized with counts and percentages, whereas medians and interquartile ranges (IQRs) summarized continuous variables.

By using a previously published method,^5–7  ABCs for the 5 SHM-CWC pediatric metrics included in this study were estimated. ABCs are calculated by first ranking the hospitals on the basis of their performance on a specific metric. Then, the best-performing hospitals that comprise 10% of the total patient population are selected. Finally, calculated ABCs represent the average from those top-performing centers.⁵ 

Results are presented in tables and graphics. All statistical analyses were performed by using SAS version 9.3 (SAS Institute Inc, Cary, NC).

Table 2 shows the demographic information and the APR-DRG distribution of the cohorts for each condition included in the SHM-CWC pediatric recommendations. The study included 32 487 cases of asthma, 23 531 cases of bronchiolitis, 5963 cases of LRTI, and 2946 cases of GER. A slight predominance of boys in all categories was noted: 61.9% in asthma, 59.6% in bronchiolitis, 55% in LRTI, and 53.6% in GER. We also observed a higher prevalence of non-Hispanic African American patients with asthma (43%). Inversely, non-Hispanic white patients were more prevalent in the bronchiolitis, LRTI, and GER groups, mostly in the GER cohort (53.7%). The “minor” (severity level 1) APR-DGR severity of illness for each condition represents more than half of the patients in each of the diagnostic groups, ranging from 52.7% in LRTI to 76.4% in asthma.

Table 3 shows the calculated ABCs for each of study populations, along with the number of hospitals used to create these benchmarks. These calculated ABCs are 22.3% of patients with asthma and 19.8% of patients with bronchiolitis having a CXR performed, 17.9% of patients with bronchiolitis receiving bronchodilators, 5.5% of patients with LRTI treated with systemic corticosteroids, and 32.2% of patients with GER treated with acid suppressor therapy. According to the methodology described above, only 3 to 4 hospitals were included in the estimation of the ABCs, representing ∼10% of the population for each cohort.

Figure 1 represents the variation in performance among the participating hospitals for the 5 metrics included in the study in box plots. Each measure, except steroid use in LRTIs, shows significant variability among institutions, as depicted by the whiskers on each box plot: CXR in asthma (median: 34.7%, IQR: 28.5%–45.9%), CXR use in bronchiolitis (median: 34.4%, IQR: 27.9%–49%), bronchodilators use in bronchiolitis (median: 55.4%, IQR: 32.3%–64.9%), and use of acid suppressor therapy in GER (median: 59.4%, IQR: 49.9%–71.2%). The use of systemic corticosteroids in LRTI shows less variance (median: 13.5%, IQR: 11.1%–17.9%). Additionally, we observed a clear gap between hospital performances and ABCs, as represented by the black circles, in all conditions.

Figure 2 shows the performance of each hospital, benchmarked against the ABCs for each of the SHM-CWC pediatric recommendations in a heat map. In general, more hospitals performed further from the ABC in the use of relievers for bronchiolitis and the use of acid suppressor therapy in GER, represented by the darkest gray boxes (>20 percentage points from benchmark) and medium gray boxes (11–20 percentage points from benchmark). On the other hand, participating hospitals were closer to or met the ABC regarding the avoidance of steroids for LRTI, represented by the more frequent lighter gray (0–10 percentage points from benchmark) and clear boxes (meet benchmark). Few hospitals were closer to or met benchmarks (ABCs) on most metrics, and none were able to meet benchmarks or to reach <10% variability from the benchmark on all 5 metrics.

Figure 3 displays bar graphics that show the variation in performance on the 5 metrics among the 32 hospitals included in the study. The shorter bars, closer to the ABC line, represent the best-performing centers, whereas the tallest bars correspond to institutions performing further away from the benchmark. The ranges of variation from the best-performing hospital (lower percentage in each metric) to the worst-performing center (higher percentage) are as follows: CXR in asthma (6.8%–71.2%); CXR in bronchiolitis (7.5%–69.8%); relievers in bronchiolitis (14.3%–82.6%); systemic corticosteroids in LRTI (4.8%–28%); and acid suppressor therapy in GER (26.2%–100%).

This report card is displayed in the standard format used by CHA in other clinical report cards. This format allows for a visual and numeric comparison of a given institution or physician group with both the peer group of participating hospitals and with the ABC for each metric. The report card allows customization by selecting specific timeframes and provider groups for benchmarking purposes. CHA- and PHIS-participating institutions can generate their customized, landscape version of the report card by accessing www.childrenshospitals.org.¹⁰ 

The SHM-CWC pediatric recommendations report card is the latest available for PHIS members. This is the first clinical report card that includes calculated ABCs as benchmarks of excellence and has a focus on overuse. Institutions can use these data to identify areas in which they are significant outliers for local QI projects.

The aggregated results presented here represent the first available data set of the performance of children’s hospital members of CHA after the recommendations were released.

As previously described, the use of ABCs is an original method intended to help individual clinician groups and institutions benchmark their performance. ABCs represent (1) a measurable level of excellence, (2) attainable goals, and (3) a benchmark derived from data in an objective and reproducible manner.^5–7  The calculated ABCs in our study compare well with the ABCs published by Parikh et al in 2014⁵  in 4 of our 5 metrics. The 2 studies used the same database (PHIS), but their study included data from the calendar year 2012, whereas our study included larger cohorts corresponding to 11 quarters (January 2013–September 2015). Furthermore, we used more restrictive criteria to define uncomplicated conditions, adding extreme APR-DRG, intensive care services, and any LOS of more than 5 days as exclusion criteria, in addition to including multiple chronic conditions or comorbidities. Despite some differences in defining uncomplicated cases, the finding of similar ABCs obtained in 2 different periods may suggest that no significant changes were present in practices among the included children’s hospitals since 2012, when the calculation of ABCs took place for the first time.

Peer-reviewed, society-endorsed guidelines for the conditions included in this study are widely available.^11–13  Clinical guidelines are designed to provide guidance while still preserving individual physician autonomy, modification on the basis of the clinical scenario, and accommodation for patient/family preferences. Although full adherence to clinical guidelines is not expected, reasonable and attainable goals of compliance are. However, as shown in Figs 1–3, significant variation in care still exists among the tertiary centers included in this study, as well as a clear gap between ABCs and actual performance by most of the institutions. These facts suggest an overuse of resources with an associated increase in costs related to hospital care. These findings have been previously reported by others.^5,14–17 

Recent evidence suggests some progress in the care of hospitalized children with bronchiolitis since the publication of American Academy of Pediatrics guidelines for this common inpatient condition.^18–21  However, as we reaffirm in this study, much unwarranted variation is still the norm, specifically in the excessive use of bronchodilators. Improvement will require concerted local efforts aimed at specific conditions. Recent evidence demonstrates that it is possible to change the culture and create meaningful and sustainable improvement through local efforts¹⁸  or multicenter QI collaboratives,²¹  both in tertiary children hospitals and community settings. Similar collaborative efforts to decrease unnecessary care in children with other lower respiratory tract illnesses, including asthma, are lacking. The significant improvements noted through such efforts addressing bronchiolitis, and the evidence of room for improvement demonstrated through our scorecard, could serve as the impetus for such work.

This work adds a newly calculated an achievable benchmark for the use of acid suppressor therapy in children with uncomplicated GER (32.2%), as shown in Table 2. Several studies, including systematic reviews, have described the ineffectiveness of acid-reducing medications in the treatment of children with GER.^22,23  Despite this, since the year 2000, a threefold increase in diagnosis (3.4%–12.3%) and a concomitant doubling of the use of proton pump inhibitors (31.5%–62.6%) have occurred in infants.²⁴  A substantial growing body of evidence suggests that this practice may also lead to significant harm.²⁵  Our study suggests that wide variation still exists in the use of acid suppressor therapy in children with GER across institutions (Figs 1–3). The fact that most institutions are far from meeting this benchmark of excellence, coupled with the strong evidence of ineffectiveness and potential harm of this therapy, makes this topic ripe for QI efforts at local and national levels.

The SHM-CWC pediatric recommendations report card could be specifically tailored by each institution member of the CHA consortium and has been readily available to use for internal QI prioritization since 2016. As shown in Figs 2 and 3, most hospitals tended to underperform on the use of relievers in bronchiolitis and prescription of acid suppressor therapy in GER. On the other hand, the use of steroids in LRTI represents the indicator with the lowest performance gap from the benchmark. Thus, prioritization of QI efforts could involve decreasing bronchodilator use in children with bronchiolitis and prescribing acid suppressor therapy in infants with uncomplicated, physiologic GER.

This report card represents an initial, valid tool to assess ongoing performance on the metrics selected for this study and their comparison with “best practices.” The inclusion of attainable benchmarks, newly and previously described for these specific conditions, makes the report card a much more meaningful instrument for benchmarking, by providing realistic goals on the basis of peer performance.

This study has several limitations, including those limitations associated with administrative and billing database research. Variation in documentation and coding practices may result in the presence of misidentified children in the cohorts and may hinder the risk adjustment for meaningful benchmarking. Additionally, the extensive list of inclusion and exclusion criteria based on chronic and/or comorbid conditions may exclude some uncomplicated cases for which the recommendations were intended.

Future versions of this report card based on data from the more granular International Classification of Diseases, Tenth Revision classification may be more revealing. Also, this study does not include information on whether a hospital implemented any changes or interventions in an attempt to follow the SHM-CWC pediatric recommendations.

Another limitation is the lack of data on hospital care provided at institutions that are not members of CHA. It is estimated that the majority (∼75%) of pediatric hospitalizations in the United States take place in hospitals that are not children’s hospitals.²⁰  Both the ABCs and the aggregated results may be different in that population.

The study does not include directions or suggestions on how to change practices through particular interventions or QI efforts. One final limitation is that the report card is available only to PHIS member institutions. However, by using the methodology and definitions described here and the information provided in the online supplement, other institutions could reproduce the report card and use the calculated ABCs as a benchmarking reference.

A report card for the purpose of assessing and benchmarking 5 metrics derived from the SHM-CWC pediatric recommendations is currently available for CHA and/or PHIS member institutions. In this study, we describe the development of the report card and the performance of these institutions from the first quarter (Q1) of 2013 to the third quarter (Q3) of 2015, along with calculated ABCs for each metric. In this study, we primarily demonstrate variation in practices and provide an initial pathway for change. In the future, individual institutions and collaborative projects can use these dashboards to track their performance and as a roadmap for improvement. QI efforts derived from the report card findings may lead to decreased variation in the care that hospitalized children receive for these common inpatient pediatric illnesses.

We acknowledge the Research Institute and the Medical Library from Nicklaus Children’s Hospital in Miami, Florida, for their assistance and involvement in the preparation and submission of this article.