BACKGROUND AND OBJECTIVES:

The Pediatric Respiratory Illness Measurement System (PRIMES) generates condition-specific composite quality scores for asthma, bronchiolitis, croup, and pneumonia in hospital-based settings. We sought to determine if higher PRIMES composite scores are associated with improved health-related quality of life, decreased length of stay (LOS), and decreased reuse.

METHODS:

We conducted a prospective cohort study of 2334 children in 5 children’s hospitals between July 2014 and June 2016. Surveys administered on admission and 2 to 6 weeks postdischarge assessed the Pediatric Quality of Life Inventory (PedsQL). Using medical records data, 3 PRIMES scores were calculated (0–100 scale; higher scores = improved adherence) for each condition: an overall composite (including all quality indicators for the condition), an overuse composite (including only indicators for care that should not be provided [eg, chest radiographs for bronchiolitis]), and an underuse composite (including only indicators for care that should be provided [eg, dexamethasone for croup]). Multivariable models assessed relationships between PRIMES composite scores and (1) PedsQL improvement, (2) LOS, and (3) 30-day reuse.

RESULTS:

For every 10-point increase in PRIMES overuse composite scores, LOS decreased by 8.8 hours (95% confidence interval [CI] −11.6 to −6.1) for bronchiolitis, 3.1 hours (95% CI −5.5 to −1.0) for asthma, and 2.0 hours (95% CI −3.9 to −0.1) for croup. Bronchiolitis overall composite scores were also associated with shorter LOS. PRIMES composites were not associated with PedsQL improvement or reuse.

CONCLUSIONS:

Better performance on some PRIMES condition-specific composite measures is associated with decreased LOS, with scores on overuse quality indicators being a primary driver of this relationship.

What’s Known on This Subject:

The face validity and feasibility of implementation of Pediatric Respiratory Illness Measurement System quality indicators has previously been established.

What This Study Adds:

Better performance on some of the Pediatric Respiratory Illness Measurement System condition-specific composite measures is associated with decreased length of stay, with overuse quality indicators being the primary driver. De-implementation of unnecessary care may shorten length of stay.

Health care quality can be measured by assessing the structure, process, or outcomes of care. However, care processes do not signify quality until their relationship to desirable outcomes is established.1 Because of small patient sample sizes with specific conditions and the rare occurrence of readily measured outcomes (eg, death), authors of few pediatric studies have examined the process-outcome link. In fact, most pediatric practice guidelines include many recommendations that are based on expert consensus opinion rather than strong scientific evidence.2,4 Often, providing care consistent with these guidelines is presumed, without an evidence base, to result in improved outcomes.

We previously developed a quality of care assessment tool, the Pediatric Respiratory Illness Measurement System (PRIMES).5 PRIMES is a quality measurement tool structured to generate composite quality scores by using readily available medical records (MRs) data for 4 respiratory conditions, asthma, bronchiolitis, croup, and community-acquired pneumonia (CAP), in the emergency department (ED) and inpatient settings (Supplemental Tables 4 through 11). The face validity and feasibility of implementation of PRIMES quality indicators has previously been established.5 The PRIMES indicators for each condition are based on scientific evidence linking included processes of care to outcomes or, when such evidence is lacking, expert consensus practice guidelines.6,9 Ultimately, 65% of the indicators were based on expert consensus opinion, versus 24% that were based on evidence from randomized controlled trials,5 thus revealing the need for additional validation to establish a link to improved outcomes.

Our objective was to further assess the validity of PRIMES by prospectively evaluating the relationship between performance on the PRIMES condition-specific composite scores and outcomes. We hypothesized that higher PRIMES composite scores would be associated with better health-related quality of life (HRQOL) after hospitalization, shorter length of stay (LOS), and decreased 30-day same-cause ED and/or inpatient reuse.

Between July 1, 2014, and June 30, 2016, we enrolled a prospective cohort of children receiving care at 1 of 5 US children’s hospitals belonging to the Pediatric Research in Inpatient Settings Network. Caregivers and their children were eligible to participate if the child was 2 weeks to 16 years old, if the family spoke English or Spanish, if the caregiver had access to a phone or the Internet, and if the child had 1 of the following diagnoses: asthma, bronchiolitis, croup, or CAP. Children with asthma, bronchiolitis, and CAP were recruited from the inpatient setting, whereas children with croup were recruited from both the ED and inpatient settings because of the low inpatient admission rates for croup.

Because the PRIMES indicators may not apply to children with underlying chronic illness, we excluded those with chronic comorbid conditions that could influence management of a PRIMES condition (eg, immunodeficiency). Because the PRIMES indicators apply to the routine management of the targeted respiratory conditions and may not apply to children who are critically ill, those directly admitted or transferred to the ICU during their hospitalization were also excluded (6%; n = 147). Additionally, children were excluded if they had a severe developmental delay because the HRQOL measure used has not been validated in this population. If a child’s primary diagnosis changed during their hospitalization (eg, from asthma to CAP), the diagnosis noted on the discharge summary was used for assignment to condition group. If the child’s discharge diagnosis was ultimately not a PRIMES condition, the child was excluded.

Monday through Friday (8 am–4 pm), trained research assistants attempted to enroll eligible caregivers and children on day 1 of admission; however, enrollment could occur any time during the first 72 hours of the admission or before discharge from the ED (croup only). The 72-hour window facilitated enrollment of those admitted on a Friday after 4 pm and still hospitalized the following Monday. Research assistants determined study eligibility by reviewing the child’s admission diagnosis and ED or inpatient history and physical using a standard study protocol. For unclear cases, the PRIMES site principal investigator confirmed eligibility. Enrolled caregivers provided consent to complete all study procedures.

The study procedures were reviewed and approved by the institutional review boards of the participating hospitals or the Western Institutional Review Board.

To assess HRQOL, caregivers who consented and children ≥8 years old who assented completed the 23-item Pediatric Quality of Life Inventory (PedsQL) 4.0 Generic Core Scales10 or the 36-item PedsQL Infant Scales.11 Caregivers also completed several demographic questions. In the hospital, surveys were completed at the time of study enrollment by using laptop computers or tablets (self-administered) or by in-person or telephone interview for those uncomfortable or unable to complete self-administered surveys. The survey was offered in both English and Spanish. At 2 to 6 weeks after discharge, the PedsQL was administered again; caregivers could complete their follow-up survey either online (sent as a Web link in an e-mail) or by telephone interview. This time interval allowed for multiple call attempts to complete follow-up assessments not completed online by 3 weeks postdischarge.

For each hospital, 2 research team members were trained on the clinical content and use of the PRIMES abstraction tool by the developer. Team members completed test abstractions for each target condition that were compared with gold standard abstractions created by the developer. Abstractors were considered fully trained when they could reliably abstract the test records for each condition with κ statistics of ≥0.75 for both indicator eligibility and scoring. Each abstractor was assigned half of the cases for each condition at their hospital. Eligibility for each sampled case was confirmed at the beginning of abstraction.

Because all enrolled children with asthma, bronchiolitis, and CAP were recruited from the inpatient setting, PRIMES indicators assessing ED discharge care were excluded (n = 10 indicators).5 All included PRIMES indicators could be classified as either overuse indicators that specify care processes that should not occur (eg, obtaining a chest radiograph in children with bronchiolitis) or underuse indicators that specify care processes that should occur (eg, offering an influenza vaccine for children with asthma who are not immunized). Three composite scores were constructed for each condition: a composite that included all overuse indicators, a composite that included all underuse quality indicators, and an overall composite that included all indicators (Supplemental Tables 4 through 11).

PRIMES condition-specific composite scores were calculated as continuous variables (0–100 scale) by dividing the number of times indicated care was received by the number of eligible cases, with higher scores indicating better quality of care.

Level of improvement in physical functioning after hospital discharge was assessed by using the PedsQL physical functioning subscale. We only used the physical functioning subscale because we hypothesized that higher adherence to the PRIMES standards of care would most likely influence physical (rather than psychosocial) functioning, and previous research has revealed the physical component to be most responsive to post-hospital recovery.12 For analyses, caregiver proxy report was used for all children aged <8 years. For older children, self-report was used when available and, if missing, parent proxy reports were used. On the basis of previous research, 4.5 was considered the minimal clinically important difference (MCID) on the 0 to 100 scale.10 

Each participating hospital contributes to the Pediatric Health Information System (PHIS) database.13 The PHIS was used to ascertain LOS, ICU admission during the study hospitalization, and 30-day same-cause ED and/or inpatient reuse. PHIS data were also used to determine insurance type (public versus commercial) and level of medical complexity for each child by applying the Pediatric Medical Complexity Algorithm (PMCA) version 3.0.14 PHIS data for enrolled children were de-identified and linked to their study identification numbers. Study identification numbers were used to link PHIS, MR abstraction, and survey data for each patient.

All analyses were performed at the patient level. Univariate statistics included medians, proportions, means, and SDs for all continuous predictor and outcome variables.

For all multivariable-adjusted analyses, we included the following covariates: child age, child sex, child race and/or ethnicity, PMCA category, insurance type, caregiver education, secondhand smoke exposure (≥1 smoker at home), days to follow-up survey (14–42 days postdischarge), and study site. A site-specific random effect was also included to account for clustering within study site.

We applied linear mixed-effects regression models to examine the associations between performance on the PRIMES condition-specific composite scores and the magnitude of change in the PedsQL scores. We regressed the PedsQL score at follow-up on PRIMES condition-specific composite scores (overall, overuse, and underuse) while controlling for admission PedsQL score. To assist with interpretation, we assessed the change in PedsQL scores associated with a 10-point increase in PRIMES composite scores.

We modeled inpatient LOS in hours as a continuous outcome and applied linear mixed-effects regression models. The primary predictors in these models were the PRIMES condition-specific composite scores. We assessed the magnitude of change in LOS in hours associated with a 10-point increase in PRIMES composite scores. For croup, children discharged from the ED were excluded.

Thirty-day same-cause ED and/or inpatient reuse at the original site of care was treated as a binary outcome (≥1 vs 0 ED return visits and/or inpatient readmissions) and analyzed by using mixed-effects logistic regression models. PRIMES condition-specific composite scores were the main predictors. We assessed the odds of a reuse event associated with a 10-point increase in PRIMES composite scores. Coefficient estimates for 2 covariates, PMCA category, and caregiver education would not converge because of the small number of reuse events and were dropped from models to obtain valid estimates.

We performed sensitivity analyses to examine the relationships between PRIMES condition-specific composite scores and PedsQL follow-up scores that included imputed physical functioning values for children with missing PedsQL follow-up surveys. Ten imputed data sets were generated by using multivariate imputation by chained equation.15,16 Mixed-effects models were fitted to each imputed data set. The estimates were then pooled to generate the final results.

All analyses were conducted by using R version 3.4.2.17 

We enrolled 2334 children and their caregivers across the 5 participating hospitals (sample size range n = 284–547 participants per hospital; response rate = 2334 of 3428 [68%]). The sample was 40% non-Hispanic white, 24% Hispanic, and 22% African American. The mean child age was 3.6 years (SD 3.8). Only 4% of the sample was classified as having a complex chronic illness by PMCA. These cases were chart reviewed by site principal investigators and deemed as conditions that would not influence respiratory illness management (Table 1).

TABLE 1

Study Population Characteristics and Outcomes

Patient and Caregiver CharacteristicsnaOverall Study Population (N = 2334)
Patient age, mean (SD), y 2334 3.6 (3.8) 
Patient sex 2260  
 Male, n (%) — 1333 (59) 
Patient race ethnicity, n (%) 2317  
 White — 917 (40) 
 African American — 518 (22) 
 Hispanic — 564 (24) 
 Otherb — 318 (14) 
Diagnosis, n (%) 2334  
 Asthma — 669 (29) 
 Bronchiolitis — 749 (32) 
 Croup — 345 (15) 
 CAP — 571 (24) 
PMCA category, n (%) 2328  
 No chronic illness — 1292 (56) 
 Noncomplex chronic illness — 935 (40) 
 Complex chronic illness — 101 (4) 
Secondhand smoke exposure 2326  
 Yes, n (%) — 551 (23.7) 
Insurance type, n (%) 2331  
 Private — 1010 (43) 
 Public or other — 1321 (57) 
Caregiver education, n (%) 2311  
 Less than high school — 236 (10) 
 High school graduate — 559 (24) 
 Higher than high school — 1516 (66) 
PedsQL physical functioning subscale score, mean (SD)c   
 Admission score 2310 56 (23) 
 Follow-up score 1686 91 (13) 
 Change scored 1669 36 (23) 
Inpatient LOS, mean (SD), median, h 2331 50.4 (43.2), 48.0 
Same-cause 30-d reuse, n (%)e 2331 107 (5) 
Patient and Caregiver CharacteristicsnaOverall Study Population (N = 2334)
Patient age, mean (SD), y 2334 3.6 (3.8) 
Patient sex 2260  
 Male, n (%) — 1333 (59) 
Patient race ethnicity, n (%) 2317  
 White — 917 (40) 
 African American — 518 (22) 
 Hispanic — 564 (24) 
 Otherb — 318 (14) 
Diagnosis, n (%) 2334  
 Asthma — 669 (29) 
 Bronchiolitis — 749 (32) 
 Croup — 345 (15) 
 CAP — 571 (24) 
PMCA category, n (%) 2328  
 No chronic illness — 1292 (56) 
 Noncomplex chronic illness — 935 (40) 
 Complex chronic illness — 101 (4) 
Secondhand smoke exposure 2326  
 Yes, n (%) — 551 (23.7) 
Insurance type, n (%) 2331  
 Private — 1010 (43) 
 Public or other — 1321 (57) 
Caregiver education, n (%) 2311  
 Less than high school — 236 (10) 
 High school graduate — 559 (24) 
 Higher than high school — 1516 (66) 
PedsQL physical functioning subscale score, mean (SD)c   
 Admission score 2310 56 (23) 
 Follow-up score 1686 91 (13) 
 Change scored 1669 36 (23) 
Inpatient LOS, mean (SD), median, h 2331 50.4 (43.2), 48.0 
Same-cause 30-d reuse, n (%)e 2331 107 (5) 

—, not applicable.

a

Indicates sample size when characteristic or outcome was not missing in the study population.

b

Other: Asian American or Pacific Islander and American Indian or Alaskan native.

c

Scored on a 0–100 scale with an MCID of 4.5.10 

d

The change score was obtained by subtracting the follow-up score from the admission score.

e

ED and/or inpatient reuse.

Unadjusted analyses used to examine PRIMES condition-specific composite scores revealed that average performance on the overall composites was highest for CAP (mean 88.7 [SD 12.8]) and lowest for asthma (mean 79.8 [SD 8.8]; Table 2). Scores were significantly higher on the overuse composites for asthma, bronchiolitis, and CAP compared with the corresponding underuse composite scores (Table 2). In contrast, the underuse composite score for croup (mean 89.9 [SD 10.5]) was significantly higher than the overuse composite score (mean 53.8 [SD 40.7]; P < .001).

TABLE 2

Unadjusted PRIMES Condition-Specific Composite Scores

ConditionPRIMES Composite Scores (0–100 Scale)Pa
Overall CompositeOveruse CompositebUnderuse Compositec
Asthma, mean (SD) 79.8 (8.8) 98.9 (7.6) 73.8 (11.4) <.001 
n 27 22 — 
Bronchiolitis, mean (SD) 86.9 (8.1) 91.8 (10.6) 80.5 (11.8) <.001 
n 16 — 
CAP, mean (SD) 88.7 (12.8) 89.6 (18.5) 87.4 (15.3) .03 
n — 
Croup, mean (SD) 86.8 (13.0) 53.8 (40.7) 89.9 (10.5) <.001 
n 15 14 — 
ConditionPRIMES Composite Scores (0–100 Scale)Pa
Overall CompositeOveruse CompositebUnderuse Compositec
Asthma, mean (SD) 79.8 (8.8) 98.9 (7.6) 73.8 (11.4) <.001 
n 27 22 — 
Bronchiolitis, mean (SD) 86.9 (8.1) 91.8 (10.6) 80.5 (11.8) <.001 
n 16 — 
CAP, mean (SD) 88.7 (12.8) 89.6 (18.5) 87.4 (15.3) .03 
n — 
Croup, mean (SD) 86.8 (13.0) 53.8 (40.7) 89.9 (10.5) <.001 
n 15 14 — 

n represents the number of quality indicators included in the composite score. —, not applicable.

a

The P value for the paired t test was used to compare overuse composite scores with underuse composite scores.

b

Overuse composites are calculated as a summary score across all quality indicators used to assess avoidance of unnecessary health services (eg, obtaining a respiratory syncytial virus test for children with bronchiolitis). Higher scores reveal better adherence to the indicators in the composite. For a full listing of the overuse quality indicators see Supplemental Tables 5, 7, 9, and 11.

c

Underuse composites are calculated as a summary score across all quality indicators used to assess the provision of indicated health services (eg, assessing influenza vaccination status for children with asthma admitted between November and March). Higher scores reveal better adherence to the indicators in the composite. For a full listing of the underuse quality indicators see Supplemental Tables 4, 6, 8, and 10.

Follow-up surveys were completed by 1686 enrolled families 2 to 6 weeks after discharge (follow-up response rate = 72%). The mean PedsQL physical functioning subscale score at admission was 56 (SD ±23; Table 1) and the mean postdischarge follow-up score was 91 (SD ±13), resulting in an average change score of 36 (SD ±23). Adjusted analyses revealed no association between PRIMES condition-specific composite scores and change in physical functioning at follow-up, with the exception of the croup underuse composite (Table 3). Contrary to our hypothesis, for every 10-point increase in the croup underuse composite score, the PedsQL physical functioning score decreased by 1.51 points (95% confidence interval [CI] −2.96 to −0.07; P < .05). The sensitivity analyses, which included imputed physical functioning follow-up scores, revealed similar results for all composite scores; however, the association between the croup underuse composite score and decreased improvement in PedsQL scores at follow-up was no longer significant (data not shown).

TABLE 3

Adjusted Associations Between PRIMES Condition-Specific Composite Scores and Outcomes of Care

PRIMES CompositeOutcomes of Care
PedsQL Change Score (95% CI)aLOS, h (95% CI)bReuse (95% CI)c
Asthma overall score (27 indicators) 0.09 (−1.54 to 1.72) 0.98 (−1.03 to 2.99) 1.10 (0.61 to 2.00) 
 Overuse score (5 indicators) 1.39 (−0.19 to 2.97) −3.10 (−5.51 to −0.70)* NAd 
 Underuse score (22 indicators) −0.04 (−1.28 to 1.19) 1.29 (−0.24 to 2.82) 1.02 (0.65 to 1.60) 
Bronchiolitis overall score (16 indicators) 0.51 (−1.13 to 2.14) −9.97 (−13.89 to −6.05)*** 1.04 (0.68 to 1.58) 
 Overuse score (7 indicators) 0.12 (−1.02 to 1.25) −8.81 (−11.57 to −6.06)*** 1.10 (0.80 to 1.51) 
 Underuse score (9 indicators) 0.65 (−0.53 to 1.84) 0.07 (−2.71 to 2.84) 1.00 (0.75 to 1.33) 
CAP overall score (7 indicators) −0.02 (−1.15 to 1.18) −1.13 (−4.26 to 2.01) 0.91 (0.62 to 1.33) 
 Overuse score (3 indicators) 0.20 (−0.59 to 0.98) −1.49 (−3.65 to 0.66) 0.97 (0.73 to 1.28) 
 Underuse score (4 indicators) −0.45 (−1.43 to 0.52) 1.45 (−1.12 to 4.01) 0.90 (0.66 to 1.22) 
Croup overall score (15 indicators) −1.02 (−2.14 to 0.10) −1.81 (−3.74 to 0.13) 1.11 (0.78 to 1.59) 
 Overuse score (1 indicator) −0.28 (−0.91 to 0.35) −2.24 (−4.16 to −0.32)* 0.85 (0.69 to 1.06) 
 Underuse scores (14 indicators) −1.52 (−2.97 to −0.07)* −0.78 (−3.74 to 2.18) 1.26 (0.78 to 2.03) 
PRIMES CompositeOutcomes of Care
PedsQL Change Score (95% CI)aLOS, h (95% CI)bReuse (95% CI)c
Asthma overall score (27 indicators) 0.09 (−1.54 to 1.72) 0.98 (−1.03 to 2.99) 1.10 (0.61 to 2.00) 
 Overuse score (5 indicators) 1.39 (−0.19 to 2.97) −3.10 (−5.51 to −0.70)* NAd 
 Underuse score (22 indicators) −0.04 (−1.28 to 1.19) 1.29 (−0.24 to 2.82) 1.02 (0.65 to 1.60) 
Bronchiolitis overall score (16 indicators) 0.51 (−1.13 to 2.14) −9.97 (−13.89 to −6.05)*** 1.04 (0.68 to 1.58) 
 Overuse score (7 indicators) 0.12 (−1.02 to 1.25) −8.81 (−11.57 to −6.06)*** 1.10 (0.80 to 1.51) 
 Underuse score (9 indicators) 0.65 (−0.53 to 1.84) 0.07 (−2.71 to 2.84) 1.00 (0.75 to 1.33) 
CAP overall score (7 indicators) −0.02 (−1.15 to 1.18) −1.13 (−4.26 to 2.01) 0.91 (0.62 to 1.33) 
 Overuse score (3 indicators) 0.20 (−0.59 to 0.98) −1.49 (−3.65 to 0.66) 0.97 (0.73 to 1.28) 
 Underuse score (4 indicators) −0.45 (−1.43 to 0.52) 1.45 (−1.12 to 4.01) 0.90 (0.66 to 1.22) 
Croup overall score (15 indicators) −1.02 (−2.14 to 0.10) −1.81 (−3.74 to 0.13) 1.11 (0.78 to 1.59) 
 Overuse score (1 indicator) −0.28 (−0.91 to 0.35) −2.24 (−4.16 to −0.32)* 0.85 (0.69 to 1.06) 
 Underuse scores (14 indicators) −1.52 (−2.97 to −0.07)* −0.78 (−3.74 to 2.18) 1.26 (0.78 to 2.03) 

Models were adjusted for child age, sex, race and/or ethnicity, PMCA category, secondhand smoke exposure (≥1 smoker at home), insurance type, caregiver education, days to follow-up survey (14–42 days postdischarge; PedsQL models only), and study site. A site-specific random effect was included to account for clustering within study site. Models used to predict 30-day ED and/or inpatient reuse included the same set of covariates, with the exception of PMCA category and caregiver education because of large SEs. A site-specific random effect was included to account for clustering within study site. NA, not applicable.

a

Scored on a 0–100 scale with an MCID of 4.5.10 The point estimate reveals the change in the PedsQL score between hospital admission and follow-up (14–42 days later) associated with a 10-point increase in the PRIMES condition-specific composite score.

b

The point estimate reveals the change in LOS (hours) associated with a 10-point increase in the PRIMES condition-specific composite score.

c

Same-cause 30-day ED and/or inpatient reuse at the original site of care. The point estimate reveals the odds of a reuse event associated with a 10-point increase in the PRIMES condition-specific composite score.d There were 17 reuse events among children with asthma, and all scored 100 on the overuse composite.

*

P < .05.

***

P < .001.

The PRIMES condition-specific overuse composite scores for bronchiolitis, asthma, and croup were all associated with significantly decreased LOS. For every 10-point increase in the PRIMES overuse composite scores (0–100 scale), LOS decreased by 8.8 hours (95% CI −11.6 to −6.1) for children with bronchiolitis, by 3.1 hours (95% CI −5.5 to −1.0) for children with asthma, and by 2.2 hours (95% CI −3.9 to −0.1) for patients with croup (Table 3).

Adjusted analyses did not reveal significant associations between PRIMES condition-specific composite scores and same-cause reuse (Table 3).

In this outcome validation study, higher scores on PRIMES bronchiolitis, asthma, and croup overuse composites were associated with significantly shorter inpatient LOS; however, this was not the case for the corresponding underuse composite scores. Higher PRIMES composite scores were not significantly associated with increased improvement in physical functioning at follow-up or decreased risk of reuse events. These findings indicate the potential importance of conducting outcome validation studies for individual process-of-care quality indicators.

Previous studies have demonstrated an association between implementing standardized clinical pathways and decreased LOS,18,22 including 1 pediatric study that was focused on asthma, bronchiolitis, and CAP21 and 2 others that were focused on bronchiolitis.20,22 These studies, however, did not distinguish between adherence to overuse versus underuse pathway standards of care and LOS. There are 2 conflicting interpretations of our study findings that may also apply to these previous investigations. One interpretation would be that de-implementation of unnecessary tests or treatments (ie, adhering to PRIMES overuse quality indicators), may reduce the risk of having to follow-up on false-positive tests (eg, positive results on blood cultures that are likely due to contaminants rather than true pathogens) or managing adverse reactions to unnecessary medications (eg, allergic reactions or diarrhea caused by treatment with unwarranted antibiotics).23,24 Avoiding these types of complications may result in shorter LOS and thus explain the observed associations. An alternative interpretation would be that children with more severe illness have longer LOS and are more likely to receive “off-guideline” tests and treatments. Some of these additional tests and treatments may go against the standards of care laid out by the PRIMES overuse indicators (eg, obtaining a chest radiograph or starting antibiotics in children hospitalized with bronchiolitis). Although we attempted to control for severity of illness in the current study by excluding children with ICU admissions and by adjusting for PMCA category, this likely did not adequately control for the wide range of severity observed in children with these respiratory illnesses cared for outside the ICU setting.

PedsQL physical functioning scores improved significantly between admission and follow-up for most children, with an average improvement of 36 points on the 0 to 100 scale. However, this improvement was not significantly associated with performance on PRIMES composite measures during hospitalization (Table 3). The degree of improvement in physical functioning for children with no chronic illness or noncomplex chronic illness after an acute hospitalization is not unexpected.12 The lack of association between higher PRIMES composite scores and greater improvement in physical functioning at follow-up suggests that this outcome may not be particularly useful when trying to outcome validate process-of-care quality indicators in populations of children who are generally healthy at baseline.

Higher adherence to the PRIMES condition-specific composites was also not associated with 30-day same-cause reuse. Reuse is a commonly used outcome measure in adults and is increasingly used in pediatric quality-of-care studies.25,27 However, in this large multicenter study, only 5% (n = 107) of participating children had a same-cause reuse event. Although this outcome measure may be useful in populations of children with medical complexity,28 its utility for other children hospitalized with acute respiratory illnesses may be limited.28,30 

Our primarily negative findings suggest that an alternative approach to outcome validation for both the PRIMES quality indicators and pediatric quality–measure composites in general may be needed. The PRIMES composites, with the exception of the croup overuse composite, include multiple quality indicators. Although our goal for this study was to explore the association between composite scores and outcomes, it is possible that at the individual quality-indicator level, we may have been able to observe significant associations between some of these process measures and the outcomes assessed. By combining multiple quality indicators into a composite measure, the potential association between one indicator and the outcomes examined may have been diminished by a lack of association observed for other indicators included in that composite. Assessing the associations between the individual PRIMES quality indicators and outcomes might also be more useful in terms of informing quality improvement (QI) interventions at the hospital level by providing guidance regarding selection of care processes to focus on for QI purposes.

This study has several limitations. Although the study was conducted in 5 geographically dispersed US children’s hospitals, our results may not be generalizable to children cared for in regions not represented by these hospitals or to children cared for in community hospitals, where most pediatric care is provided.31 Families without phone or Internet access were excluded, which may have further reduced the generalizability of the sample. However, national rates of smartphone ownership during the time frame of this study were 90% to 91% and desktop and/or laptop ownership was 70% to 74%.32 We were unable to adequately control for severity of illness, which may have confounded the relationship we observed between PRIMES overuse composite scores and LOS. For PedsQL assessments, increased time in days between discharge and follow-up survey completion was 1 of the strongest predictors of greater improvement in PedsQL scores after hospitalization for children with bronchiolitis, croup, or CAP (data not shown). Allowing >2 weeks for follow-up survey completion may have limited our ability to observe associations between performance on PRIMES composites and levels of improvement in physical functioning. We may have been under-powered to detect associations between performance on PRIMES composites and reuse outcomes given the rarity of these events. It is possible that other outcome measures (eg, ability to return to day care or school or parent ability to return to work) may be more strongly associated with performance on PRIMES composite measures than the outcomes we assessed.

Higher scores on the PRIMES overuse composites for bronchiolitis, asthma, and croup were significantly associated with shorter LOS. This finding suggests that QI interventions focused on de-implementation of unnecessary care for these conditions may be warranted. However, higher scores on underuse composites were not associated with better outcomes. Given these inconsistent findings, examining relationships between quality-measure composites and outcomes may not be the best approach to process-of-care quality-measure validation. Outcome validation at the individual quality-indicator level may be a better approach.

Dr Mangione-Smith secured funding for the study, coordinated and supervised data collection at 1 of the 5 sites, contributed to study conceptualization and design, contributed to interpretation of results, and drafted the initial manuscript and subsequent versions of the manuscript; Dr Zhou conducted the statistical analyses, assisted with interpretation of results, and critically reviewed and revised the manuscript; Drs Williams, Johnson, Kenyon, Tyler, Quinonez, and Vachani coordinated and supervised data collection at 1 of the 5 sites, contributed to interpretation of results, and critically reviewed and revised the manuscript; Ms McGalliard designed the data collection instruments, provided abstractor guidance during data collection at all 5 sites, and critically reviewed the manuscript; Drs Tieder and Simon contributed to interpretation of results and critically reviewed and revised the manuscript; Dr Wilson contributed to study conceptualization and design, contributed to interpretation of results, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by National Heart, Lung, and Blood Institute grant 1R01HL121067-01 (principal investigator: Dr Mangione-Smith). Funded by the National Institutes of Health (NIH).

     
  • CAP

    community-acquired pneumonia

  •  
  • CI

    confidence interval

  •  
  • ED

    emergency department

  •  
  • HRQOL

    health-related quality of life

  •  
  • LOS

    length of stay

  •  
  • MCID

    minimal clinically important difference

  •  
  • MR

    medical record

  •  
  • PedsQL

    Pediatric Quality of Life Inventory

  •  
  • PHIS

    Pediatric Health Information System

  •  
  • PMCA

    Pediatric Medical Complexity Algorithm

  •  
  • PRIMES

    Pediatric Respiratory Illness Measurement System

  •  
  • QI

    quality improvement

1
Donabedian
A
.
The Methods and Findings of Quality Assessment and Monitoring: An Illustrated Analysis. Vol 3. Explorations in Quality Assessment and Monitoring
.
Ann Arbor, MI
:
Health Administration Press
;
1980
2
Bacharier
LB
,
Boner
A
,
Carlsen
KH
, et al;
European Pediatric Asthma Group
.
Diagnosis and treatment of asthma in childhood: a PRACTALL consensus report.
Allergy
.
2008
;
63
(
1
):
5
34
[PubMed]
3
Melnyk
BM
,
Grossman
DC
,
Chou
R
, et al;
US Preventive Services Task Force
.
USPSTF perspective on evidence-based preventive recommendations for children.
Pediatrics
.
2012
;
130
(
2
). Available at: www.pediatrics.org/cgi/content/full/130/2/e399
[PubMed]
4
Wiener-Ogilvie
S
,
Pinnock
H
,
Huby
G
,
Sheikh
A
,
Partridge
MR
,
Gillies
J
.
Do practices comply with key recommendations of the British Asthma Guideline? If not, why not?
Prim Care Respir J
.
2007
;
16
(
6
):
369
377
[PubMed]
5
Mangione-Smith
R
,
Roth
CP
,
Britto
MT
, et al
.
Development and testing of the pediatric respiratory illness measurement system (PRIMES) quality indicators.
Hosp Pediatr
.
2017
;
7
(
3
):
125
133
[PubMed]
6
Bradley
JS
,
Byington
CL
,
Shah
SS
, et al;
Pediatric Infectious Diseases Society
;
the Infectious Diseases Society of America
.
The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America.
Clin Infect Dis
.
2011
;
53
(
7
):
e25
e76
[PubMed]
7
Johnson
D
,
Klassen
TP
,
Kellner
JD
.
Diagnosis and Management of Croup: Alberta Medical Association Clinical Practice Guidelines
.
Edmonton, Canada
:
Alberta Medical Association
;
2005
8
Ralston
SL
,
Lieberthal
AS
,
Meissner
HC
, et al;
American Academy of Pediatrics
.
Clinical practice guideline: the diagnosis, management, and prevention of bronchiolitis published correction appears in Pediatrics. 2015;136(4):782.
Pediatrics
.
2014
;
134
(
5
). Available at: www.pediatrics.org/cgi/content/full/134/5/e1474
[PubMed]
9
National Asthma Education and Prevention Program, Third Expert Panel on the Diagnosis and Management of Asthma
.
Expert Panel Report 3: Guidelines for the Diagnosis and Management of Asthma
.
Bethesda, MD
:
National Heart, Lung, and Blood Institute
;
2007
10
Varni
JW
,
Burwinkle
TM
,
Seid
M
,
Skarr
D
.
The PedsQL 4.0 as a pediatric population health measure: feasibility, reliability, and validity.
Ambul Pediatr
.
2003
;
3
(
6
):
329
341
[PubMed]
11
Varni
JW
,
Limbers
CA
,
Neighbors
K
, et al
.
The PedsQL™ Infant Scales: feasibility, internal consistency reliability, and validity in healthy and ill infants.
Qual Life Res
.
2011
;
20
(
1
):
45
55
[PubMed]
12
Desai
AD
,
Zhou
C
,
Stanford
S
,
Haaland
W
,
Varni
JW
,
Mangione-Smith
RM
.
Validity and responsiveness of the pediatric quality of life inventory (PedsQL) 4.0 generic core scales in the pediatric inpatient setting.
JAMA Pediatr
.
2014
;
168
(
12
):
1114
1121
[PubMed]
13
Children’s Hospital Association
. PHIS. 2019. Available at: https://www.childrenshospitals.org/phis. Accessed January 20, 2019
14
Simon
TD
,
Haaland
W
,
Hawley
K
,
Lambka
K
,
Mangione-Smith
R
.
Development and validation of the Pediatric Medical Complexity Algorithm (PMCA) version 3.0.
Acad Pediatr
.
2018
;
18
(
5
):
577
580
[PubMed]
15
Molenberghs
G
,
Kenward
MG
.
Missing Data in Clinical Studies
.
Hoboken, NJ
:
John Wiley & Sons, Inc
;
2007
16
Zhou
XH
,
Zhou
C
,
Liu
D
,
Sding
X
.
Applied Missing Data Analysis in the Health Sciences
.
Hoboken, NJ
:
John Wiley & Sons, Inc
;
2014
17
R Core Team
.
R: A Language and Environment for Statistical Computing
.
Vienna, Austria
:
R Foundation for Statistical Computing
;
2017
.
18
Lion
KC
,
Wright
DR
,
Spencer
S
,
Zhou
C
,
Del Beccaro
M
,
Mangione-Smith
R
.
Standardized clinical pathways for hospitalized children and outcomes.
Pediatrics
.
2016
;
137
(
4
):
e20151202
[PubMed]
19
Rotter
T
,
Kinsman
L
,
James
E
, et al
.
Clinical pathways: effects on professional practice, patient outcomes, length of stay and hospital costs.
Cochrane Database Syst Rev
.
2010
;(
3
):
CD006632
20
Akenroye
AT
,
Baskin
MN
,
Samnaliev
M
,
Stack
AM
.
Impact of a bronchiolitis guideline on ED resource use and cost: a segmented time-series analysis.
Pediatrics
.
2014
;
133
(
1
). Available at: www.pediatrics.org/cgi/content/full/133/1/e227
[PubMed]
21
Dayal
A
,
Alvarez
F
.
The effect of implementation of standardized, evidence-based order sets on efficiency and quality measures for pediatric respiratory illnesses in a community hospital.
Hosp Pediatr
.
2015
;
5
(
12
):
624
629
[PubMed]
22
Bryan
MA
,
Desai
AD
,
Wilson
L
,
Wright
DR
,
Mangione-Smith
R
.
Association of bronchiolitis clinical pathway adherence with length of stay and costs.
Pediatrics
.
2017
;
139
(
3
):
e20163432
[PubMed]
23
Upvall
MJ
,
Bourgault
AM
.
De-implementation: a concept analysis.
Nurs Forum
.
2018
;
53
(
3
):
376
382
[PubMed]
24
Prasad
V
,
Ioannidis
JP
.
Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices.
Implement Sci
.
2014
;
9
:
1
[PubMed]
25
Demiralp
B
,
He
F
,
Koenig
L
.
Further evidence on the system-wide effects of the hospital readmissions reduction program.
Health Serv Res
.
2018
;
53
(
3
):
1478
1497
[PubMed]
26
Berry
JG
,
Toomey
SL
,
Zaslavsky
AM
, et al
.
Pediatric readmission prevalence and variability across hospitals.
JAMA
.
2013
;
309
(
4
):
372
380
[PubMed]
27
Kenyon
CC
,
Melvin
PR
,
Chiang
VW
,
Elliott
MN
,
Schuster
MA
,
Berry
JG
.
Rehospitalization for childhood asthma: timing, variation, and opportunities for intervention.
J Pediatr
.
2014
;
164
(
2
):
300
305
[PubMed]
28
Gay
JC
,
Hain
PD
,
Grantham
JA
,
Saville
BR
.
Epidemiology of 15-day readmissions to a children’s hospital.
Pediatrics
.
2011
;
127
(
6
). Available at: www.pediatrics.org/cgi/content/full/127/6/e1505
[PubMed]
29
Gay
JC
,
Agrawal
R
,
Auger
KA
, et al
.
Rates and impact of potentially preventable readmissions at children’s hospitals.
J Pediatr
.
2015
;
166
(
3
):
613
619.e5
[PubMed]
30
Hain
PD
,
Gay
JC
,
Berutti
TW
,
Whitney
GM
,
Wang
W
,
Saville
BR
.
Preventability of early readmissions at a children’s hospital.
Pediatrics
.
2013
;
131
(
1
). Available at: www.pediatrics.org/cgi/content/full/131/1/e171
[PubMed]
31
Leyenaar
JK
,
Ralston
SL
,
Shieh
MS
,
Pekow
PS
,
Mangione-Smith
R
,
Lindenauer
PK
.
Epidemiology of pediatric hospitalizations at general hospitals and freestanding children’s hospitals in the United States.
J Hosp Med
.
2016
;
11
(
11
):
743
749
[PubMed]
32
Pew Research Center
. Mobile fact sheet. 2018. Available at: https://www.pewinternet.org/fact-sheet/mobile/. Accessed April 19, 2019

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data