Pediatric Respiratory Illness Measurement System (PRIMES) Scores and Outcomes

Between July 1, 2014, and June 30, 2016, we enrolled a prospective cohort of children receiving care at 1 of 5 US children’s hospitals belonging to the Pediatric Research in Inpatient Settings Network. Caregivers and their children were eligible to participate if the child was 2 weeks to 16 years old, if the family spoke English or Spanish, if the caregiver had access to a phone or the Internet, and if the child had 1 of the following diagnoses: asthma, bronchiolitis, croup, or CAP. Children with asthma, bronchiolitis, and CAP were recruited from the inpatient setting, whereas children with croup were recruited from both the ED and inpatient settings because of the low inpatient admission rates for croup.

Because the PRIMES indicators may not apply to children with underlying chronic illness, we excluded those with chronic comorbid conditions that could influence management of a PRIMES condition (eg, immunodeficiency). Because the PRIMES indicators apply to the routine management of the targeted respiratory conditions and may not apply to children who are critically ill, those directly admitted or transferred to the ICU during their hospitalization were also excluded (6%; n = 147). Additionally, children were excluded if they had a severe developmental delay because the HRQOL measure used has not been validated in this population. If a child’s primary diagnosis changed during their hospitalization (eg, from asthma to CAP), the diagnosis noted on the discharge summary was used for assignment to condition group. If the child’s discharge diagnosis was ultimately not a PRIMES condition, the child was excluded.

Monday through Friday (8 am–4 pm), trained research assistants attempted to enroll eligible caregivers and children on day 1 of admission; however, enrollment could occur any time during the first 72 hours of the admission or before discharge from the ED (croup only). The 72-hour window facilitated enrollment of those admitted on a Friday after 4 pm and still hospitalized the following Monday. Research assistants determined study eligibility by reviewing the child’s admission diagnosis and ED or inpatient history and physical using a standard study protocol. For unclear cases, the PRIMES site principal investigator confirmed eligibility. Enrolled caregivers provided consent to complete all study procedures.

The study procedures were reviewed and approved by the institutional review boards of the participating hospitals or the Western Institutional Review Board.

To assess HRQOL, caregivers who consented and children ≥8 years old who assented completed the 23-item Pediatric Quality of Life Inventory (PedsQL) 4.0 Generic Core Scales¹⁰ or the 36-item PedsQL Infant Scales.¹¹ Caregivers also completed several demographic questions. In the hospital, surveys were completed at the time of study enrollment by using laptop computers or tablets (self-administered) or by in-person or telephone interview for those uncomfortable or unable to complete self-administered surveys. The survey was offered in both English and Spanish. At 2 to 6 weeks after discharge, the PedsQL was administered again; caregivers could complete their follow-up survey either online (sent as a Web link in an e-mail) or by telephone interview. This time interval allowed for multiple call attempts to complete follow-up assessments not completed online by 3 weeks postdischarge.

For each hospital, 2 research team members were trained on the clinical content and use of the PRIMES abstraction tool by the developer. Team members completed test abstractions for each target condition that were compared with gold standard abstractions created by the developer. Abstractors were considered fully trained when they could reliably abstract the test records for each condition with κ statistics of ≥0.75 for both indicator eligibility and scoring. Each abstractor was assigned half of the cases for each condition at their hospital. Eligibility for each sampled case was confirmed at the beginning of abstraction.

Because all enrolled children with asthma, bronchiolitis, and CAP were recruited from the inpatient setting, PRIMES indicators assessing ED discharge care were excluded (n = 10 indicators).⁵ All included PRIMES indicators could be classified as either overuse indicators that specify care processes that should not occur (eg, obtaining a chest radiograph in children with bronchiolitis) or underuse indicators that specify care processes that should occur (eg, offering an influenza vaccine for children with asthma who are not immunized). Three composite scores were constructed for each condition: a composite that included all overuse indicators, a composite that included all underuse quality indicators, and an overall composite that included all indicators (Supplemental Tables 4 through 11).

PRIMES condition-specific composite scores were calculated as continuous variables (0–100 scale) by dividing the number of times indicated care was received by the number of eligible cases, with higher scores indicating better quality of care.

Level of improvement in physical functioning after hospital discharge was assessed by using the PedsQL physical functioning subscale. We only used the physical functioning subscale because we hypothesized that higher adherence to the PRIMES standards of care would most likely influence physical (rather than psychosocial) functioning, and previous research has revealed the physical component to be most responsive to post-hospital recovery.¹² For analyses, caregiver proxy report was used for all children aged <8 years. For older children, self-report was used when available and, if missing, parent proxy reports were used. On the basis of previous research, 4.5 was considered the minimal clinically important difference (MCID) on the 0 to 100 scale.¹⁰ 

Each participating hospital contributes to the Pediatric Health Information System (PHIS) database.¹³ The PHIS was used to ascertain LOS, ICU admission during the study hospitalization, and 30-day same-cause ED and/or inpatient reuse. PHIS data were also used to determine insurance type (public versus commercial) and level of medical complexity for each child by applying the Pediatric Medical Complexity Algorithm (PMCA) version 3.0.¹⁴ PHIS data for enrolled children were de-identified and linked to their study identification numbers. Study identification numbers were used to link PHIS, MR abstraction, and survey data for each patient.

All analyses were performed at the patient level. Univariate statistics included medians, proportions, means, and SDs for all continuous predictor and outcome variables.

For all multivariable-adjusted analyses, we included the following covariates: child age, child sex, child race and/or ethnicity, PMCA category, insurance type, caregiver education, secondhand smoke exposure (≥1 smoker at home), days to follow-up survey (14–42 days postdischarge), and study site. A site-specific random effect was also included to account for clustering within study site.

We applied linear mixed-effects regression models to examine the associations between performance on the PRIMES condition-specific composite scores and the magnitude of change in the PedsQL scores. We regressed the PedsQL score at follow-up on PRIMES condition-specific composite scores (overall, overuse, and underuse) while controlling for admission PedsQL score. To assist with interpretation, we assessed the change in PedsQL scores associated with a 10-point increase in PRIMES composite scores.

We modeled inpatient LOS in hours as a continuous outcome and applied linear mixed-effects regression models. The primary predictors in these models were the PRIMES condition-specific composite scores. We assessed the magnitude of change in LOS in hours associated with a 10-point increase in PRIMES composite scores. For croup, children discharged from the ED were excluded.

Thirty-day same-cause ED and/or inpatient reuse at the original site of care was treated as a binary outcome (≥1 vs 0 ED return visits and/or inpatient readmissions) and analyzed by using mixed-effects logistic regression models. PRIMES condition-specific composite scores were the main predictors. We assessed the odds of a reuse event associated with a 10-point increase in PRIMES composite scores. Coefficient estimates for 2 covariates, PMCA category, and caregiver education would not converge because of the small number of reuse events and were dropped from models to obtain valid estimates.

We performed sensitivity analyses to examine the relationships between PRIMES condition-specific composite scores and PedsQL follow-up scores that included imputed physical functioning values for children with missing PedsQL follow-up surveys. Ten imputed data sets were generated by using multivariate imputation by chained equation.^15,16 Mixed-effects models were fitted to each imputed data set. The estimates were then pooled to generate the final results.

All analyses were conducted by using R version 3.4.2.¹⁷ 

We enrolled 2334 children and their caregivers across the 5 participating hospitals (sample size range n = 284–547 participants per hospital; response rate = 2334 of 3428 [68%]). The sample was 40% non-Hispanic white, 24% Hispanic, and 22% African American. The mean child age was 3.6 years (SD 3.8). Only 4% of the sample was classified as having a complex chronic illness by PMCA. These cases were chart reviewed by site principal investigators and deemed as conditions that would not influence respiratory illness management (Table 1).

Unadjusted analyses used to examine PRIMES condition-specific composite scores revealed that average performance on the overall composites was highest for CAP (mean 88.7 [SD 12.8]) and lowest for asthma (mean 79.8 [SD 8.8]; Table 2). Scores were significantly higher on the overuse composites for asthma, bronchiolitis, and CAP compared with the corresponding underuse composite scores (Table 2). In contrast, the underuse composite score for croup (mean 89.9 [SD 10.5]) was significantly higher than the overuse composite score (mean 53.8 [SD 40.7]; P < .001).

Follow-up surveys were completed by 1686 enrolled families 2 to 6 weeks after discharge (follow-up response rate = 72%). The mean PedsQL physical functioning subscale score at admission was 56 (SD ±23; Table 1) and the mean postdischarge follow-up score was 91 (SD ±13), resulting in an average change score of 36 (SD ±23). Adjusted analyses revealed no association between PRIMES condition-specific composite scores and change in physical functioning at follow-up, with the exception of the croup underuse composite (Table 3). Contrary to our hypothesis, for every 10-point increase in the croup underuse composite score, the PedsQL physical functioning score decreased by 1.51 points (95% confidence interval [CI] −2.96 to −0.07; P < .05). The sensitivity analyses, which included imputed physical functioning follow-up scores, revealed similar results for all composite scores; however, the association between the croup underuse composite score and decreased improvement in PedsQL scores at follow-up was no longer significant (data not shown).

The PRIMES condition-specific overuse composite scores for bronchiolitis, asthma, and croup were all associated with significantly decreased LOS. For every 10-point increase in the PRIMES overuse composite scores (0–100 scale), LOS decreased by 8.8 hours (95% CI −11.6 to −6.1) for children with bronchiolitis, by 3.1 hours (95% CI −5.5 to −1.0) for children with asthma, and by 2.2 hours (95% CI −3.9 to −0.1) for patients with croup (Table 3).

Adjusted analyses did not reveal significant associations between PRIMES condition-specific composite scores and same-cause reuse (Table 3).

In this outcome validation study, higher scores on PRIMES bronchiolitis, asthma, and croup overuse composites were associated with significantly shorter inpatient LOS; however, this was not the case for the corresponding underuse composite scores. Higher PRIMES composite scores were not significantly associated with increased improvement in physical functioning at follow-up or decreased risk of reuse events. These findings indicate the potential importance of conducting outcome validation studies for individual process-of-care quality indicators.

Previous studies have demonstrated an association between implementing standardized clinical pathways and decreased LOS,^18,–22 including 1 pediatric study that was focused on asthma, bronchiolitis, and CAP²¹ and 2 others that were focused on bronchiolitis.^20,22 These studies, however, did not distinguish between adherence to overuse versus underuse pathway standards of care and LOS. There are 2 conflicting interpretations of our study findings that may also apply to these previous investigations. One interpretation would be that de-implementation of unnecessary tests or treatments (ie, adhering to PRIMES overuse quality indicators), may reduce the risk of having to follow-up on false-positive tests (eg, positive results on blood cultures that are likely due to contaminants rather than true pathogens) or managing adverse reactions to unnecessary medications (eg, allergic reactions or diarrhea caused by treatment with unwarranted antibiotics).^23,24 Avoiding these types of complications may result in shorter LOS and thus explain the observed associations. An alternative interpretation would be that children with more severe illness have longer LOS and are more likely to receive “off-guideline” tests and treatments. Some of these additional tests and treatments may go against the standards of care laid out by the PRIMES overuse indicators (eg, obtaining a chest radiograph or starting antibiotics in children hospitalized with bronchiolitis). Although we attempted to control for severity of illness in the current study by excluding children with ICU admissions and by adjusting for PMCA category, this likely did not adequately control for the wide range of severity observed in children with these respiratory illnesses cared for outside the ICU setting.

PedsQL physical functioning scores improved significantly between admission and follow-up for most children, with an average improvement of 36 points on the 0 to 100 scale. However, this improvement was not significantly associated with performance on PRIMES composite measures during hospitalization (Table 3). The degree of improvement in physical functioning for children with no chronic illness or noncomplex chronic illness after an acute hospitalization is not unexpected.¹² The lack of association between higher PRIMES composite scores and greater improvement in physical functioning at follow-up suggests that this outcome may not be particularly useful when trying to outcome validate process-of-care quality indicators in populations of children who are generally healthy at baseline.

Higher adherence to the PRIMES condition-specific composites was also not associated with 30-day same-cause reuse. Reuse is a commonly used outcome measure in adults and is increasingly used in pediatric quality-of-care studies.^25,–27 However, in this large multicenter study, only 5% (n = 107) of participating children had a same-cause reuse event. Although this outcome measure may be useful in populations of children with medical complexity,²⁸ its utility for other children hospitalized with acute respiratory illnesses may be limited.^28,–30 

Our primarily negative findings suggest that an alternative approach to outcome validation for both the PRIMES quality indicators and pediatric quality–measure composites in general may be needed. The PRIMES composites, with the exception of the croup overuse composite, include multiple quality indicators. Although our goal for this study was to explore the association between composite scores and outcomes, it is possible that at the individual quality-indicator level, we may have been able to observe significant associations between some of these process measures and the outcomes assessed. By combining multiple quality indicators into a composite measure, the potential association between one indicator and the outcomes examined may have been diminished by a lack of association observed for other indicators included in that composite. Assessing the associations between the individual PRIMES quality indicators and outcomes might also be more useful in terms of informing quality improvement (QI) interventions at the hospital level by providing guidance regarding selection of care processes to focus on for QI purposes.

This study has several limitations. Although the study was conducted in 5 geographically dispersed US children’s hospitals, our results may not be generalizable to children cared for in regions not represented by these hospitals or to children cared for in community hospitals, where most pediatric care is provided.³¹ Families without phone or Internet access were excluded, which may have further reduced the generalizability of the sample. However, national rates of smartphone ownership during the time frame of this study were 90% to 91% and desktop and/or laptop ownership was 70% to 74%.³² We were unable to adequately control for severity of illness, which may have confounded the relationship we observed between PRIMES overuse composite scores and LOS. For PedsQL assessments, increased time in days between discharge and follow-up survey completion was 1 of the strongest predictors of greater improvement in PedsQL scores after hospitalization for children with bronchiolitis, croup, or CAP (data not shown). Allowing >2 weeks for follow-up survey completion may have limited our ability to observe associations between performance on PRIMES composites and levels of improvement in physical functioning. We may have been under-powered to detect associations between performance on PRIMES composites and reuse outcomes given the rarity of these events. It is possible that other outcome measures (eg, ability to return to day care or school or parent ability to return to work) may be more strongly associated with performance on PRIMES composite measures than the outcomes we assessed.

Higher scores on the PRIMES overuse composites for bronchiolitis, asthma, and croup were significantly associated with shorter LOS. This finding suggests that QI interventions focused on de-implementation of unnecessary care for these conditions may be warranted. However, higher scores on underuse composites were not associated with better outcomes. Given these inconsistent findings, examining relationships between quality-measure composites and outcomes may not be the best approach to process-of-care quality-measure validation. Outcome validation at the individual quality-indicator level may be a better approach.

CAP

community-acquired pneumonia

CI

confidence interval

ED

emergency department

HRQOL

health-related quality of life

LOS

length of stay

MCID

minimal clinically important difference

MR

medical record

PedsQL

Pediatric Quality of Life Inventory

PHIS

Pediatric Health Information System

PMCA

Pediatric Medical Complexity Algorithm

PRIMES

Pediatric Respiratory Illness Measurement System

QI

quality improvement