Video Abstract

Video Abstract

BACKGROUND:

Quality improvement (QI) efforts can improve guideline-recommended asthma care processes in the pediatric office setting. We sought to assess whether practice participation in an asthma QI collaborative was associated with decreased asthma-related emergency department (ED) visits.

METHODS:

A statewide network of practices participated in a pediatric asthma QI collaborative from 2015 to 2016. We evaluated asthma-related ED visit rates per 100 child-years for children ages 3 to 21 years with asthma, using the state’s all-payer claims database. We used a difference-in-differences approach, with mixed-effects negative binomial regression models to control for practice and patient covariates. Our main analysis measured the outcome before (2014) and after (2017) the QI collaborative at fully participating and control practices. Additional analyses assessed (1) associations during the intervention period (2016) and (2) associations including practices partially participating in QI collaborative activities.

RESULTS:

In the postintervention year (2017), participating practices’ (n = 20) asthma-related ED visit rate decreased by 5.8 per 100 child-years, compared to an increase of 1.8 per 100 child-years for control practices (n = 15; difference in differences = −7.3; P = .002). Within the intervention year (2016), we found no statistically significant differences in asthma-related ED visit rates compared to controls (difference in differences = −4.3; P = .17). The analysis including partially participating practices yielded similar results and inferences to our main analysis.

CONCLUSIONS:

Participation in an asthma-focused QI collaborative was associated with decreased asthma-related ED visit rates. For those considering implementing this type of QI collaborative, our findings indicate that it takes time to see measurable improvements in ED visit rates. Further study is warranted regarding QI elements contributing to success for partial participants.

What’s Known on This Subject:

Participation in an asthma quality improvement learning collaborative in primary care is associated with improved processes of care in the office setting for children with asthma.

What This Study Adds:

Participation in an asthma quality improvement learning collaborative in primary care is associated with a substantial decrease in asthma-related emergency department visits >1 year after the end of the collaborative.

Asthma leads to substantial health care use for children, accounting for >10 million office visits and 1.6 million visits to emergency departments (EDs) annually.1  Between 2001 and 2010, the rate of asthma-related ED visits remained high and unchanged.2  In 2010, ED visits for asthma cost Medicaid >$272 million.3  National data indicated that children accounted for an increasing percentage of all asthma-related ED visits (from 36% in 2011 to 40% in 2016).4 

Improving care in pediatric primary care may help improve asthma management and decrease exacerbations, thereby decreasing ED visits. The National Asthma Education and Prevention Program, sponsored by the National Heart, Lung, and Blood Institute (NHLBI), includes evidence-based recommendations to improve practitioner and patient management of asthma.5  Unfortunately, significant variability in primary care adherence to NHLBI guidelines persists.68 

Quality improvement (QI) in primary care provides mechanisms to address variation in guideline adherence and therefore potentially decrease ED use. Studies of asthma-focused QI in primary care have been focused on process measures like adherence to NHLBI guidelines: tracking asthma scores by using a validated tool,9  increasing use of asthma action plans,9,10  increasing prescription of inhaled corticosteroids,11,12  and increasing use of spirometry.13  Few studies of primary care, asthma-focused QI in which researchers examine ED use exist, and the results of these studies are mixed. In 1 study, researchers reported providing education to primary care practices and implementing clinical decision-making tools led to a decrease in ED visits and hospitalizations for pediatric asthma patients.6  This study was limited to 6 practices in 1 city, did not use a learning collaborative approach, and did not have a control group. In a more recent multistate study, researchers using a modified QI collaborative model reported improvements to clinical asthma management measures by primary care practitioners but no significant change in ED or urgent care use.14  This study did not report statistical tests on ED run chart data and did not have a control group.

The Vermont Child Health Improvement Program (VCHIP), the longest-running improvement partnership nationally,15  leads statewide QI collaboratives based on the Model for Improvement16  for pediatric-serving primary care practices.17,18  A recent VCHIP QI collaborative, focused on asthma care and management in primary care, revealed significant improvement in multiple NHLBI guideline-recommended processes (eg, increased controller medication prescription);19  however, it was not investigated whether this also led to reduced asthma-related ED use. This presents an opportunity for a longer follow-up study of a primary care asthma-focused QI collaborative examining asthma-related ED use as the outcome. Our overall objective is to test whether participation in an asthma QI collaborative is associated with a decrease in asthma-related ED use over time, compared to controls.

In 2012, VCHIP created Child Health Advances Measured in Practice (CHAMP), a voluntary network of pediatric and family medicine primary care practices working collaboratively on QI. Practices in the CHAMP network care for ∼60% of children in Vermont. Each year, in collaboration with these practices and state partners, VCHIP selects a focus area for the CHAMP QI learning collaborative. After successful QI collaboratives on improving immunization rates17  and addressing adolescent depression,18  from mid-September 2015 through mid-May 2016, CHAMP practices worked collaboratively to improve asthma management.19  During the CHAMP asthma learning collaborative, VCHIP’s approach followed the Model for Improvement,16  a QI methodology designed to advance patient care and outcomes through setting an aim, measuring processes, and identifying areas of improvement through iterative Plan-Do-Study-Act cycles.20,21  Full details of the asthma QI collaborative, clinical measures, and office systems strategies are published,19  and a brief summary of key elements is included below in the QI collaborative section.

Administrative claims spanning 2014–2017 from Vermont’s all-payer claims database (Vermont Health Care Uniform Reporting and Evaluation System) served as our primary source of data. Medicaid, Medicare, and most commercial insurers in Vermont submitted medical and pharmacy claims to this database for this period. Vermont’s all-payer claims database excluded the uninsured and included an estimated 90% of state residents receiving health care in 2014 and 2015. This percentage decreased to ∼75% in 2016 and 2017 because of a Supreme Court ruling,22  which resulted in some commercial self-pay insurance plans opting out of submitting data. VCHIP collected data on practice characteristics (Table 1) from state administrative data and national registries and compiled these data annually as part of VCHIP’s attribution process.23 

TABLE 1

Characteristics of the Participating and Control Practices

Participating Practices (n = 20), n (%)Control Practices (n = 15), n (%)P
Practice size   .22 
 <500 patients 2 (10) 3 (20)  
 500–1499 patients 8 (40) 9 (60)  
 >1500 patients 10 (50) 3 (20)  
Practice specialty   .87 
 Pediatrics 11 (55) 7 (47)  
 Family medicine 8 (40) 7 (47)  
 Mixed practitioners 1 (5) 1 (7)  
Organization structure   .04 
 Hospital owned 12 (60) 3 (20)  
 Independently owned 8 (40) 12 (80)  
HSA   .49 
 One metropolitan HSA 8 (40) 4 (27)  
 All other HSAs 12 (60) 11 (73)  
FQHC or RHC   .27 
 FQHC or RHC 4 (20) 7 (47)  
 Neither FQHC nor RHC 16 (80) 8 (53)  
Participating Practices (n = 20), n (%)Control Practices (n = 15), n (%)P
Practice size   .22 
 <500 patients 2 (10) 3 (20)  
 500–1499 patients 8 (40) 9 (60)  
 >1500 patients 10 (50) 3 (20)  
Practice specialty   .87 
 Pediatrics 11 (55) 7 (47)  
 Family medicine 8 (40) 7 (47)  
 Mixed practitioners 1 (5) 1 (7)  
Organization structure   .04 
 Hospital owned 12 (60) 3 (20)  
 Independently owned 8 (40) 12 (80)  
HSA   .49 
 One metropolitan HSA 8 (40) 4 (27)  
 All other HSAs 12 (60) 11 (73)  
FQHC or RHC   .27 
 FQHC or RHC 4 (20) 7 (47)  
 Neither FQHC nor RHC 16 (80) 8 (53)  

Mixed practitioners include pediatricians and family medicine physicians. Unadjusted analyses were done by using Fisher’s exact tests at the practice level.

In brief, practitioners and practice staff in the CHAMP network interested in participating in the QI collaborative attended an all-day in-person learning session in mid-September 2015 covering topics on office systems strategies (eg, asthma assessment, control, and management and patient education). Practices received evidence-based resources, such as NHLBI guidelines and validated tools. Those that decided to participate in the QI collaborative subsequently attended monthly phone calls, including didactics on asthma topics (eg, the use of validated asthma control tests, implementing and documenting asthma action plans, and coding and billing) and knowledge sharing on successes and challenges. During the collaborative, practitioners worked to set practice goals, implement changes, and measure improvements with their practice teams. Participating practitioners submitted monthly convenience samples of patients to monitor progress on clinical asthma measures and received monthly feedback reports with summaries of their performance on the clinical measures, overall collaborative performance, and QI coaching notes. The main findings from this study revealed that practitioners exceeded their 20% improvement goals, aligning with NHLBI guidelines for the assessment of asthma with a validated tool, planned asthma visits, assessment of tobacco exposure, provision of asthma education, and instruction on asthma device use.19 

The VCHIP approach to QI is inclusive, encouraging primary care practices interested in the annual QI topic to send practitioners and staff to the all-day learning session, without requiring a commitment to participate in the subsequent QI collaborative. This inclusive approach led to the definition of 3 categories of practices for this study on the basis of level of participation: full participants, partial participants, and nonparticipants. Fully participating practices met 3 criteria: (1) attended the in-person learning session, (2) submitted medical record review data for 6 of 7 months, and (3) attended at least 1 all-practice conference call. Of 46 primary care practices in the CHAMP network in 2015, 20 fully participated in the asthma QI collaborative, including 2 practices with practitioners working across both sites and, later, merging. We included 15 nonparticipating CHAMP network practices as controls, and we designated the remaining 11 practices as partial participants because they attended the in-person learning session but did not attend monthly phone calls and submitted <6 months of QI data. For the partial participants, we did not collect information on the extent to which they did or did not implement asthma QI at their practices. Therefore, we excluded partial participants from our main analyses but included them in additional analyses described below.

Patients ages 3 to 21 years, who met criteria for identifiable asthma (Supplemental Table 5)24,25  and who were attributed to primary care practices (see below), were included. In 2014, the data set included 9883 patients 3 to 21 years old with identifiable asthma, of whom we attributed 4589 patients to CHAMP network practices: 2376 at participating, 1282 at control, and 931 at partially participating practices. In 2017, the data set included 8677 patients 3 to 21 years old with identifiable asthma, of whom we attributed 4186 patients to CHAMP network practices: 2257 at participating, 899 at control, and 1030 at partially participating practices.

Attributing each child to a single practice each year allowed us to measure improvement at the practice level over time. There were 2 steps in our attribution of children to primary care practice, by using a hierarchical approach based on previous work by Christensen et al.26  First, we selected a single primary care practitioner for each child out of all practitioners listed on claims within a year. Second, we linked the child to a practice on the basis of the chosen primary care practitioner from step 1. We describe the full details of our attribution of children to practices using Vermont’s all-payer claims database in a previous publication.23 

The outcome measure was the ED visit rate (the number of visits per 100 child-years) for children ages 3 to 21 years with identifiable asthma.24,25  The measure definition is from the Pediatric Quality Measures Program (PQMP), created under the Children’s Health Insurance Program Reauthorization Act, funded through the Centers for Medicare and Medicaid Services, and overseen by the Agency for Healthcare Research and Quality. The PQMP was established to address gaps in assessing quality in pediatric care and led to the development of numerous pediatric quality measures,27  including the one used in this study.

The denominator and numerator definitions followed the asthma-related ED visit rate measure specifications developed28  and refined through the PQMP.24,25,27,29  Eligibility was assessed for each month of data. The criteria for the measure were (1) having ≥3 months of consecutive enrollment in the same insurance plan (the measurement month and the 2 months before) and (2) evidence of claims for identifiable asthma (see Supplemental Table 5 for definition) during a look-back period, including the measurement month, all previous months in the measurement year, and the year before the measurement year (see Supplemental Fig 2). The total number of eligible child-months in each year was summed and divided by 1200 to calculate the denominator in units of 100 child-years for the measurement year. The numerator was a count of all ED visits or hospitalizations with a first or second diagnosis of asthma in eligible patients for the measurement month. The inclusion of hospitalizations is in response to measure development work,30  which revealed that the measure was more accurate when including hospitalizations because claims are often not submitted for ED care that leads to hospitalization. The inclusion of visits with an asthma diagnosis in the second position reflects measure development Delphi panel recommendations and exploratory analyses that found that claims with a second diagnosis of asthma often had a primary diagnosis with a related symptom (eg, fever or wheezing) or a known asthma trigger (eg, upper respiratory tract infection, pneumonia, or influenza).

For comparing participating and control practices and for inclusion in the difference-in-differences analyses, we used the following practice characteristics (Table 1): size (<500, 500–1499, or >1500 patients), specialty (pediatrics, family medicine, or mixed, including pediatricians and family medicine physicians), organization structure (hospital owned versus not), hospital service area (HSA) location (located in the largest metropolitan HSA versus not), and federally qualified health center (FQHC) or rural health center (RHC) versus neither. We also included covariates for patient characteristics: age categories (3–5, 6–11, 12–17, or 18–21 years), sex (male, female, or unknown), and insurance (Medicaid versus not), for comparing participating and control practices (Table 2).

TABLE 2

Characteristics of Patients at Participating and Control Practices Before (2014) and After (2017) the Asthma QI Learning Collaborative

20142017
ParticipatingControlPParticipatingControlP
n (%)n (%)n (%)n (%)
Total study sample 2376 (100) 1282 (100) — 2257 (100) 899 (100) — 
Age categories, y   .047   <.001 
 3–5 453 (19) 206 (16)  430 (19) 160 (18)  
 6–11 935 (39) 498 (39)  868 (38) 326 (36)  
 12–17 839 (35) 478 (37)  854 (38) 327 (36)  
 18–21 149 (6) 100 (8)  105 (5) 86 (10)  
Sex   .26   .32 
 Male 1193 (50) 644 (50)  1125 (50) 442 (49)  
 Female 1046 (44) 580 (45)  1009 (45) 419 (47)  
 Unknown 137 (6) 58 (5)  123 (5) 38 (4)  
Insurance   .001   <.001 
 Non-Medicaid 897 (38) 412 (32)  627 (28) 183 (20)  
 Medicaid 1479 (62) 870 (68)  1630 (72) 716 (80)  
20142017
ParticipatingControlPParticipatingControlP
n (%)n (%)n (%)n (%)
Total study sample 2376 (100) 1282 (100) — 2257 (100) 899 (100) — 
Age categories, y   .047   <.001 
 3–5 453 (19) 206 (16)  430 (19) 160 (18)  
 6–11 935 (39) 498 (39)  868 (38) 326 (36)  
 12–17 839 (35) 478 (37)  854 (38) 327 (36)  
 18–21 149 (6) 100 (8)  105 (5) 86 (10)  
Sex   .26   .32 
 Male 1193 (50) 644 (50)  1125 (50) 442 (49)  
 Female 1046 (44) 580 (45)  1009 (45) 419 (47)  
 Unknown 137 (6) 58 (5)  123 (5) 38 (4)  
Insurance   .001   <.001 
 Non-Medicaid 897 (38) 412 (32)  627 (28) 183 (20)  
 Medicaid 1479 (62) 870 (68)  1630 (72) 716 (80)  

P values are from bivariate χ2 comparing patient-level percentages within demographic categories across participant and control practices within each year. —, not applicable.

We used Fisher’s exact and χ2 analyses, respectively, to compare practice and patient characteristics between participating and control practices.

To assess the relationship between asthma QI collaborative participation and asthma-related ED use, we used a difference-in-differences approach. This approach compared the asthma-related ED visit rate per 100 child-years before (2014) and after (2017) the QI collaborative, at participating versus control practices. The choice of 2017 reflects the assumption that improvements will continue past the end of the QI collaborative. We used a mixed-effects negative binomial multivariable regression model, with random intercepts for patient and practice to account for clustering within patient and practice. To estimate the difference-in-differences effect, models included variables for participation, year, and the interaction term between participation and year. The P value for the interaction term was used to test whether the change in ED use was different between participating and control practices. Practice (Table 1) and patient (Table 2) covariates were included in models as potential confounders. Adjusted ED rates at each time point were obtained by using the postestimation margins command in Stata version 15 (Stata Corp, College Station, TX).31  This is used to calculate a marginal (or average) rate by averaging predicted values derived for each observation, assuming it was in each of the intervention groups at each time point (but by using its covariate values for all the other variables to generate the predicted values). We tested for a difference in participating versus nonparticipating preintervention slopes from monthly ED visit rate data (January 2014 to September 2015) to ensure there were no preintervention trend differences. We ran the same model over time in months, including an interaction between participation and time, and we found no difference (P = .684; results not shown), thus supporting the notion that trends in the control practices serve as a reasonable counterfactual for what would have occurred to the participating practices in the absence of the QI learning collaborative.

We conducted two additional analyses to test the robustness of our findings and better understand the potential effects of the QI collaborative. We hypothesized that the effects of the asthma improvements may take some time to translate into changes in the distal use outcome. Thus, we repeated our analysis using an earlier time point (2016) for the postintervention period to examine whether immediate effects were apparent, although the QI collaborative was still active during the beginning of 2016. In addition, we hypothesized that the effect of the QI may be different in partially participating practices, so we repeated our analysis, including children with asthma from those partial participating practices.

All analyses were conducted in Stata version 15.1.32  The study was approved by the institutional review board (Committee on Human Research in the Medical Sciences 17-0232).

The 20 participating practices did not have significantly different characteristics compared to the 15 control practices, except there were a larger number owned by hospitals (Table 1). Participating practices had slightly younger children and fewer Medicaid patients in both 2014 and 2017 (Table 2), compared to control practices.

In our main analysis, the asthma-related ED visit rate per 100 child-years for participating practices decreased from 15.3 in 2014 to 9.5 in 2017, for an overall decrease of −5.8 (nearly 40%). Over the same period, the asthma-related ED visit rate per 100 child-years for control practices increased from 17.0 to 18.8, for an overall increase of +1.8. These findings led to a statistically significant adjusted difference in differences of −7.6 visits per 100 child-years associated with participation in the collaborative (Table 3; P = .002). Figure 1 reveals the ED visit rates over time, comparing participants to controls in 2014 and 2017. The dashed lines between our comparison years are data excluded from our analyses, revealing variation in the ED visit rates during that time. For point estimates from the full negative binomial model, including confounders, see Supplemental Table 6.

TABLE 3

Comparison of Participating and Control Practice Mean Asthma-Related ED Visit Rates From Baseline and Postcollaborative Years (Main Difference-in-Differences Analysis)

Main AnalysisAsthma-Related ED Visits per 100 Child-YearsP95% Confidence Interval
LowerUpper
Control 2014 16.97 — 11.99 21.95 
Participating 2014 15.29 — 11.62 18.96 
Control 2017 18.80 — 12.89 24.71 
Participating 2017 9.50 — 7.07 11.93 
Difference in differences −7.62 .002 −13.45 −1.78 
Main AnalysisAsthma-Related ED Visits per 100 Child-YearsP95% Confidence Interval
LowerUpper
Control 2014 16.97 — 11.99 21.95 
Participating 2014 15.29 — 11.62 18.96 
Control 2017 18.80 — 12.89 24.71 
Participating 2017 9.50 — 7.07 11.93 
Difference in differences −7.62 .002 −13.45 −1.78 

Marginal rates are adjusted for patient (age, sex, and insurance) and practice (specialty, organizational structure, size, geographic region, and FQHC or RHC) characteristics and, also, for patients clustering within practices; SEs are estimated by using the δ method; Coefficients for relative rates are in Supplemental Table 6. —, not applicable.

FIGURE 1

Quarterly number of ED visits per 100 child-years for participants versus controls from 2014 to 2017, highlighting the QI collaborative (mid-September 2015 to mid-May 2016). Solid lines indicate the data analyzed and presented in the main analysis comparing the calendar year 2017 vs 2014.

FIGURE 1

Quarterly number of ED visits per 100 child-years for participants versus controls from 2014 to 2017, highlighting the QI collaborative (mid-September 2015 to mid-May 2016). Solid lines indicate the data analyzed and presented in the main analysis comparing the calendar year 2017 vs 2014.

Results from additional analyses comparing the change in ED visit rate (1) from 2014 to 2016 and (2) including partial participants are in Table 4. Comparing the slight decrease of −0.8 ED visits per 100 child-years for participating practices to the increase of +3.5 ED visits per 100 child-years for control practices, the overall adjusted difference in differences of −4.3 visits per 100 child-years was not statistically significant (P = .17). The analysis, including the partial with fully participating practices, yielded similar results and inferences as our main analysis, with an adjusted difference in differences of −7.3 visits per 100 child-years (P = .003).

TABLE 4

Comparison of Participating and Control Practices Change in Mean ED Asthma Visit Rate per 100 Child-Years (Additional Difference-in-Differences Analyses) for (1) Comparison of Baseline (2014) to 2016 (Includes the Last 3 Months of the Learning Collaborative) and (2) Including Partial With Full Participants Compared to Control Practices

Additional AnalysesAsthma-Related ED Visits per 100 Child-YearsP95% Confidence Interval
LowerUpper
2014–2016     
 Control 2014 17.93 — 13.83 22.04 
 Participating 2014 14.11 — 11.57 16.65 
 Control 2016 21.46 — 15.94 26.98 
 Participating 2016 13.33 — 10.96 15.70 
Difference in differences −4.31 .17 −10.49 1.88 
Including partial participants (2014–2017)     
 Control 2014 16.43 — 11.01 21.86 
 Participating and partial 2014 15.90 — 12.07 19.72 
 Control 2017 18.15 — 11.79 24.52 
 Participating and partial 2017 10.34 — 7.84 12.84 
Difference in differences −7.27 .003 −12.97 −1.58 
Additional AnalysesAsthma-Related ED Visits per 100 Child-YearsP95% Confidence Interval
LowerUpper
2014–2016     
 Control 2014 17.93 — 13.83 22.04 
 Participating 2014 14.11 — 11.57 16.65 
 Control 2016 21.46 — 15.94 26.98 
 Participating 2016 13.33 — 10.96 15.70 
Difference in differences −4.31 .17 −10.49 1.88 
Including partial participants (2014–2017)     
 Control 2014 16.43 — 11.01 21.86 
 Participating and partial 2014 15.90 — 12.07 19.72 
 Control 2017 18.15 — 11.79 24.52 
 Participating and partial 2017 10.34 — 7.84 12.84 
Difference in differences −7.27 .003 −12.97 −1.58 

Marginal rates are adjusted for patient (age, sex, and insurance) and practice (specialty, organizational structure, size, geographic region, and FQHC or RHC status). Because all participants are included in the marginal effect calculation, the control estimates for the same year across models are not the same (eg, control 2014 estimates across models). Analyses are also adjusted for patients clustering within practices; SEs are estimated by using the δ method. —, not applicable.

Our study of a statewide asthma QI collaborative was focused on a use measure, asthma-related ED visit rate, building on our previous findings that the QI collaborative led to improvements in care processes.19  We found that participation in the QI collaborative was associated with a substantial decrease of nearly 40% in asthma-related ED visit rates >1 year after the end of the collaborative. The large number of nonparticipating practices as a control group strengthens the rigor and validity of our findings by using a difference-in-differences approach. Our findings indicate that successful QI collaboratives, with documented improvements in NHLBI guideline adherent process improvements, are associated with a meaningful decrease in ED visits compared to controls. This is a potential benefit not only to public and commercial payers but also to children and parents, by avoiding missed school and work and, potentially, achieving an improved quality of life.33 

We found that there was a time lag to the association between participation in the QI collaborative and improved asthma-related ED rates. Although there was an improvement by 2016, it was not statistically significant. The improvement by 2017 was nearly twice that by 2016 and represented close to a 40% difference from the control group rate in 2014. Although the confidence intervals for these effect estimates overlap, there are several possible explanations for this lag worth mentioning. Over the course of a collaborative, process measures, like asthma control, take time to improve, and seeing improvement on downstream clinical outcomes, like ED visits, may take additional time. Also, asthma exacerbations are seasonal, with ED visits occurring often in winter months. The collaborative ended early in 2016, so calendar year data from 2016 included 3 winter months when the QI collaborative was still underway, capturing the higher rate of asthma ED visits, before participants had finished improvements. This may explain why the effect of the improvements was not apparent until the next year. Finally, in previous asthma clinical trials revealing that primary care interventions can deliver changes in ED outcomes in a shorter 1-year time frame,34  researchers specifically recruited patients at “high risk” for asthma exacerbations (eg, enrolled only patients with a recent asthma-related ED visit). This focus on higher risk patients can lead to more rapid changes in ED use. In our study, we included all children with asthma (and did not use severity as eligibility criteria), which may also explain the longer observation period before detecting changes. This additional analysis may be useful to payers (or primary care practices participating in value-based accountable care organizations), to forecast potential decreased ED use through a primary care asthma QI collaborative.

We found that including patients from partially participating practices did not change our main results or inferences. One potential implication for those implementing a QI collaborative is that partial participation can still be associated with improvements in ED use. We would add the caveat that the partially participating practices in this study previously engaged in VCHIP QI collaboratives and, therefore, were not naïve to QI implementation. Full participation may be more important in a less experienced group of practices. In a future study, researchers using an implementation science approach (tracking types and extent of participation, previous QI experience, and asthma-related process and outcome measures) may more accurately pinpoint which components of the collaborative implemented in which practice settings most directly affect improvement in asthma health outcomes.

There were several limitations to our current study. Although we demonstrate difference in differences in ED visit rates from 2014 to 2017, during the intervening years, there is variation in the ED visit rates between the participating and control practices. It is possible that unmeasured confounders, such as other concurrent QI or public health efforts that differed across practices, or other unmeasured differences between participating and control practices, such as practice culture and management structure, contributed to the association between participation and decreased ED use and may limit the difference-in-difference approach. However, we accounted for secular trends and state-level efforts in our analysis. Also, the Supreme Court ruling22  resulting in a loss of some self-insured patients could have biased our results, if there was differential loss between participating and control practices. This may be why the percent of non-Medicaid patients decreased in both groups over time but the difference in the proportion of non-Medicaid patients between participating and control practices was stable over time. This study was also limited to children attributed annually to practices in a QI network, limiting the generalizability of our study findings to practices with previous exposure to QI. Finally, children were reattributed to practices each year, and we did not limit our analyses to only children attributed to the same practice over time. This attribution method could have biased our results to find a positive effect, if children with more severe asthma differentially moved away from participating practices to control practices over time, but there are no data to suggest this differential movement of patients with asthma.

Participation in a 9-month QI collaborative to improve primary care systems and supports for children with asthma led to a substantial decrease in asthma-related ED visits, with improvements continuing over time, reaching statistical significance a year after completion of the QI collaborative. Our study suggests that implementation of an asthma-related QI collaborative may be associated with reductions in asthma-related ED use, although the return on investments may take >1 year to realize after completion. Our network practices that participated partially in the QI collaborative activities still showed improvement in asthma outcomes, implying that practices benefited from partial participation. Future studies may better delineate which collaborative elements result in greatest benefit.

We thank Robert Thombley, BS, and Victoria Hart, PhD, for their statistical programming to measure ED visit rates. The analyses, conclusions, and recommendations from these data are solely those of the authors and are not necessarily those of the Green Mountain Care Board.

Dr Harder designed the study, conducted statistical analyses, interpreted results, and drafted the manuscript; Ms Kill extracted and cleaned data for analysis, helped identify relevant literature for citing, and drafted sections of the methods; Dr McCulloch supported the statistical analyses and guided the display and interpretation of results; Drs Shepard and Robinson were part of the quality improvement implementation team and drafted sections of the introduction; Drs Shaw and Cabana helped interpret findings and contributed to the discussion; Dr Bardach supported the analytic design, helped interpret findings, and contributed to all sections of the manuscript draft; and all authors reviewed and revised the manuscript, approved the final manuscript as submitted, and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Agency for Healthcare Research and Quality (U18HS025297).

     
  • CHAMP

    Child Health Advances Measured in Practice

  •  
  • ED

    emergency department

  •  
  • FQHC

    federally qualified health center

  •  
  • HSA

    hospital service area

  •  
  • NHLBI

    National Heart, Lung, and Blood Institute

  •  
  • PQMP

    Pediatric Quality Measures Program

  •  
  • QI

    quality improvement

  •  
  • RHC

    rural health center

  •  
  • VCHIP

    Vermont Child Health Improvement Program

1
Centers for Disease Control and Prevention
.
2016 National Health Interview Survey (NHIS) data
.
Available at: https://www.cdc.gov/asthma/nhis/2016/data.htm. Accessed January 14, 2020
2
Nath
JB
,
Hsia
RY
.
Children’s emergency department use for asthma, 2001–2010
.
Acad Pediatr
.
2015
;
15
(
2
):
225
230
3
Pearson
WS
,
Goates
SA
,
Harrykissoon
SD
,
Miller
SA
.
State-based Medicaid costs for pediatric asthma emergency department visits
.
Prev Chronic Dis
.
2014
;
11
:
E108
4
Agency for Healthcare Research and Quality
.
HCUPnet, Healthcare Cost and Utilization Project
.
Rockville, MD
:
Agency for Healthcare Research and Quality
;
2019
.
Available at: https://hcupnet.ahrq.gov/. Accessed December 10, 2019
5
National Heart, Lung, and Blood Institute
.
Expert Panel Report 3: Guidelines for the Diagnosis and Management of Asthma
.
Bethesda, MD
:
National Heart, Lung, and Blood Institute, National Institutes of Health, US Department of Health and Human Services
;
2007
6
Cloutier
MM
,
Hall
CB
,
Wakefield
DB
,
Bailit
H
.
Use of asthma guidelines by primary care providers to reduce hospitalizations and emergency department visits in poor, minority, urban children
.
J Pediatr
.
2005
;
146
(
5
):
591
597
7
Shapiro
A
,
Gracy
D
,
Quinones
W
,
Applebaum
J
,
Sarmiento
A
.
Putting guidelines into practice: improving documentation of pediatric asthma management using a decision-making tool
.
Arch Pediatr Adolesc Med
.
2011
;
165
(
5
):
412
418
8
Yawn
BP
,
Yawn
RA
.
Measuring asthma quality in primary care: can we develop better measures?
Respir Med
.
2006
;
100
(
1
):
26
33
9
Vernacchio
L
,
Francis
ME
,
Epstein
DM
, et al
.
Effectiveness of an asthma quality improvement program designed for maintenance of certification
.
Pediatrics
.
2014
;
134
(
1
).
10
Mangione-Smith
R
,
Schonlau
M
,
Chan
KS
, et al
.
Measuring the effectiveness of a collaborative for quality improvement in pediatric asthma care: does implementing the chronic care model improve processes and outcomes of care?
Ambul Pediatr
.
2005
;
5
(
2
):
75
82
11
Hogan
AH
,
Rastogi
D
,
Rinke
ML
.
A quality improvement intervention to improve inpatient pediatric asthma controller accuracy
.
Hosp Pediatr
.
2018
;
8
(
3
):
127
134
12
Homer
CJ
,
Forbes
P
,
Horvitz
L
,
Peterson
LE
,
Wypij
D
,
Heinrich
P
.
Impact of a quality improvement program on care and outcomes for children with asthma
.
Arch Pediatr Adolesc Med
.
2005
;
159
(
5
):
464
469
13
Mangione-Smith
R
,
Zhou
C
,
Corwin
MJ
,
Taylor
JA
,
Rice
F
,
Stout
JW
.
Effectiveness of the Spirometry 360 quality improvement program for improving asthma care: a cluster randomized trial
.
Acad Pediatr
.
2017
;
17
(
8
):
855
862
14
Dolins
JC
,
Powell
J
,
Wise
E
, et al
.
Improving asthma care by building statewide quality improvement infrastructure
.
Pediatrics
.
2017
;
140
(
2
):
e20161612
15
Shaw
JS
,
Norlin
C
,
Gillespie
RJ
,
Weissman
M
,
McGrath
J
.
The national improvement partnership network: state-based partnerships that improve primary care quality
.
Acad Pediatr
.
2013
;
13
(
suppl 6
):
S84
S94
16
Institute for Healthcare Improvement
.
The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement
.
Boston, MA
:
Institute for Healthcare Improvement
;
2003
17
Harder
VS
,
Barry
SE
,
Ahrens
B
,
Davis
WS
,
Shaw
JS
.
Quality improvement to immunization coverage in primary care measured in medical record and population-based registry data
.
Acad Pediatr
.
2018
;
18
(
4
):
437
444
18
Harder
VS
,
Barry
SE
,
French
S
,
Consigli
AB
,
Frankowski
BL
.
Improving adolescent depression screening in pediatric primary care
.
Acad Pediatr
.
2019
;
19
(
8
):
925
933
19
Weinberger
SJ
,
Cowan
KJ
,
Robinson
KJ
, et al
.
A primary care learning collaborative to improve office systems and clinical management of pediatric asthma [published online ahead of print December 14, 2019]
.
J Asthma
. doi:
20
Langley
GL
,
Moen
RD
,
Nolan
KM
,
Nolan
TW
,
Norman
CL
,
Provost
LP
.
The Improvement Guide: A Practical Approach to Enhancing Organizational Performance
, 2nd ed.
San Francisco, CA
:
Jossey-Bass
;
2009
21
Courtlandt
CD
,
Noonan
L
,
Feld
LG
.
Model for improvement - part 1: a framework for health care quality
.
Pediatr Clin North Am
.
2009
;
56
(
4
):
757
778
22
.
Gobeille v Liberty Mutual Insurance Company, 136 US 936 (2016)
23
Wasserman
RC
,
Varni
SE
,
Hollander
MC
,
Harder
VS
.
Change in site of children’s primary care: a longitudinal population-based analysis
.
Ann Fam Med
.
2019
;
17
(
5
):
390
395
24
Bardach
NS
,
Neel
C
,
Kleinman
LC
, et al
.
Depression, anxiety, and emergency department use for asthma
.
Pediatrics
.
2019
;
144
(
4
):
e20190856
25
Implement for Child Health
.
Documents
.
Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed January 14, 2020
26
Christensen
AL
,
Zickafoose
JS
,
Natzke
B
,
McMorrow
S
,
Ireys
HT
.
Associations between practice-reported medical homeness and health care utilization among publicly insured children
.
Acad Pediatr
.
2015
;
15
(
3
):
267
274
27
Mistry
KB
,
Chesley
F
,
LLanos
K
,
Dougherty
D
.
Advancing children’s health care and outcomes through the Pediatric Quality Measures Program
.
Acad Pediatr
.
2014
;
14
(
suppl 5
):
S19
S26
28
Collaboration for the Advancement of Pediatric Quality Measures
.
Collaboration for the Advancement of Pediatric Quality Measures
.
Available at: www.capquam.org. Accessed January 14, 2020
29
Agency for Healthcare Research and Quality
.
About the Pediatric Quality Measures Program
.
Available at: www.ahrq.gov/pqmp/about/index.html. Accessed January 14, 2020
30
Egorova
N
,
Gesten
F
,
Anarella
J
.
Measuring undesirable utilization outcomes when assessing the quality of care for children with asthma: findings and considerations. In: Proceedings from the AcademyHealth Annual Research Meeting; June 8–10, 2014; San Diego, CA
31
Norton
EC
,
Miller
MM
,
Kleinman
LC
.
Computing adjusted risk ratios and risk differences in Stata
.
Stata J
.
2013
;
13
(
3
):
492
509
32
.
StataCorp LLC
.
Stata Statistical Software: Release 15 [computer program]
.
College Station, TX
:
StataCorp LLC
;
2017
33
Luskin
AT
,
Chipps
BE
,
Rasouliyan
L
,
Miller
DP
,
Haselkorn
T
,
Dorenbaum
A
.
Impact of asthma exacerbations and asthma triggers on asthma-related quality of life in patients with severe or difficult-to-treat asthma
.
J Allergy Clin Immunol Pract
.
2014
;
2
(
5
):
544
552–2
34
Cabana
MD
,
Slish
KK
,
Evans
D
, et al
.
Impact of physician asthma care education on patient outcomes
.
Pediatrics
.
2006
;
117
(
6
):
2149
2157

Competing Interests

POTENTIAL CONFLICT OF INTEREST: Dr Cabana is a member of the United States Preventive Services Task Force. This article does not necessarily reflect the views of the United States Preventive Services Task Force; the other authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: Dr Cabana is a member of the United States Preventive Services Task Force. This article does not necessarily reflect the views of the United States Preventive Services Task Force; the other authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data