Written discharge instructions help to bridge hospital-to-home transitions for patients and families, though substantial variation in discharge instruction quality exists. We aimed to assess the association between participation in an Institute for Healthcare Improvement Virtual Breakthrough Series collaborative and the quality of pediatric written discharge instructions across 8 US hospitals.
We conducted a multicenter, interrupted time-series analysis of a medical records-based quality measure focused on written discharge instruction content (0–100 scale, higher scores reflect better quality). Data were from random samples of pediatric patients (N = 5739) discharged from participating hospitals between September 2015 and August 2016, and between December 2017 and January 2020. These periods consisted of 3 phases: 1. a 14-month precollaborative phase; 2. a 12-month quality improvement collaborative phase when hospitals implemented multiple rapid cycle tests of change and shared improvement strategies; and 3. a 12-month postcollaborative phase. Interrupted time-series models assessed the association between study phase and measure performance over time, stratified by baseline hospital performance, adjusting for seasonality and hospital fixed effects.
Among hospitals with high baseline performance, measure scores increased during the quality improvement collaborative phase beyond the expected precollaborative trend (+0.7 points/month; 95% confidence interval, 0.4–1.0; P < .001). Among hospitals with low baseline performance, measure scores increased but at a lower rate than the expected precollaborative trend (−0.5 points/month; 95% confidence interval, −0.8 to −0.2; P < .01).
Participation in this 8-hospital Institute for Healthcare Improvement Virtual Breakthrough Series collaborative was associated with improvement in the quality of written discharge instructions beyond precollaborative trends only for hospitals with high baseline performance.
Written discharge instructions serve as an important tool to bridge hospital-to-home transitions for patients and families. Studies show substantial variation in the quality of discharge instructions. Few studies report on multicenter strategies to improve discharge instruction quality in pediatric populations.
This study describes interventions tested within an 8-hospital improvement collaborative to improve the quality of written discharge instructions. Participation in the collaborative was associated with sustained improvement in the quality of written discharge instruction content.
Written discharge instructions (ie, information provided to patients) are an important tool for bridging hospital-to-home transitions.1–3 Written discharge instructions as part of bundled transition interventions (including needs assessments, patient-tailored verbal instructions, and follow-up scheduling help) have been associated with higher follow-up appointment rates, higher patient or caregiver postdischarge self-efficacy, and hospital reutilization reductions among adults.2 However, pediatric studies note that written discharge instructions are often confusing, conflict with verbal instructions, or omit important details that families need after hospitalization.4,5 To our knowledge, only 1 previous pediatric study reported on multicenter strategies to improve the discharge process; however, this study did not specifically focus on improving written discharge instruction processes using a validated quality measure.6
In 2014, the Agency for Healthcare Research and Quality-funded Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN), 1 of 7 Pediatric Quality Measures Program Centers of Excellence, developed and validated a medical records-based measure to assess pediatric written discharge instructions quality.7,8 The quality measure assesses whether hospitalized children had medical record documentation of receiving written discharge instructions containing 9 elements: (1) admission and discharge dates, (2) admission and discharge diagnoses, (3) discharge medications, (4) follow-up appointments, (5) telephone number for scheduling needed appointments, (6) 24/7 hospital telephone contact number if problems arise related to the admission, (7) pending test results, (8) necessary follow-up tests, and (9) immunizations given during hospitalization.8 The measure is scored on a 0 to 100 scale, where higher scores reflect better quality. Previous field testing revealed substantial variation in measure performance across 5 hospitals, with average scores ranging from 47 to 81 points.8
In 2016, during the second phase of the Agency for Healthcare Research and Quality-funded Pediatric Quality Measures Program, institutions within the Pediatric Research in Inpatient Settings Network9 partnered with the COE4CCN to address variation in written discharge instruction quality across children’s hospitals by implementing an Institute for Healthcare Improvement Virtual Breakthrough Series (IHI VBTS) quality improvement (QI) collaborative.10 An IHI VBTS QI collaborative approach was chosen because this approach facilitates shared learnings on implementation of clinical workflows among multidisciplinary teams at participating sites to decrease variation and promote best practices.10
The primary objective of this study was to examine the association between participation in this IHI VBTS collaborative and the quality of written discharge instructions (assessed using the COE4CCN measure) within and across 8 hospitals. We hypothesized that QI collaborative participation would improve measure performance above and beyond secular trends and improved performance would be sustained postcollaborative. A secondary objective was to evaluate whether hospital measure performance in the precollaborative phase moderated the association between QI collaborative participation and trends in measure performance. We hypothesized that hospitals with lower initial measure performance would have greater performance improvement during and after the collaborative compared with hospitals with higher initial performance.
Methods
Study Design, Setting, and Population
To assess the association between IHI VBTS collaborative participation and written discharge instructions quality measure performance, we conducted a quasi-experimental interrupted time-series (ITS) analysis11 divided into 3 phases:
a 14-month precollaborative phase from September 2015 to August 2016, and December 2017 to January 2018;
a 12-month QI collaborative phase from February 2018 to January 2019; and
a 12-month postcollaborative phase from February 2019 to January 2020 (Supplemental Table 4).
The QI collaborative consisted of 8 university-affiliated hospitals within the Pediatric Research in Inpatient Settings network9 (Table 1).
Characteristics of Participating Hospitals
| No. of Hospitals | |
|---|---|
| Hospital category | |
| Freestanding children’s hospital | 5a |
| Nested children’s hospital | 3 |
| Annual No. of pediatric admissions | |
| >2000–<5000 | 1 |
| ≥5000–<10 000 | 4 |
| ≥10 000–<15 000 | 1 |
| ≥15 000–<20 000 | 2 |
| Geographic location | |
| Northeast | 1 |
| Midwest | 2 |
| Southeast | 2 |
| West | 3 |
| Electronic medical record vendor | |
| Cerner | 1 |
| Epic | 7b |
| No. of Hospitals | |
|---|---|
| Hospital category | |
| Freestanding children’s hospital | 5a |
| Nested children’s hospital | 3 |
| Annual No. of pediatric admissions | |
| >2000–<5000 | 1 |
| ≥5000–<10 000 | 4 |
| ≥10 000–<15 000 | 1 |
| ≥15 000–<20 000 | 2 |
| Geographic location | |
| Northeast | 1 |
| Midwest | 2 |
| Southeast | 2 |
| West | 3 |
| Electronic medical record vendor | |
| Cerner | 1 |
| Epic | 7b |
One of these hospitals transitioned from a nested children’s hospital to a freestanding children’s hospital during the baseline phase.
One of these hospitals transitioned from an institutionally built EMR to Epic during the baseline phase.
All pediatric patients (aged 0–18 years) discharged from a participating hospital inpatient unit were included in the sampling frame. Patients discharged from the NICU, psychiatry/behavioral health unit, or the ICU were excluded.
Each hospital and Rand Corporation received study approval from their respective institutional review boards.
Study Phases
In the precollaborative phase (Supplemental Table 4), each site pulled electronic medical record (EMR) data for patients hospitalized between September 1, 2015, and August 30, 2016, at a single time point (October 2016) to assess baseline performance on 2 previously developed COE4CCN EMR-based hospital-to-home transition quality measures: 1 focused on written discharge instructions and the other focused on inpatient–outpatient provider communication.8 These data were analyzed to select 1 quality measure to target during the planned IHI VBTS collaborative. We aimed to select a measure with:
both high- and low-performing hospitals to facilitate interhospital learning during the collaborative;
high EMR abstractor interrater reliability (assessed using the κ statistic)12 ; and
more consistent measure performance within rather than between hospitals (higher intraclass correlation coefficient assessed using the Brown-Spearman calculation).13–15
Among the 2 measures considered, the written discharge instructions quality measure best met these criteria. We collected 2 months of additional precollaborative data (December 2017 and January 2018) for the selected measure before launching the collaborative. For hospitals with high baseline measure performance (N = 5), study team QI subject-matter experts then conducted interviews with hospital personnel to inform development of a key driver diagram (Supplemental Fig 3).16
For the QI collaborative phase, we implemented the IHI VBTS model.10 This model uses a structured process of alternating learning sessions and action periods to facilitate change across multiple hospitals simultaneously.10 Two months before QI collaborative phase initiation, participating sites finalized formation of their QI teams. Teams were presented with their hospital’s baseline measure performance data at the beginning of the 12-month collaborative phase (February 2018) and asked to identify 1 to 3 elements of the written discharge instructions measure to improve upon during the collaborative. Each QI team developed 90-day aim statements to guide their improvement work and implemented multiple small rapid cycle tests of change (Plan–Do–Study–Act cycles) during the collaborative. QI team support included bimonthly webinars and monthly virtual coaching sessions with QI collaborative staff to foster joint learning. Both site-specific and collaborativewide statistical process control charts were frequently used to share measure performance (Supplemental Fig 4). QI interventions trialed by teams included EMR changes to auto-populate content into discharge instruction templates, provider education sessions, and partnership building with subspecialty services, among other interventions detailed in Table 2. The proportion of elements that were auto-populated in the baseline period and postcollaborative are provided in Supplemental Table 5. A publicly available change package based on learnings from this collaborative provides more detailed intervention descriptions.16
QI Collaborative Team Characteristics and Interventions Tested
| Hospital | Measure Focus | Interventions Tested | QI Team Composition | Baseline Performance |
|---|---|---|---|---|
| A | Admission and discharge dates | Provider and staff education about clicking “discharge date button” in EMR “discharge navigator” to auto-populate date into discharge instructions | Physician QI lead (1), inpatient medical directors (2), subspecialist, physician QI expert, RN manager of inpatient unit | High |
| 24/7 hospital contact number | Added text to EMR emergency contact SmartPhrase to include hospital unit contact number and instructions to ask for provider on-call for patients discharged from the hospital medicine service | |||
| Partnered with “readmission collaborative” to develop a general discharge instruction template with space for 24/7 contact number | ||||
| Pending tests | Added an EMR SmartPhrase directing families to view results in patient portal when available | |||
| B | Pending test results | Auto-population of pending test results into discharge instructions using data tokens | Physician QI leaders (3), director of care coordination, primary care physician, RN informaticists (2) | High |
| Follow-up tests | Education to add “future labs/studies plan” into discharge orders to facilitate auto-population into discharge instructions | |||
| Auto-population of follow-up tests into discharge summary using data tokens | ||||
| Immunizations administered | Updated data tokens that auto-populated hospital-administered immunizations to enhance formatting and reliability | |||
| C | Admission and discharge diagnoses | Educational campaign in collaboration with hospital informatics to document diagnoses | Physician QI lead (1), RN QI specialist, discharge instruction education RN committees, senior IT engineers, discharge coordinators | Low |
| Follow-up tests | Incorporated follow-up test documentation in an EMR upgrade near the end of the QI collaborative period | |||
| Immunizations administered | Partnered with IT team to create an immunization block text | |||
| D | Admission and discharge diagnoses | Developed an EMR SmartPhrase to add the diagnosis to discharge instructions | Physician QI lead (1), chief residents, clinical informaticist, RN care coordinator | High |
| Provided education to trainees and providers | ||||
| Pending test results | Developed an EMR SmartPhrase to add pending test results to discharge instructions | |||
| Provided education to trainees and providers | ||||
| Follow-up tests | Educated trainees and providers on existing workflows to include follow-up tests in discharge instructions | |||
| E | Admission and discharge dates | Transitioned to new EMR during collaborative, which included both dates (old EMR only included 1) | Project manager, hospitalists, residents, clinical informaticists, pharmacists, RNs, family advisory council members | Low |
| Admission and discharge diagnoses | Educational interventions for providers to increase use of “principle diagnosis flag” | |||
| Added discussion about diagnoses to rounding checklist as a reminder to use flag | ||||
| Immunizations administered | Partnered with IT to auto-populate hospital-administered immunizations into discharge instructions | |||
| F | Assistance scheduling appointments contact | Developed a list of all services and corresponding appointment contact information | Physician QI leaders (2), chief residents and other trainees, subspecialty physicians. | Low |
| Disseminated list to residents | ||||
| Developed a design process to maintain the list | ||||
| 24/7 hospital contact number | Developed a list of all services and corresponding emergency contact information | |||
| Disseminated list to residents | ||||
| Developed a design process to maintain the list | ||||
| G | 24/7 hospital contact number | Updated discharge instructions template to include 24/7 contact number information | Physician QI lead (1), RN, hospitalist, hospital QI and safety administrative leader | High |
| Partnered with subspecialty teams to gather accurate contact information to auto-populate the template | ||||
| Pending test results | Updated EMR SmartPhrase to auto-populate pending laboratories | |||
| Provider education about new EMR SmartPhrase | ||||
| H | Admission and discharge diagnoses | Developed provider education materials | Physician QI lead (1), chief residents, inpatient care managers, RN administration of hospital units | High |
| 24/7 hospital contact number | EMR SmartPhrase to document hospital contact number | |||
| Developed provider education materials | ||||
| Pending test results | Updated EMR SmartPhrase to auto-populate pending laboratories | |||
| Provider education about new EMR SmartPhrase |
| Hospital | Measure Focus | Interventions Tested | QI Team Composition | Baseline Performance |
|---|---|---|---|---|
| A | Admission and discharge dates | Provider and staff education about clicking “discharge date button” in EMR “discharge navigator” to auto-populate date into discharge instructions | Physician QI lead (1), inpatient medical directors (2), subspecialist, physician QI expert, RN manager of inpatient unit | High |
| 24/7 hospital contact number | Added text to EMR emergency contact SmartPhrase to include hospital unit contact number and instructions to ask for provider on-call for patients discharged from the hospital medicine service | |||
| Partnered with “readmission collaborative” to develop a general discharge instruction template with space for 24/7 contact number | ||||
| Pending tests | Added an EMR SmartPhrase directing families to view results in patient portal when available | |||
| B | Pending test results | Auto-population of pending test results into discharge instructions using data tokens | Physician QI leaders (3), director of care coordination, primary care physician, RN informaticists (2) | High |
| Follow-up tests | Education to add “future labs/studies plan” into discharge orders to facilitate auto-population into discharge instructions | |||
| Auto-population of follow-up tests into discharge summary using data tokens | ||||
| Immunizations administered | Updated data tokens that auto-populated hospital-administered immunizations to enhance formatting and reliability | |||
| C | Admission and discharge diagnoses | Educational campaign in collaboration with hospital informatics to document diagnoses | Physician QI lead (1), RN QI specialist, discharge instruction education RN committees, senior IT engineers, discharge coordinators | Low |
| Follow-up tests | Incorporated follow-up test documentation in an EMR upgrade near the end of the QI collaborative period | |||
| Immunizations administered | Partnered with IT team to create an immunization block text | |||
| D | Admission and discharge diagnoses | Developed an EMR SmartPhrase to add the diagnosis to discharge instructions | Physician QI lead (1), chief residents, clinical informaticist, RN care coordinator | High |
| Provided education to trainees and providers | ||||
| Pending test results | Developed an EMR SmartPhrase to add pending test results to discharge instructions | |||
| Provided education to trainees and providers | ||||
| Follow-up tests | Educated trainees and providers on existing workflows to include follow-up tests in discharge instructions | |||
| E | Admission and discharge dates | Transitioned to new EMR during collaborative, which included both dates (old EMR only included 1) | Project manager, hospitalists, residents, clinical informaticists, pharmacists, RNs, family advisory council members | Low |
| Admission and discharge diagnoses | Educational interventions for providers to increase use of “principle diagnosis flag” | |||
| Added discussion about diagnoses to rounding checklist as a reminder to use flag | ||||
| Immunizations administered | Partnered with IT to auto-populate hospital-administered immunizations into discharge instructions | |||
| F | Assistance scheduling appointments contact | Developed a list of all services and corresponding appointment contact information | Physician QI leaders (2), chief residents and other trainees, subspecialty physicians. | Low |
| Disseminated list to residents | ||||
| Developed a design process to maintain the list | ||||
| 24/7 hospital contact number | Developed a list of all services and corresponding emergency contact information | |||
| Disseminated list to residents | ||||
| Developed a design process to maintain the list | ||||
| G | 24/7 hospital contact number | Updated discharge instructions template to include 24/7 contact number information | Physician QI lead (1), RN, hospitalist, hospital QI and safety administrative leader | High |
| Partnered with subspecialty teams to gather accurate contact information to auto-populate the template | ||||
| Pending test results | Updated EMR SmartPhrase to auto-populate pending laboratories | |||
| Provider education about new EMR SmartPhrase | ||||
| H | Admission and discharge diagnoses | Developed provider education materials | Physician QI lead (1), chief residents, inpatient care managers, RN administration of hospital units | High |
| 24/7 hospital contact number | EMR SmartPhrase to document hospital contact number | |||
| Developed provider education materials | ||||
| Pending test results | Updated EMR SmartPhrase to auto-populate pending laboratories | |||
| Provider education about new EMR SmartPhrase |
IT, information technology; RN, registered nurse.
During the postcollaborative phase, QI collaborative teams no longer met regularly or disbanded. Study team members from each hospital continued to collect data to assess postcollaborative measure performance.
Data Collection
During all 3 phases, participating hospitals retrospectively identified cohorts of eligible patients. Research team members at each hospital collected data using a standardized EMR abstraction tool to maximize data integrity, completeness, and scoring efficiency.16 The initial EMR data pull to assess baseline performance occurred at a single time point (October 2016) in which a target of 200 eligible patients per hospital were randomly selected for study inclusion. Once the written discharge instructions quality measure was selected for the collaborative, hospitals began monthly retrospective data pulls in December 2017 (when precollaborative phase data collection resumed) through the end of the postcollaborative phase (January 31, 2020). Each month, hospital-based data programmers randomly selected a sample of 20 eligible patients who were discharged in the previous month. This sample size achieved high measure reliability (on the basis of Brown-Spearman calculations) while balancing data collection burden. The EMR abstraction tool was then used to score the measure for each monthly sample.
Quality Measure Scoring
Measure performance was assessed on the basis of previously described COE4CCN specifications.8 Briefly, we calculated a composite score for the measure on the basis of the presence or absence of each of the 9 measure elements (eg, admission/discharge diagnoses). Each element was scored as 100 if the patient’s discharge instructions contained the element, 0 if the element was not included, and missing if the patient was ineligible for inclusion of the item (eg, no immunizations were given during the hospitalization). Individual element scores were summarized to produce a mean composite score for the measure on a 0 to 100 scale, in which higher scores reflected better-quality discharge instruction content. Because achieving high scores on some elements is more difficult than for others (eg, admission/discharge dates versus 24/7 contact number), we adjusted scores for patients eligible for only a subset of measures to account for the observed degree of difficulty (ODD) associated with achieving a high score on a given element.8 ODD adjustment occurred within each study phase because subcomponent eligibility rates may differ depending on study phase.17–19 Because of the gap in data collection during the precollaborative phase, we applied ODD adjustment for the baseline period (September 1, 2015–August 30, 2016) and the 2-month period before the start of the collaborative phase (December 1 2017–January 31, 2018) separately.
Statistical Analysis
To assess our primary objective, we used ITS models where the independent variable was time (discharge month) and the dependent variable was measure performance (ie, the ODD-adjusted composite measure score on a 0–100 scale).11 We tested for
changes in slope at the start of the (a) QI collaborative phase, and (b) the postcollaborative phase; and
a change in intercept at the start of the postcollaborative phase.
Given that IHI VBTS collaborative activities were initiated gradually at the beginning of the QI collaborative phase, we also expected interventions to be implemented gradually; therefore, we did not allow for an intercept change at that time.20 In contrast, we allowed for an intercept change at the start of the postcollaborative phase because grant support for QI efforts ended and an abrupt change in performance might be expected. Models were adjusted for seasonality and hospital fixed effects.
For our secondary objective, we hypothesized that hospitals with lower baseline performance would demonstrate greater improvement in response to the collaborative for 2 reasons:
they would benefit more from shared learnings across the collaborative and potentially replicate change strategies that hospitals with high baseline performance had already implemented before the QI collaborative (Supplemental Fig 3); and
they had more room for improvement.
To evaluate whether baseline precollaborative hospital measure performance moderated associations between QI collaborative and subsequent measure performance, we added interaction terms between sites with high versus low baseline performance and time into the model; we additionally included interaction terms which allowed for the changes in slope and intercept (described previously) to vary by sites with high versus low baseline performance. High baseline performing sites were defined as hospitals whose measures scores in the baseline period (September 1, 2015–August 30, 2016) were higher than the average baseline person–level performance using all sites. Given that the interaction term between sites with high baseline performance and time was significant, meaning associations between QI collaborative participation and measure performance over time were significantly different on the basis of baseline measure performance, we executed the final model (as previously described) stratified by high versus low baseline performance.
Because differences in quality measure scores based on demographic characteristics have been reported,17 we examined whether patient characteristics (age, sex, and race/ethnicity) were confounders, and because they were not, we did not include them in our final ITS models.
Results
Our study sample consisted of 5739 eligible patients. Patients were on average aged 7.2 (SD = 6.1) years; most were male and non-Hispanic white (Table 3).
Study Sample Characteristics
| Precollaborative Phase (N = 1902) | QI Collaborative Phase (N = 1923) | Postcollaborative Phase (N = 1914) | ||
|---|---|---|---|---|
| No. (%)a | No. (%)a | No. (%)a | Pb | |
| Age, y | <.001 | |||
| 0–1 | 435 (22.9) | 540 (28.1) | 631 (33.0) | |
| 2–5 | 366 (19.2) | 386 (20.1) | 358 (18.7) | |
| 6–12 | 546 (28.7) | 459 (23.9) | 459 (24.0) | |
| 13–18 | 555 (29.2) | 535 (27.9) | 465 (24.3) | |
| Sex | .937 | |||
| Female | 890 (47.0) | 863 (46.9) | 881 (46.5) | |
| Race and ethnicityc | .213 | |||
| Asian American/Pacific Islander | 61 (3.4) | 88 (4.9) | 69 (3.7) | |
| Black/African American | 286 (15.7) | 262 (14.5) | 296 (15.9) | |
| Hispanic | 316 (17.4) | 326 (18.0) | 352 (19.0) | |
| White | 1010 (55.6) | 995 (54.9) | 980 (52.8) | |
| None of the above | 143 (7.9) | 141 (7.8) | 160 (8.6) | |
| Season | <.001 | |||
| Spring, March–May | 404 (21.2) | 477 (24.8) | 506 (26.4) | |
| Summer, June–August | 308 (16.2) | 482 (25.1) | 482 (25.2) | |
| Fall, September–November | 474 (24.9) | 480 (25.0) | 474 (24.8) | |
| Winter, December–February | 716 (37.6) | 484 (25.2) | 452 (23.6) |
| Precollaborative Phase (N = 1902) | QI Collaborative Phase (N = 1923) | Postcollaborative Phase (N = 1914) | ||
|---|---|---|---|---|
| No. (%)a | No. (%)a | No. (%)a | Pb | |
| Age, y | <.001 | |||
| 0–1 | 435 (22.9) | 540 (28.1) | 631 (33.0) | |
| 2–5 | 366 (19.2) | 386 (20.1) | 358 (18.7) | |
| 6–12 | 546 (28.7) | 459 (23.9) | 459 (24.0) | |
| 13–18 | 555 (29.2) | 535 (27.9) | 465 (24.3) | |
| Sex | .937 | |||
| Female | 890 (47.0) | 863 (46.9) | 881 (46.5) | |
| Race and ethnicityc | .213 | |||
| Asian American/Pacific Islander | 61 (3.4) | 88 (4.9) | 69 (3.7) | |
| Black/African American | 286 (15.7) | 262 (14.5) | 296 (15.9) | |
| Hispanic | 316 (17.4) | 326 (18.0) | 352 (19.0) | |
| White | 1010 (55.6) | 995 (54.9) | 980 (52.8) | |
| None of the above | 143 (7.9) | 141 (7.8) | 160 (8.6) | |
| Season | <.001 | |||
| Spring, March–May | 404 (21.2) | 477 (24.8) | 506 (26.4) | |
| Summer, June–August | 308 (16.2) | 482 (25.1) | 482 (25.2) | |
| Fall, September–November | 474 (24.9) | 480 (25.0) | 474 (24.8) | |
| Winter, December–February | 716 (37.6) | 484 (25.2) | 452 (23.6) |
Percentages are calculated excluding missing values.
χ2 tests were performed to test for differences across phases; χ2 tests were performed among nonmissing cases only; P < .05 indicates a significant difference.
Each hospital provided an indicator of Hispanic ethnicity and a separate race variable. We created mutually exclusive race/ethnicity categories such that patients who identified as Hispanic ethnicity, regardless of racial identification, were classified as Hispanic, and non-Hispanic patients were classified as white, Black, Asian American/Pacific Islander, or none of the above categories (but nonmissing).
Changes in Measure Performance Across All Hospitals During and After the QI Collaborative
Across all hospitals, unadjusted measure performance increased from an average of 66.16 (SD = 20.17) precollaborative to 75.56 (SD = 17.15) during the collaborative to 79.85 (SD = 18.02) postcollaborative.
For all participating hospitals, adjusted measure performance improved throughout the study period (Fig 1 and Supplemental Table 6). During the collaborative phase, the trend in measure performance significantly increased beyond what was expected given the increasing precollaborative trend (change in slope at the start of the QI collaborative, +0.3 points/month; 95% confidence interval [CI], 0.0–0.5; P < .05). During the postcollaborative phase, the trend in measure performance significantly decreased beyond what was expected given the increasing precollaborative and QI- collaborative trends (change in slope at the start of the postcollaborative phase, −0.6 points/month; 95% CI, −0.9 to −0.3; P < .001), meaning the rate of measure performance improvement stopped increasing and stabilized (ie, 0.3 [precollaborative slope] + 0.3 [change in slope during collaborative] – 0.6 [change in slope postcollaborative] = 0).
ITS tested for changes in measure scores (ie, change in slope) for all hospitals at the start of the (1) QI collaborative phase, and (2) postcollaborative phase; adjusted for seasonality and hospital. Higher scores reflect better quality. Data not collected during gray period.
ITS tested for changes in measure scores (ie, change in slope) for all hospitals at the start of the (1) QI collaborative phase, and (2) postcollaborative phase; adjusted for seasonality and hospital. Higher scores reflect better quality. Data not collected during gray period.
Changes in Measure Performance Stratified by Baseline Performance
The overall ITS model showed that, during the precollaborative baseline period, sites with higher baseline performance had significantly different performance over time compared with sites with lower baseline performance (P < .001). Therefore, we conducted the ITS analysis stratified by baseline performance (Fig 2 and Supplemental Table 6). Among hospitals with high baseline performance, measure scores significantly increased during the precollaborative phase (+0.1 points/month; 95% CI, 0.0 to 0.2; P < .01) and significantly increased during the QI collaborative phase above the expected precollaborative trend (change in slope at the start of the QI collaborative, +0.7 points/month; 95% CI, 0.4 to 1.0; P < .001). The trend in measure performance during the postcollaborative phase significantly decreased beyond what was expected given the precollaborative and QI collaborative trend (change of slope at the start of the postcollaborative phase, −0.7 points per month; 95% CI, −1.1 to −0.3; P < .001), but not to the extent that the combined slope of the precollaborative, QI collaborative, and postcollaborative phases was negative. Thus, scores continued to increase after the collaborative, but at a less steep rate than during the QI collaborative phase.
ITS model stratified by baseline performance tested for changes in measure scores (ie, change in slope) at the start of the (1) QI collaborative phase, and (2) postcollaborative phase; adjusted for seasonality and hospital. Higher scores reflect better quality. Data not collected during gray period.
ITS model stratified by baseline performance tested for changes in measure scores (ie, change in slope) at the start of the (1) QI collaborative phase, and (2) postcollaborative phase; adjusted for seasonality and hospital. Higher scores reflect better quality. Data not collected during gray period.
Among hospitals with low baseline performance, measure scores significantly increased during the precollaborative phase (+0.7 points/month; 95% CI, 0.6 to 0.8; P < .001), and increased during the QI collaborative phase, but at a significantly lower rate than the expected precollaborative trend (change in slope at the start of the QI collaborative, −0.5 points per month; 95% CI, −0.8 to −0.2; P < .01). After the QI collaborative ended, there was a significantly positive intercept change, but the trend in measure performance significantly decreased beyond what was expected given the precollaborative and QI collaborative trends (change of slope at the start of the postcollaborative period, −0.6 points per month; 95% CI, −0.9 to −0.2; P < .01).
Discussion
Participation in this 8-hospital QI collaborative was associated with improvement in the written discharge instructions quality, driven by hospitals with high baseline performance. For all hospitals, improved quality measure scores were maintained during the postcollaborative period. Participating hospitals primarily focused on developing and implementing EMR changes to auto-populate recommended discharge instruction elements into EMR-generated templates. This change strategy aligned with the key drivers of “standardizing discharge instruction processes” and “building in error proofs or constraints” into documentation processes. The QI collaborative provided an opportunity for teams to share and implement successful EMR SmartPhrases or replicate auto-populated links that some hospitals already had in place. Automating EMR documentation practices likely led to sustained measure performance postcollaborative, because these types of change strategies are more reliable than manual processes or educational interventions.21–23 Some hospitals faced challenges implementing EMR changes during the QI collaborative phase because of limited institutional capacity, lower priority for changes relevant to pediatric populations within hospital systems focused on adult populations, and privacy concerns about some elements (eg, pending tests for adolescent patients). Therefore, these hospitals were limited to implementing educational interventions and non-EMR changes during the QI collaborative phase, with minimal improvement in measure performance. Although QI efforts centered around EMR changes were shared across the collaborative, these shared learnings seemed to be more effective for hospitals with strong clinical informatics partnerships and more agile information technology infrastructure.24
Stratifying analyses by baseline performance facilitated detection of differential associations between QI collaborative participation and measure performance, providing a better understanding of overall trends in measure performance over time. Hospitals with standardized discharge instruction templates containing more auto-populated elements precollaborative also had higher measure performance in the baseline period (Supplemental Table 5). Except for 1 hospital (hospital D), measure performance among hospitals with high baseline performance remained constant or worsened between the baseline period (September 2015–August 2016) and the 2 months before launch of the QI collaborative (December 2017–January 2018). In contrast, among 2 of the 3 hospitals with low baseline performance (hospitals C and F), we observed a substantial improvement in measure performance between these 2 periods. During the gap in data collection, hospital C implemented EMR changes to populate some elements into their discharge instructions template, because this timing aligned with planned institutional EMR changes. For hospital F, heightened awareness of the COE4CCN quality measure itself influenced site leads to tackle EMR changes for discharge instruction elements that were more feasible to implement. Had these interventions occurred during the QI collaborative phase, we may have seen similar trends in measure performance during both the precollaborative and QI collaborative phases for hospitals with high and low baseline performance.
This study has some limitations. First, the sample only included university-affiliated children’s hospitals; therefore, these findings may not be generalizable to nonacademic or community hospitals without a dedicated pediatric unit or department. We did not have measure scores for 15 months of the precollaborative phase and could not retroactively collect these data because of resource limitations; therefore, fitted slopes assumed that trends continued in a linear fashion, adjusting for seasonality, through the 2 months before the launch of the collaborative when data collection resumed. This assumption is based on linear trends in measure scores observed throughout the study time frame (Supplemental Fig 4). However, if trends differed during this gap, we may have over- or underestimated improvements during the QI collaborative phase in comparison with actual precollaborative trends. Additionally, we did not have a control group to examine measure performance trends for hospitals not exposed to the QI collaborative.
Results of this collaborative demonstrate the feasibility of documenting appropriate discharge instructions content in a geographically diverse sample of pediatric hospitals. Resources to execute improvement work related to written discharge instructions, such as a change package, were made publicly available for implementation and adaptation.16 More research could assess whether improvements in written discharge instructions quality may lead to better patient-centered health outcomes and reductions in readmissions, emergency department revisits, and health care costs.
Conclusions
Hospitals participating in this IHI VBTS collaborative successfully implemented interventions to improve documentation of appropriate content within written discharge instructions. QI collaborative participation was associated with significant improvement in measure performance over time only among hospitals with high baseline performance. Facilitating improvement strategies, such as auto-population of EMR content, will likely enable hospitals to provide higher-quality written discharge instructions to families without the need for continued resource allocation to sustain these improvements.
Acknowledgments
We thank the following individuals who served as counsel, reviewers, and sources of expertise for the execution of this project: Q Burkhart, MS, RAND Corporation and Lowrie Ward, Children’s Hospital Association.
Dr Desai participated in the design of the study, provided oversight of the data analysis, drafted the initial manuscript, and reviewed and revised the manuscript; Dr Parast participated in the design of the study, provided oversight of data analysis, and critically reviewed and revised the manuscript; Ms Tolpadi participated in the design of the study, conducted data analysis, drafted sections of the initial manuscript, and critically reviewed and revised the manuscript; Drs Britto, Bardach, Basco, Brittan, Johnson, Wood, and Yung participated in the design of interventions, coordinated and supervised data collection, participated in interpretation of the data, and critically reviewed the manuscript for important intellectual content; Ms Dawley, Ms Gregoire, and Ms Manges, and Drs Hodo, Leggett, Sartori, Weber participated in the design of interventions, supervised and collected data, participated in interpretation of the data, and critically reviewed the manuscript for important intellectual content; Ms Esporas, Ms Gidengil, and Dr Wilson participated in the design of the study, participated in interpretation of the data, and critically reviewed the manuscript for important intellectual content; Dr Mangione-Smith conceptualized and designed the study, obtained funding for the study, provided oversight for all study procedures including the analysis, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
FUNDING: Supported with funding from the Agency for Healthcare Research and Quality, grant #U18HS025291 (Principal Investigator: Dr Mangione-Smith). The funder had no role in the design or conduct of this study.
CONFLICT OF INTEREST DISCLOSURES: The authors have indicated they have no conflicts of interest relevant to this article to disclose.
Comments