Development of Primary Palliative Care End-of-Life Quality Measures: A Modified Delphi Process

We followed a modified Delphi approach for expert opinion gathering, which is routinely used in QM development.^10,12  Our process included 5 steps:

1. a comprehensive review of the literature and of relevant adult and pediatric measures;

2. formation of a multidisciplinary expert panel composed of researchers, clinicians, and administrators focused on PPC quality;

3. development and refinement of candidate measures;

4. a national survey to assess measure importance and abstraction feasibility; and

5. final endorsement by the expert panel (Fig 1).

The institutional review board at Seattle Children’s Hospital approved this study.

A review of existing literature was conducted from September to November 2020. Because a scoping review by Widger et al in 2019 identified PPC measures and included an evaluation of the evidence available at that time, we focused on articles subsequent to this publication.¹³  We used the terms EOL, “pediatric or child,” and “quality” to identify studies that evaluated QMs related to EOL care for children.

We paired this literature review with a comparison of existing adult and pediatric QMs available through national/international quality networks, including the Center to Advance Palliative Care,¹⁴  the National Quality Forum,¹⁵  and other organizations^16–18  to identify areas of overlap, emphasis, and divergence. We used the literature review and these existing measures to develop our candidate measures.

We convened an expert panel composed of a leadership team and an advisory workgroup. The 5-person leadership team included nationally prominent PPC researchers with expertise in QM development. The leadership team met virtually over 3 months to review the candidate measures. They also identified advisory workgroup members to support the project (see Acknowledgments). This workgroup of 9 additional members included researchers, interprofessional PPC clinicians, program directors, parents of children with CCCs, international PPC QI thought-leaders, and hospital administrators. Notably, because of the funder’s focus on the Northwest United States, stakeholders were predominantly from this region (Table 1).

The expert panel met over a series of 4 1-hour virtual meetings from July to December 2021 to review and refine the candidate measures. These discussions also informed:

1. how to identify children with CCCs (ie, patient “denominator” for QM assessment); and

2. how to gain national input from PPC stakeholders.

Ultimately, the panel defined children with serious illness as those with CCCs, as identified by Feudtner et al,¹⁹  who died both in and out of the hospital. The panel also decided to conduct a national survey to involve a broad range of PPC stakeholders.

The 2-part survey was adapted from other published national PPC surveys and refined by the expert panel.^20,21  The first part collected participants’ sociodemographic characteristics, professional role, characteristics of their PPC program, and their QI experience. The second part presented candidate measures for feedback.^10,12  Respondents evaluated each candidate measure’s importance and abstraction feasibility. Importance was evaluated on a 1 to 9 Likert Scale, with 1, 2, 3 = not important; 4, 5, 6 = important but not critical; and 7, 8, 9 = important and critical. Respondents were asked if the measure was feasible to abstract from their EHR without manual review with “Yes,” “No,” and “Unsure” response options. Each candidate measure had an area for comments. The survey also included a free-response portion where respondents could suggest additional measures.

We distributed the electronic survey via Research Electronic Data Capture to existing PPC networks, including: (1) the Pediatric Palliative Improvement Network,²²  (2) the National Coalition for Hospice and Palliative Care Pediatric Task Force,²³  and (3) the PPC Research Network. These networks provided access to >400 multidisciplinary PPC clinicians and advocates/families from >75 organizations. Eligible participants included those on the e-mail listserv from these networks, and surveys could be distributed by respondents to additional participants. Surveys were sent in October 2021 with 2 reminders sent over the following 6 weeks.

Survey responses were analyzed descriptively and presented in aggregate to the expert panel for discussion and to determine whether: (1) a measure should be endorsed or withdrawn, (2) any modifications were needed, or (3) additional measures were needed. Only measures with >50% of respondents rating a measure’s importance >4 and the measure as feasible were candidates for endorsement. Panel discussions were facilitated by J.B. or E.J. using a structured format where: (1) the project aims were stated, (2) new data since the last meeting were presented, and (3) >50% of time was left for discussion, where differences of opinion/discussion were encouraged.

The literature review and review of existing measures produced a list of 22 candidate measures for the expert panel to evaluate, which was narrowed to 20 measures for inclusion in the survey by the leadership team. There were 95 survey respondents (Table 2). The majority were physicians (54%), with the largest portion associated with hospital-based PPC programs (46%) using EPIC (51%). On the basis of the survey results and final panel discussion, 17 measures were endorsed in 5 domains: (1) health care utilization, (2) interprofessional services, (3) medical intensity, (4) symptom management, and (5) communication. All candidate and final measures are shown in Table 3 with importance and feasibility ratings. Additional details are presented below.

• QM1: patient referred to hospice ≥5 days before death if discharged from the hospital. Referral to hospice had the highest importance ratings, although there were survey comments and panel discussions about the ideal length of time to ensure sufficient benefit from hospice; 5 days was selected because it was endorsed as a marker of high-quality EOL care for children with cancer.¹⁰ 

• QM2: patient seen in the emergency department <3 times in the last 30 days of life.

There was discussion about whether different indications for emergency department visits should be handled differently (eg, nasogastric tube replacement versus poor pain control), but such nuances were considered infeasible using automated data capture.

• QM3: patient hospitalized <14 days in the last 30 days of life. There was discussion of the number of days out of the final 30 days of life to consider, and 50% was chosen on the basis of previous studies examining total days spent inpatient during the EOL period and panel discussion.¹³ 

• QM4: patient hospitalized <2 times in the last 30 days of life.

Although measures about hospitalizations were ultimately endorsed, survey respondents and panelists acknowledged that time at home may not be important for all children/families at EOL.

• QM5: patient/family received specialty palliative care within 30 days before death.

Survey respondents and the panel both felt that PPC involvement was critical.

• QM6: patient/family received social work support within 14 days before death.

• QM7: patient/family received services from child life within 14 days before death.

• QM8: patient/family received spiritual care support within 14 days before death.

Survey respondents and panelists emphasized the importance of multidisciplinary psychosocial support for children/families at EOL. There was discussion about whether offering these services would be a better measure, because some families offered services may decline. However, given the goal of automated EHR abstraction, this was deemed infeasible because of different documentation practices across sites/EHRs. The panelists decided that presence of a note by the appropriate interprofessional team would indicate receipt of services.

• QM9: patient received services from ≥1 of the following: psychology/psychiatry, within 14 days before death.

• QM10: patient received services from ≥1 of the following: physical, occupation, or speech therapy, within 14 days before death.

• QM11: patient received services from hospital school within 14 days before death.

Survey respondents and panelists had 2 main concerns about these measures. First, they did not think the receipt of these services would be goal-concordant for many children. Second, it was acknowledged that some children’s hospitals may not have these services available.

• QM12: patient did not receive cardiopulmonary resuscitation at time of death (including chest compressions, electric defibrillation).

• QM13: patient died outside of the ICU.

• QM14: patient was not intubated in the last 14 days of life (aside from intubations associated with a procedure).

• QM15: patient had <2 documented procedures requiring sedation in the 14 days before death.

Survey respondents and panelists thought avoidance of medically intense care at EOL was goal-concordant for most families and would prevent suffering at EOL. However, they acknowledged that cardiopulmonary resuscitation and other procedures at EOL would be goal-concordant for some children/families. Therefore, they felt that thoughtfully determining potential target rates for these measures by obtaining baseline measurements was critical. Additionally, some children have procedures at EOL that help alleviate suffering (eg, nerve blocks) that would require sedation/intubation that they did not want to discourage, hence allowing for sedation for <2 procedures at EOL was determined to be a reasonable benchmark.

• QM16: patient did not receive artificial nutrition through 1 of the following: total parenteral nutrition/peripheral parenteral nutrition, in 2 days before death.

Although excessive artificial hydration and nutrition may cause harm at EOL, many recognized that it is emotionally difficult for some parents to discontinue. Therefore, only parenteral (not enteral) nutrition was carried forward in this measure. The time frame of 2 days was chosen on the basis of the panel’s discussion about when artificial nutrition might be most harmful rather than helpful to the patient.

• QM17: documented pain scores ≤7 of 10 in the last 3 days of life.

• M18: documented pain score within 24 hours of admission if admitted in the last 14 days of life.

Survey respondents and panelists agreed that pain assessment and management at EOL was critical, although difficult to operationalize in a standard way across differing age groups. Panelists ultimately decided that these measures were a good starting point but that patient/family-reported measures were needed.

• QM19: patient received nonpharmacologic symptom management with ≥1 of the following: massage, acupressure/acupuncture, integrative therapy, music therapy, within 14 days before death.

Although many thought this was an important measure, there were concerns that not all these services were routinely documented in the EHR, nor available at every center.

• QM20: documentation of code status ≥2 days before death (including “full code”).

Survey respondents and panelists thought documentation was critical, particularly since it allowed for clear communication of any code status (not just allowed for natural death or do not resuscitate).

• QM21: documentation of bereavement outreach to the family within 30 days after death.

There were some concerns that not all programs chart bereavement outreach in the EHR, nor have dedicated bereavement programs. However, given the importance of continued contact with families after a child’s death, this measure was endorsed.

In this study, we used a modified Delphi approach to develop the first set of EHR-abstractable, hospital-based primary PPC quality measures for children with CCCs at EOL. The 17 measures represent 5 domains: (1) health care utilization, (2) interprofessional services, (3) medical intensity, (4) symptom management, and (5) communication and highlighting some of the key similarities and differences between pediatric and adult measures, as well as challenges inherent to PPC QM development. There are implications of focusing on QMs that are easily abstractable from the EHR, which limits assessment of aspects of EOL care, including goal-concordant care. Despite this, these measures are a critical foundation for QM development in PPC and QI initiatives.

Notably, our measures were intentionally developed to be abstracted from the EHR. Most measures therefore focused on health care utilization, interprofessional support, and medical intensity, with fewer measures focused on symptom management and communication. This does not mean that these latter domains are less important, but rather that other mechanisms are needed to extract data.⁸  QMs that are patient/caregiver-reported, such as the Consumer Assessment of Health Care Providers and Systems hospice survey,²⁴  and measures being developed in pediatric oncology^9,10  will require additional time to create and significant resources to implement. Our measures provide an important foundation to understand the quality of EOL care for children with CCCs, while recognizing that measures that capture the patient/family experience will require more robust measurement approaches.

Our endorsed measures also demonstrate some of the key similarities and differences between adult and PPC QMs. First, in both pediatric and adult palliative care QMs, there are no “never” or “always” events.²⁵  For example, although our panelists endorsed patient death outside of the ICU, this does not mean that all children with CCCs should die outside the ICU. Rather, we need to find levels of ICU death across various centers so that we can begin to examine variations in practice. This variation may provide important benchmarking data that may then lead to examination of local barriers to pediatric EOL care practices such as clinician comfort/knowledge of PPC or access to home hospice care in certain regions. Second, adult QMs have historically centered on access to services and avoidance of medically intense care.²⁶  Although some of the measures in our study involve avoidance of medically intense EOL care, we proposed additional measures specifically related to pain assessment and symptom management. Third, in adults, there are QMs related to interprofessional supportive care services and communication.^27,28  In our study, measures in these domains were also endorsed and specifically related to involvement of hospice, specialty palliative care, social work, spiritual care, child life, and bereavement outreach. Ultimately, our measures were developed from existing adult QMs, which facilitates further comparison and future QI across the age spectrum.

In the future, more QMs are needed for pediatric EOL care, including measures for high-quality, specialty PPC, primary PPC before EOL, specialty-specific measures, and patient/family-reported measures. The development of robust measures will allow us to evaluate the quality of EOL care provided to children in the United States, especially among those at risk for receiving poor PPC, and pave the way for targeted interventions to ensure high-quality care for all children. For instance, limited access to palliative and EOL resources, both in the hospital and community, are likely barriers to high-quality care,^29–31  because >20% of US children live in areas without hospice services and studies suggest that many hospices lack pediatric training.^21,32,33  QMs can help identify existing receipt of services and gaps in services to facilitate equitable access to high-quality PPC.

Our next steps will be to pilot our QMs to determine the feasibility and reliability of abstracting valid data across multiple children’s hospitals and EHRs. We will then work to further validate the measures with diverse stakeholders and consider ways to include measures that can be feasibly collected into national registries, such as the Palliative Care Quality Collaborative.¹⁸  These measures can then be used by PPC programs and children’s hospitals to improve primary PPC through QI initiatives. Until now, there have been no expert panel-endorsed, pediatric-specific QMs for programs to use to improve the quality of EOL care. Instead, they had to determine their own markers of quality and/or use adult QMs. Going forward, it will be essential for programs to consider their local practices and cultures when implementing these measures and benchmarking with how other programs perform.

This study has several limitations, in particular, given the focus on automated, EHR-abstractable quality data. Focus was placed on measures related to hospital care because of the challenges with obtaining automated EHR data in the outpatient/home setting. We also recognize that there are likely tensions between what is easily measured through EHR abstraction versus what is most important to patients/families, including a focus on measuring EOL quality data after a patient’s death rather than in real time. These were constant considerations in panel discussions that are certainly critical areas for future work. Additionally, there are limited data to support established PPC QMs, and therefore we had to consider ongoing work in pediatric oncology, adult QMs, and the data from mostly single-site studies. Care was taken to include as many perspectives and experts as possible, but notably, there was emphasis placed on the Northwest region of the United States, and most of the survey respondents were clinicians rather than patients/families. Although these measures are for primary PPC, the target audience for the survey were PPC communities. Furthermore, we were not able to calculate a response rate because of the way surveys were distributed, but it was likely low. Finally, these QMs have not been validated or otherwise tested.

Through a modified Delphi approach, we developed hospital-based, EHR-abstractable primary PPC EOL quality measures. This work can serve as foundation for future PPC QM development and QI initiatives.

The Cambia Advisory Workgroup members that are not named authors are: Anne Anderson, Jackelyn Boyden, Sarah Jackson, Jennifer Kett, Kristina Toncray, Amy Trowbridge, Kimberly Widger, and Conrad Williams. Without their input and support, this study would not have been possible.

CCCs

complex chronic conditions

EHR

electronic health record

EOL

end of life

PPC

pediatric palliative care

QI

quality improvement

QM

quality measure