Instruments to Measure Outcomes in Pediatric Palliative Care: A Systematic Review

This study is in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.¹⁸ The methods were prespecified and documented in a protocol.

We conducted primary systematic literature searches using a combination of keywords, including “child,” “infant,” “pediatric,” “adolescent,” “young adult,” “palliative care,” “comfort care,” “supportive care,” “bereavement care,” “respite care,” “organization,” “standards,” “planning,” “outcomes,” “results,” and “effects” (see Supplemental Table 6 for the full search strings). We used broad keywords and Medical Subject Headings terms to maximize inclusiveness and searched 5 databases (Embase, Scopus, The Cochrane Library, PsychInfo and PubMed [Medline]) for studies published from January 1, 2006, to June 1, 2016.

The year 2006 for starting the inclusion of studies was chosen because the first International Meeting for Palliative Care in Children took place that same year in Trento, Italy, leading to the first publication of standards for PPC in Europe in 2007.¹⁹ 

All types of PPC interventions and programs were eligible for inclusion, such as supportive care, respite care, and bereavement care.

Furthermore, all types of outcomes (not restricted to QoL but focused on the child, siblings, or parents) were taken into account.

A study was included if all of the following criteria were fulfilled: (1) the full text was written in English, French, German, Italian, or Dutch; (2) the study sample included a clear description of infants, children, and/or adolescents ranging in age from 0 to 18 years; (3) children’s diseases were included in the directory of life-limiting diseases²⁰ or were labeled as life-limiting or life-limited diseases or complex chronic conditions; (4) the study included empirical data; (5) the study presented a combined description of a PPC intervention, outcome, and measure instrument; and (6) a minimal definition of PPC was presented in the study.

In contrast, a study was excluded if any of the following criteria were fulfilled: (1) sickle cell disease, diabetes, obesity, perinatal death, or chronic pain were included; (2) patient age was >18 years; and (3) being restricted to a specific molecule or treatment assessment, to pain as the single outcome, or to an economic assessment.

Children with sickle cell diseases are rarely referred to PPC teams. The management of pain alone was not a criterion of inclusion. Perinatal deaths were also excluded because they concern a specific population with particular PPC needs for which the literature would need to be searched independently.

After retrieving all records, the duplicates were removed. All studies were initially screened on the basis of titles and abstracts and then on the basis of the full text. Three authors (M.F., I.A., and J.M.D.) independently assessed the eligibility of the studies. Any discrepancy was discussed and resolved by consensus. The quality of studies included was assessed by using the Standard Quality Assessment Criteria for Evaluating Primary Research Papers From a Variety of Fields checklist.²¹ Supplemental Table 7 includes detailed information on the data extraction and analysis.

There are several ways to interpret a reliability coefficient of a given value depending on the type of characteristic measured and the method of obtaining the estimate of reliability. In classic test theory, a reliability coefficient can be interpreted as the proportion of the observed variance that is “true” rather than the “error variance.” However, 1 difficulty with expressing the reliability coefficient as a dimensionless ratio of variances is that it is difficult to interpret in terms of an individual score.²² The standard error of measurement (SEM) is defined in terms of the SD (σ) and reliability as SEM = σ × √1−reliability. Knowledge of the SEM makes it possible to construct a 95% confidence interval (CI) (95% CI = ± 1.96 × SEM) around a person’s observed score so that the amount of measurement error around that score can be quantified in a meaningful way.¹⁷ Thus, we decided to compute the SEM whenever possible using either the reported reliability index or any other reliability figures available in the literature.

Our electronic search was performed on June 4, 2016, and we identified 2150 studies. Figure 1 includes the PRISMA flowchart.

After duplicates were removed, 2111 studies were screened on the basis of titles and abstracts, and 44 studies were screened on the basis of the full text. With this procedure, we identified 19 studies that met all the inclusion criteria.

Table 1 includes an overview of the main characteristics of the included studies.

A total of 19 studies were included in this systematic review, and researchers in 15 used quantitative methods, those in 1 used qualitative methods, and 3 were based on mixed methods (qualitative and quantitative). Regarding the design of the studies, 9 were retrospective, 6 were prospective, and 4 consisted of randomized controlled trials. Among all the included studies (n = 19), researchers in 14 used instruments (n = 23 different instruments). Three studies were based exclusively on a chart review, and researchers in 2 studies used interviews or focus groups to measure outcomes. Researchers in the included studies covered the observational period from 1990 to 2014, a time frame of 24 years, excluding 2 studies in which researchers did not specify the time period. The patient population included patients with cancer (n = 10), children facing various life-limiting or life-threatening conditions (n = 8), or children with a “serious illness” (n = 1). Of the studies, 8 were conducted in the United States, 7 were conducted in Europe, 2 were conducted in Australia, 1 was conducted in Canada, and 1 was conducted in Lebanon. Researchers in half of the studies collected data exclusively among parents (n = 9). Children and parents were both interviewed in 5 studies. Researchers in 1 study collected data among parents and health professionals, and those in another study combined the children’s, parents’, and health care professionals’ perspectives. Sample sizes varied from 11 families to 134 families.

Classification of the quality of the studies revealed that 5 studies were of high quality (>8 out of 10), 5 were of good quality (6–8 out of 10), and 9 were of moderate quality (4–6 out of 10). None of the studies were labeled as being of poor quality.

Interventions or programs presented in the studies varied from home care (n = 6) to hospital care (n = 5), hospice care (n = 2), and respite care (n = 3) or a combination of home, community-based, and hospital care (n = 3).

Flexibility was found in the individualized approach in PPC interventions, in which the focus is placed on the personal desires and priorities expressed by the children and their families. Table 2 includes the types of interventions offered, the outcomes expected versus achieved, and the definition of PPC.

The outcomes addressed in the studies, when focused on children, were multidimensional and included physical (pain, fatigue, dyspnea, and appetite), psychological (anxiety and depression), social (relationships), and spiritual dimensions. QoL, when explicitly explored, was defined as “having fun or experiencing events that added meaning to life.”^23,26,27 Other outcomes were related to satisfaction with care and communication with health care professionals, often in parallel with the opportunity given to the children or adolescents to express their wishes, treatment preferences, or place of care. Finally, the length of hospitalization or the place of death were outcomes searched for through chart review and interpreted by authors as quality indicators of services.

Table 3 includes the psychometric properties of the instruments included in the review, and Table 4 includes the most often cited instruments.

Among the studies, 23 different instruments were identified (Pediatric Quality of Life Inventory 4.0 [PedsQL 4.0] in 3 studies, Quality of Life in Life-Threatening Illness–Family Carer version [QOLLTI-F] in 2 studies, Survey About Caring for Children With Cancer [SCCC] in 2 studies, and Hospital Anxiety and Depression Scale [HADS] in 2 studies), all of which were standardized measures. The average of the number of instruments used per study was 2, ranging from 1 to 6 instruments.

Among the psychometric properties, the Cronbach α was the most frequently reported value (indicated in 15 of 23 instruments). The Cronbach α is a measure of internal consistency that reveals how closely related a set of items are as a group. However, no data other than consideration of the face validity of the instruments were found among the included studies.

The SEM could be computed for 12 different instruments presented in 8 different studies. The SEM varied from 0.38 on a scale from 0 to 70 (with 95% CI = ± 0.74) for the QOLLTI-F to 6.27 on a scale from 0 to 100 (with 95% CI = ± 12.29) for the PedsQL 4.0.

PPC is a recent subspecialty in medicine that has rapidly developed over the past 2 decades. Attempts to assess the impact of such interventions are progressively published, but the findings of this review reveal that the psychometric properties of the instruments used in this field are not well documented. The aim of those in PCC is to promote QoL, but a precise description of how this construct is operationalized has not been provided in any of the included studies. As a matter of fact, QoL was addressed as an outcome directly or indirectly in only 6 of the 19 studies. Measuring the QoL of children in the context of a life-limiting disease requires rigorous conceptual and methodological foundations to capture core domains by using a holistic and family-centered approach. Some methodological, clinical, and conceptual challenges will be further discussed.

Overall, this review revealed that outcome measures are used predominantly to explore parental (proxy) perspectives retrospectively concerning a past event, such as children’s symptoms, circumstances of death, or satisfaction with the care received. However, retrospective studies have been found to be less reliable than prospective ones regarding assessing outcomes. Conte et al⁷⁶ affirm that prospective studies are needed to better evaluate the overall impact of PPC on the health care system from the perspectives of children and families. Furthermore, outcome measures in PPC seldom involve the perspectives of the children, as found in our review, in which researchers in only 5 studies addressed children’s views. Ethical concerns are widely documented, and researchers argue that participating in interviews may be burdensome for children facing life-limiting conditions.⁷⁷ In recent decades, however, several authors have recommended the involvement of children in research, showing that if the design is adapted to the children’s cognitive and physical conditions and researchers adopt a compassionate and sensitive approach, participation in studies would be valued by children and their parents and would even improve their decision-making capacity.⁷⁸ When children are involved in research, they are frequently associated or compared with proxy parental measures. In doing so, the discrepancy between self-report (by children) and proxy reporting (by parents) is found, as reported in other studies,^79,–82 because of the unique subjective evaluations made by children, which clearly differs from those made by their parents.

It is hypothesized that the use of patient-reported outcome measures (PROMs) would improve the quality of care provided. Nevertheless, this link is rarely assessed. Measurement instruments can be used for several purposes. They can be intended to be used as a primary guide for the caregiving process rather than an assessment of the impact of the interventions or a monitor of patient-related health outcomes over time. Physicians often consider PROMs to be useful for the assessment and screening of patients, whereas nurses perceive those instruments as almost relevant for clinical decision-making.⁸³ 

Difficulty measuring outcomes lies in the fact that all the described interventions could be labeled as complex interventions, which are defined by Craig et al⁸⁴ as requiring a large number of health care professionals, organizational levels, and outcomes and a large degree of flexibility. Whether at home or in a hospital, PPC interventions are characterized by interdisciplinary teams composed of nurses, physicians, psychologists, and social workers. Coordination of care is frequently ensured by several levels of care: primary home-care teams collaborating with specialist hospital-based teams. Interdisciplinary teams are a core element of quality care in PPC because the aim is to promote QoL not only for a child but also for the whole family, considering all the dimensions, such as physical health, psychological well-being, and social relationships.

In adults, a study revealed that questions relating to pain, symptoms, emotional needs, and family concerns are consistently considered to be the most useful and important components of palliative PROMs.⁸⁵ Dimensions that matter the most for children in PPC are currently not known. According to its definition, palliative care of children consists of a family-centered approach to achieve the best QoL of the children embedded among family members. This focus should consequently be addressed through the choice of a measure that is used to evaluate the outcomes of PPC on the children and their families. This idea was pointed out by Dussel and Medin.⁸⁶ However, patient involvement in the development of a PROM is paramount and has been documented in adults by Wiering et al⁸⁷ but not in children. As already mentioned, health-related outcome measures are either based on formative (causal) or reflective models or a mix of the 2.¹⁷ For this review, we were interested in discovering which model was used for the instruments that we identified. We found that all the instruments identified in this review were based on reflective models, which indicated that the explored dimensions reflected the underlying construct, such as the quantity of pain or the degree of physical impairment.

Another challenge is related to the choice of specific (disease-specific, such as cancer) versus generic instruments.⁸⁸ Generic measures may be useful in comparing outcomes across different populations and interventions, especially for cost-effectiveness studies. Disease-specific measures are used to assess the special states and concerns of diagnostic groups. Specific measures may be more sensitive for the detection and quantification of small changes that are important to clinicians or patients.⁸⁹ A systematic review performed by Janssens et al⁹⁰ revealed that many generic instruments are available to assess children’s health. However, the authors found that only a few PROMs were usable across all age ranges up to 18 years. Most PROMs excluded at least 1 major domain, and all of them conflated elements of functioning and well-being in the scales.⁹⁰ Additionally, the authors found at least 5 different psychometric properties for the Child Health and Illness Profile, Healthy Pathways, the Kidscreen, and the Multidimensional Student’s Life Satisfaction Scale.⁹¹ 

Standardized measures are instruments that contain predefined domains, whereas individualized measures are instruments that encourage patients to describe which domains are important to them. The only individualized measure of QoL is called the Scheduled Evaluation of Individual Quality of Life and was developed by Hickey et al⁹² in 1996.

Standardized measures of QoL have been criticized by several authors, who argue that some psychometric properties of those measures, such as the test and retest, are not valid in a PPC context. Rather, the same authors suggest the use of individualized measures of QoL,^13,14 which seem to better reflect the subjective perception of QoL independently of the decline in physical function. Another perspective is presented by Wolpert,⁹³ who recommended combining at least 1 standardized measure with 1 individualized measure to optimally assess QoL.

Systematic reviews on PROMs have been performed for children suffering from burns⁹⁴ or who are visually impaired,⁹⁵ but specific PROMs for PPC are clearly lacking.

Recently, through expert meetings and international conferences, a generic promising PROM was developed that is used to address the specific methodological and conceptual challenges emerging from the PPC context.^96,97 This multidimensional instrument is used to explore children’s perspectives on psychical, psychological, and social aspects of care while receiving palliative care. Parental perspectives on information, confidence, and worries are equally assessed by using the same tool.

See Table 5, in which we present the African Palliative Care Association Children’s Palliative Outcome Scale (APCA c-POS). Because no reports were published yet on the use of this new instrument in relation to an intervention, we did not integrate this development in our overview.

PROMs make up an instrument that is used to measure outcomes of care provided and to provide valuable information on quality of care. Nevertheless, the impact of care should be assessed not only by using outcomes measures. Documentation of the processes and types of interventions are required. Because those in PPC intend to promote QoL, a question could concern how much PROMs contribute to the overall aim to evaluate QoL or the common ground or convergent domains assessed equivalently by using PROMs compared with QoL instruments.⁹⁸ 

After the validation of an outcomes measure tool in PPC, a next step would be a rigorous implementation process for which guidelines are issued.^99,100 Dissemination and implementation science is a growing research field that is focused on the best strategies to implement evidence practice.¹⁰¹ A lack of education and training were found to be the most common reasons leading to the poor implementation of PROMs and therefore must be strengthened.¹⁰² Some authors point out the importance of involving health care professionals right from the start during the developmental stages of a measure tool to improve its implementation.^103,104 Patient satisfaction toward PROMs is an important element to consider in implementation research, which was explored by Recinos et al¹⁰⁵ in adults; however, similar research is lacking in children.

With this systematic review, we offer for the first time (to our knowledge) an in-depth description of the instruments used to assess the impact of PPC interventions.

Our assessment of the psychometric properties related to instruments was restricted to the data that were retrievable from the publications included. We had no access to the developmental process used for the instruments or the underlying concepts underpinning the construction of the scales.

This systematic review led to 2 observations. First, a broad spectrum of outcomes are targeted in the studies, demonstrating a lack of consensus on the dimensions that should be measured when assessing PPC interventions. Second, the psychometric properties of outcome measurement instruments are poorly documented.

It is recommended to strengthen prospective studies involving children in a sensitive and age-appropriate way. Rigorous methods are needed to develop new instruments that are able to measure QoL by using a multidimensional and family-centered approach. The use of instruments that combine self and proxy reports or even those that are used to address individual QoL requires further investigation. Coconstruction of the tool should be planned with health care professionals to guarantee optimal implementation into clinical care.

APCA c-POS

African Palliative Care Association Children’s Palliative Outcome Scale

CI

confidence interval

HADS

Hospital Anxiety and Depression Scale

PedsQL 4.0

Pediatric Quality of Life Inventory 4.0

PPC

pediatric palliative care

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-analyses

PROM

patient-reported outcome measure

QoL

quality of life

QOLLTI-F

Quality of Life in Life-Threatening Illness–Family Carer Version

SCCC

Survey About Caring for Children With Cancer

SEM

standard error of measurement