Quality Indicators for Youth Transitioning to Adult Care: A Systematic Review

This systematic review was registered with the International Prospective Register of Systematic Reviews (PROSPERO; CRD42020198030). The protocol has been published.³⁹  Detailed search strategies, data extraction tables from included studies, and categorization framework used to classify quality indicators can be found in the Supplemental Information. Template data abstraction forms may be provided upon request to corresponding author.

The databases Medline, Embase, and the Cumulative Index to Nursing and Allied Health Literature were searched from the earliest available date to July 9, 2021. The search strategy (Supplemental Tables 5-7) incorporated 2 concepts: (1) transition from pediatric to adult care and (2) performance measurement and benchmarking, which was developed by a librarian from the Hospital for Sick Children, Toronto, Canada (QM). Titles and abstracts were exported to Microsoft Excel for study selection.

Study eligibility was determined in 2 stages: title and abstract screening (level 1) and full-text review (level 2). Each study was assessed by a set of 2 independent reviewers (K.B., S.L., T.D.L.R., or L.L.), with disagreements resolved by a third reviewer (A.T.). At level 1, studies were included if they (1) focused on transition from pediatric to adult care and (2) used process, structure, or outcome measures or indicators to evaluate transition. Studies were excluded if they were not peer-reviewed or not in English. The research team pilot-tested the level 1 screening form using a random sample of 200 study records (κ = 0.612). At level 1, overall κ ranged from 0.691 to 0.761 between 4 sets of reviewers. Studies were included at level 2 if they used robust consensus-building methodology (eg, Delphi survey, Rand/UCLA appropriateness methodology, or prioritization exercise) to develop quality indicators, in addition to meeting the inclusion criteria for level 1. Studies were excluded if (1) metrics were used to evaluate local or specific transition interventions (eg, clinic-specific quality improvement initiatives) rather than the transition process in general, (2) no description or evidence of a rigorous development process for quality indicators was provided, or (3) focused on key elements of successful transition, rather than measurable quality indicators. The team also piloted the level 2 screening form using a sample of 25 full-text articles until a similar level of agreement was achieved (κ = 0.65). At level 2, overall κ ranged from 0.47 to 0.62 between 2 sets of reviewers.

Studies meeting eligibility criteria in levels 1 and 2 underwent data abstraction (level 3) using a piloted abstraction form. Data abstracted included study-level characteristics (ie, author, title, year, and country of origin), study methodology (ie, study design, population, sample size, panel characteristics, data collection method, and clinical context or disease), and quality indicator characteristics (ie, description, illness, level of care, or measurement properties and instructions). The data abstraction form was developed based on previous reviews of quality indicators.^40,41  Data were abstracted independently by 2 reviewers (S.L. and K.B.) and entered into a data abstraction form on Excel. Abstraction forms from both reviewers were collated.

A review of the gray literature was conducted from July 2021 to the earliest available date using the Canadian Agency for Drugs and Technologies in Health Gray Matters tool.⁴²  Search terms included “transition to adult care” and “quality indicators”. Two independent reviewers (K.B. and L.L.) searched the gray literature websites provided in the Gray Matters tool, evaluated literature using the predefined eligibility criteria, and recorded findings in the Gray Matters checklist. Further information about the gray literature search strategy is provided in the Supplemental Information (Supplemental Table 10).

Following a literature search of previous syntheses of quality indicators in health care, the research team identified the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument, a validated tool widely used to assess the quality of clinical practice guidelines.⁴³  A modified version (AGREE II-QI) was created for evaluation of quality indicators.⁴⁴  Although there is no agreement on which tool is most appropriate to appraise quality indicators, the AGREE II-QI has been applied in previous studies.^41,45,46 

The AGREE II-QI instrument consists of 6 domains: scope and purpose (eg, clarity of study scope), stakeholder involvement (eg, representation from all relevant disciplines), rigor of development (eg, methods used to develop quality indicators), clarity of presentation (eg, measurement properties of quality indicators), applicability (eg, inclusion of tools for use), and editorial independence (eg, competing interests).⁴⁴  The appraiser must rate the degree to which they believe the study achieves each core element of quality indicator development on a 7-point Likert scale (1 = strongly disagree and 7 = strongly agree). Total scores are calculated for each domain and an overall quality score for the quality indicator set. The AGREE II-QI uses the same scoring system as the original AGREE II instrument.⁴³ 

Two independent reviewers (K.B. and L.L.) pilot tested the AGREE II-QI instrument on 2 randomly selected articles included in level 3 to ensure its appropriateness for this review. Each study included in level 3 was appraised, and final scores from both reviewers were compiled into an Excel spreadsheet. Domain scores were calculated following the protocol in the AGREE II user manual, where a maximum score of 100% indicated both reviewers believed each element of the AGREE II-QI instrument was fully met.⁴⁷ 

Abstracted quality indicators were categorized by level of care (ie, patient, caregiver, provider, or health-system level) and disease (ie, disease-specific, or generic). Indicators were then mapped onto the Donabedian model²¹  and IOM domains²⁹  to identify gaps in measurement approaches. Although certain quality indicators could be classified under multiple IOM domains, each indicator was assigned to a single dominant domain, on the basis of the protocol from a previous systematic review.⁴¹  Operational definitions used for this classification are found in Supplemental Table 11. Finally, each quality indicator was assigned a clinically relevant major measurement theme (eg, satisfaction, continuity of care, and appointment keeping). Two reviewers (A.T. and K.B.) independently classified each indicator by level of care, Donabedian classification, IOM domain, and clinically relevant measurement themes identified by coauthors. Disagreements were discussed until consensus was achieved. This categorization framework was based off a systematic review of quality indicators in inflammatory arthritis.⁴¹ 

The search strategy and gray literature search identified 4581 studies after duplicates were removed (Fig 1). After title and abstract screening, 321 full-text articles were reviewed. Of these, 9 studies were included in the final systematic review, 8 of which were peer-reviewed and 1 was a clinical guideline identified by the gray literature search (Table 1). In total, 169 quality indicators across 29 themes were identified (see Supplemental Table 10 for abstracted quality indicators). Two quality indicator sets were developed in the United States,^48,49  2 in Canada,^35,50  2 in the United Kingdom,^51,52  1 in the Netherlands,³⁷  1 in Spain,⁵³  and 1 in Switzerland.⁵⁴ 

Of the 9 included studies and guidelines, 5 (56%) were developed in disease-specific populations: 1 in sickle cell disease (11%),³⁶  2 in inflammatory bowel disease (22%),^37,50  1 in haemophilia (11%),³⁵  and 1 in juvenile-onset rheumatic disease (11%).⁵³  These illness-specific studies yielded a total of 62 quality indicators, accounting for 37% of all identified indicators. The 4 remaining studies were not conducted in any particular disease population, yielding a total of 107 (63%) quality indicators.^48,51,52,54  Of these non–illness-specific studies, 1 included representation across chronic physical, mental, and developmental conditions (eg, diabetes, cystic fibrosis, cancer, mental illness, chronic pain, and neurodisability).⁵¹  Both Suris et al⁵⁴  and Fair et al⁴⁸  did not include information about the disciplines of expert panelists, thus it is unclear whether cross-conditions representation was achieved. The NICE quality standard for transition from children’s to adults’ services included a specialist committee with representation from physical and mental illnesses, as well as diverse community stakeholders across all conditions.⁵² 

Three quality indicator sets were conducted using a traditional Delphi study,^37,48,54  2 used a modified Delphi study,^35,36  and 4 employed study-specific consensus-building methodology (eg, iterative interviews/focus groups, series of surveys, etc).^50–53  None were developed via Rand/UCLA appropriateness methodology.

All studies (n = 9) included interdisciplinary health care providers as expert panelists in their consensus-building process (eg, physicians, nurses, transition navigators, psychologists, etc). Four included both pediatric and adult providers in their expert panels,^35,37,53,54  whereas 2 included exclusively adult providers.^36,50  The remaining 3 studies did not include information about the breakdown of pediatric to adult providers.^48,51,52  Four studies included exclusively health care providers,^35,36,50,54  whereas 4 included patients^37,48,51,53  and 3 included parents/caregivers.^48,51,52  Of those including patients, 1 reported age (range = 16.5–24.7 years old; median = 18.7 years old) and 2 reported sex (49.2% male in van den Brink et al³⁷  and 46.2% male in Aldiss et al⁵¹ ). None reported race, ethnicity, gender, sexual orientation, primary language, or socioeconomic status. No information was provided about the parent/caregiver characteristics. One study involved community stakeholders.⁵²  See Supplemental Table 8 for stakeholder characteristics.

Of the 169 quality indicators, 72 were patient level (43%), 50 were provider level (30%), 30 were health-system level (18%), and 14 were caregiver level (8.3%) (Table 2). Three (1.8%) quality indicators measured both provider- and patient-level responsibilities (eg, written transition plans are used that are cocreated and completed by the young person and care team). When categorized according to the Donabedian model,²¹  75 quality indicators corresponded to processes (44%), 56 to outcomes (33%), and 38 to structure of care delivery (23%). Using the IOM²⁷  domains, 87 indicators were classified as patient-centered (51%), 41 safe (24%), 21 effective (13%), 15 efficient (9.0%), and 5 timely (3.0%). No quality indicators related to equity were identified.

Abstracted indicators were further categorized into 29 distinct clinically relevant measurement themes (Table 3). The most common measurement theme was transition education (n = 21, 12%), followed by continuity of care (n = 13, 8%), satisfaction (n = 13, 8%), and self-management/self-efficacy (n = 11, 7%). Very few measurement themes related to life-skills and participation outcomes such as youth social development (n = 1, 0.6%), resilience (n = 1, 0.6%), and self-advocacy skills (n = 1, 0.6%). See Supplemental Table 9 for the categorization framework and abstracted quality indicators.

Critical appraisal using the AGREE II-QI yielded low scores across all domains, with an average overall quality score of 34.3% (Table 4). Across all quality indicator sets, the highest rated domains were scope and purpose (mean = 71.3%) and editorial independence (mean = 63.9%). Domains relating to stakeholder involvement were rated an average of 43.98%, which reflects exclusion of representative stakeholders (eg, youth, parents/caregivers, community stakeholders, etc). Rigor of development was rated an average of 27.3%, whereas clarity of presentation was rated an average of 24.1%. Ratings of applicability were the lowest, achieving an average domain score of 8.0% across studies. This highlights the lack of instructions for measurement or implementation in clinical practice.

This systematic review identified 169 quality indicators (63% generic and 37% illness specific) for transition to adult care for youth with chronic conditions developed via consensus-building methodologies. Our study highlights where emphasis has been placed in quality indicators for transition. To date, existing research has focused on quality indicators related to satisfaction, transition education and planning, continuity of care, satisfaction, and self-management/self-efficacy. Most quality indicators were patient-centered measures. We identified several gaps in quality indicators for transition. Although more than 50% of quality indicators were patient and caregiver level, these stakeholders were rarely included in the development process. Furthermore, there were no equity related metrics and many indicators had unclear measurement characteristics.

Most quality indicators were patient centered yet developed based on the perspectives of health care providers only. Youth possess valuable knowledge grounded in their experiences navigating the health care system with a chronic condition^55,56  and should be included, as well as parents/caregivers in the case of children and adolescents, as active partners in research to ensure results are meaningful, impactful, and patient centered.^55,57–59  Despite this, few studies included the perspectives of these key stakeholders in the quality indicator development process. When included, youth were often placed on the same panel as clinicians in fewer numbers, increasing the likelihood their priorities were overshadowed by those of the health care provider panelists. A recent Delphi study identified core components of successful transition from pediatric to adult mental health services using 3 homogeneous panels with patients, parents/caregivers, and health care providers.⁵⁸  Not only did youth have different priorities than their parents/caregivers and health care providers, but they also increased the validity of results by advising the use of youth-centered language.⁵⁸ 

Moreover, studies have called for increased engagement of knowledge users in the research process to improve the impact of health services research and translate results into meaningful clinical or health-system level policy changes.^60,61  Community and health system leaders are key stakeholders in transition. They have knowledge and experience to provide input into the barriers and facilitators to incorporating standardized measurement into clinical practice. Among the studies identified by this review, only 1 included community knowledge users in their expert panels.⁵²  Future research should incorporate the perspectives of diverse knowledge users in the development of quality indicators, such as community and health-system leaders, administrators, and policy makers.

Of the studies including patients and caregivers, 2 included information about participant age, sex, and type of illness. None of the studies highlighted demographic or sociocultural characteristics of participants, including race/ethnicity, gender, sexual orientation, and primary language. It is well known that the intersection of structural and social determinants of health impacts both access to health care services and quality of care received, including developmentally appropriate transitional care services. Youth and caregivers self-identifying as individuals that are structurally marginalized may have unique experiences with their transition to adult care. Future work should use a health equity lens to develop quality indicators that reflect the priorities of youth and caregivers who are marginalized and transitioning to adult care.

We identified few parent/caregiver-level quality indicators. Of those, many were assessing satisfaction with the transition process, with little emphasis placed on their specific roles and responsibilities. Parents/caregivers play a central role in the care of youth with chronic illness as they transition to adult care, such as promoting self-efficacy, independent communication, and autonomy, as well as supporting them to understand their condition and treatment plan.⁶²  Parents/caregivers often encounter unique challenges distinct from those of youth and their health care team, including apprehension about their children becoming principal health care decision makers and heightened concerns about potential poor health outcomes in young adulthood.^2,63,64  Involvement of parents/caregivers in transition education, preparation, and planning has been included as a recommendation for best practice in transition.^51,65  As such, we recommend future research involve consultation with parents/caregivers to establish appropriate caregiver-focused quality measures.

The quality indicators identified by this review are inconsistent with the IOM’s assertion that “the vast majority of measures address effectiveness and safety, a smaller number examine timeliness and patient-centeredness, and very few assess the efficiency or equity of care”.⁶⁶  This highlights the need for additional quality indicator development on the domains of effectiveness, equity, and safety. Given the established association between youth transfer to adult services and negative health outcomes, quality indicators assessing patient safety should be of particular importance.

Critically, we identified no quality indicators related to equity despite the impact of the social determinants of health on outcomes following transition to adult care.⁶⁷  For example, a 2021 retrospective study identified that Black race, Hispanic ethnicity, and socioeconomic factors were of importance in understanding outcomes following transition in youth with childhood-onset systemic lupus erythematosus.⁶⁸  Future work should identify equity metrics to ensure the relationship between the social and structural determinants of health and transition outcomes is understood and addressed at the patient, provider and health-system level. Those who are most impacted by health inequities should be actively engaged throughout the process of developing these metrics. There is also a need for purposeful measurement of existing quality indicators across diverse populations, whereby results are stratified by demographic and sociocultural characteristics to identify any deficiencies in equity and quality of care. Finally, future work may consider compiling existing health equity-related quality indicators from other care settings and contexts to assess whether they can be adapted and/or applied to the transitional care.

Abstracted quality indicators were both generic and specific to several chronic physical disease populations, including sickle cell disease, inflammatory bowel disease, haemophilia, and juvenile-onset rheumatic/musculoskeletal disorders. Notably, no studies had developed quality indicators specifically for mental illness or developmental conditions. There are increasing numbers of youth with physical, mental, and developmental conditions, and having a chronic physical health condition can place youth at increased likelihood of developing a mental health condition and vice versa.^69,70  As such, there is still a paucity of robust quality indicators that are broadly applicable across all chronic physical, mental, and developmental conditions, preventing universal evaluation across clinical settings and jurisdictions. We recommend that further research focus on developing a key set of generic quality indicators to evaluate transition across chronic conditions and compare/benchmark aspects of care that are applicable across patients accessing transition services, regardless of the diagnosis.

Many of the patient-level outcome measures identified were assessing constructs related to their disease, such as knowledge and self-management. Very few quality indicators evaluated the developmental, social, vocational, and psychological aspects of youth transitioning to adult care. Blum’s definition of transition encompasses physical outcomes and psychosocial, educational, vocational, and social development.⁹  Evidence has supported the need for a holistic approach to transition, including flexibility and accommodation for unique clinical and nonclinical aspects of a patient’s life, their developmental level, and capacity for learning.^5,65,71–76  As such, patient-level quality indicators must reflect this recommendation and evaluate transition using appropriate psychosocial, developmental, educational and vocational metrics.

No quality indicators related to cost, billing, or economics were identified by this review. As per the Institute of Healthcare Improvement Triple Aim Framework, improving the quality of the health system includes reducing the per capita cost of services delivered.⁷⁷  Some studies have reported cost savings and reduced acute care usage with the implementation transition interventions^78,79 ; however, there remains a need to develop robust cost-related metrics to evaluate transitional care.⁶⁵ 

As evidenced by the low applicability domain score on AGREE II-QI, very few quality indicators included the characteristics required for measurement and implementation in clinical practice. Critical quality indicator characteristics needed to standardize and optimize measurement include a numerator and denominator, frequency and interval of measurement, a defined target user, an instrument for assessment, and predefined cutoffs to benchmark appropriate achievement of the assessed construct.^30,47  In addition to these characteristics, many of the quality indicators did not include instructions for implementation or updating the quality indicator as new evidence emerges, highlighting their lack of utility in the clinical setting. To effectively implement measurement-based care into transitional services and meaningfully compare performance within and between health systems, further research is needed to establish quality indicators with both robust measurement characteristics and an implementation framework.

Our systematic review is limited to English literature only, which could have resulted in the exclusion of relevant studies. Additionally, the majority of studies identified were from North America and Europe, so the indicators identified may not include a broad international understanding of transition. This may limit their applicability in other countries or settings. Furthermore, this systematic review identified quality indicators developed through consensus-building methodology, which may have limited the scope of indicators included. Finally, we were unable to apply all relevant frameworks to categorize quality indicators, including the GotTransition Core Elements of Healthcare Transition.⁸⁰  Use of a transition-specific quality framework may have captured deficiencies in quality indicator domains related to transition to adult care, rather than gaps in overarching domains of health care quality.

Several of the quality indicators identified by this review did not have many of the characteristics needed for measurement in clinical practice. Future research should focus on refining these quality indicators to ensure youth/family-centered language and perspectives, clarity of description, including characteristics needed for measurement, and devising a framework for implementation. We plan to conduct a multistakeholder Delphi study that refines and prioritizes these quality indicators to develop a key set of measures to be used in clinical practice. This process will be inclusive of experts from across chronic physical, mental, and developmental conditions, including youth, parents/caregivers, health care professionals, and health-system leaders. A key set of quality indicators for transition that can be applied to any clinical setting and disease population is needed to compare health-system performance across all clinical settings and jurisdictions. Following this, future attention is needed to evaluate the utility of adding some disease-specific measures to the key set (eg, specific markers of disease stability), refining measurement characteristics, and developing an implementation framework.

This review synthesizes the vast amount of information on quality indicators for busy clinicians and policymakers involved in transition. We highlight the areas of quality measurement in transition which are important to measure in practice, as well as the areas that have been overlooked and require further attention. Importantly, this review emphasizes involving youth and caregivers in future work. Finally, these results are the first step in informing and aiding policy makers in developing benchmarks for transition, which are needed to measure health system performance and flag areas where further initiatives are needed to improve quality of care.

Despite the significant number of quality indicators identified by this review, there remains a need for robust, generalizable metrics evaluating transition to adult care for youth regardless of type of chronic illness. The majority of indicators identified were related to patient-centered outcomes; however, we identified significant gaps including the paucity of youth and parents or caregivers involvement in the quality indicator development process, equity focused metrics, and measurement characteristics of quality indicators. To elucidate a key set of metrics, future research should further prioritize and refine identified quality indicators in collaboration with youth, parents or caregivers, health care providers, and health-system leaders.

We thank the members of the Transition Quality Indicators subcommittee, including Dayle McCauley, Heather Buckingham, Dr Gail Andrews, Dr Michelle Batthish, Dr Karen Beattie, Dr Lorraine Bell, Dr Emilie Dionne, Danijela Grahovac, Dr Yani Hamdani, Amanda Higgins, Ariana Wong, Brooke Allemang, Jaime Winkler, Meghan Patton, Dilshad Kassam-Lillani, Sarah Sheffe, Christina Gilman, and Alice Soper, for their insights into the development of this protocol. Dr Gorter is the inaugural chair of the Health Hub in Transition and held the Scotiabank Chair in Child Health Research between 2013 and 2021. Dr Kristin Cleverley holds the CAMH Chair in Mental Health Nursing Research at the University of Toronto.

AGREE II-QI

Appraisal of Guidelines for Research and Evaluation II Quality Indicators

IOM

Institute of Medicine

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

PROSPERO

International Prospective Register of Systematic Reviews