Measuring What Matters to Children With Medical Complexity and Their Families

In 2017, the Health Resources and Services Administration (HRSA) Maternal and Child Health Bureau funded the first CoIIN to improve systems of care for CMC. CoIINs are often used by HRSA to address complex health problems through a combination of collaborative learning, sharing innovations, and QI. The 3 goals of the CMC CoIIN were to: improve the quality of life of CMC; improve the well-being of their families; and increase the cost effectiveness of their care. The 5 related objectives of this initiative were to: increase the number of CMC in the cohort reporting a single point or locus of management in a patient or family centered medical home; increase the number of CMC with a shared plan of care; increase the number of families of CMC reporting family engagement on the individual clinical level; and increase the number of families of CMC reporting fewer unmet needs.

The CMC CoIIN was designed to include a commitment to codesign of the initiative with CMC family leaders at every level. The CMC CoIIN project was led by Boston University (BU), in partnership with the American Academy of Pediatrics, the Association of Maternal and Child Health Programs, Health Management Associates, Family Voices (comprised of family leaders), and Population Health Improvement Partners. The CMC CoIIN was additionally guided by input from a National Advisory Committee, consisting of 20 national experts in the following areas: CMC family leaders, medical care and systems of care for CMC, payment models, and QI.

The CMC CoIIN included interdisciplinary teams from 10 states: Alabama, Colorado, Indiana, Kentucky, Massachusetts, Minnesota, Oregon, Texas, Washington, and Wisconsin. The state teams were composed of family leaders, Title V Maternal and Child Health Program and Medicaid leaders, pediatricians, family led community-based organizations, and other relevant stakeholders. During the active planning and implementation phase of the initiative from 2018 to 2021, state teams worked collaboratively with the CMC CoIIN leadership team to develop, implement, and test new and innovative systems of care, purposefully customized to the needs in their state and their CMC populations. States had unique and varied foci for their QI strategies (eg, improving access, transitions, and care coordination) and populations in their CMC cohorts (eg, neonates, rural populations, and youth).

To facilitate and support state teams in their QI efforts, the CMC CoIIN created a common set of quantitative QI measures that all teams would collect throughout the CMC CoIIN (Appendix 1). We convened a Measurement Workgroup to evaluate and select the most appropriate measures to facilitate the state teams’ improvement efforts. Consistent with the commitment to codesign all aspects of the CMC CoIIN with family leaders as partners, we included family leaders who had lived experience with CMC in the measurement development process. Thus, the Measurement Workgroup consisted of 13 members, roughly equally divided between family leaders, other state team members, and the CMC CoIIN project team.

The Measurement Workgroup used a charter to outline its goals and methods, and mutually decided upon the following guiding principles and processes, in addition to the primary goal of measuring what matters for CMC families:

• Align measures with CMC CoIIN goals and objectives and show progress toward these goals and objectives over time

• Begin with existing, validated measures

• Adapt or modify existing measures if possible, before creating new measures

• Obtain input from all state teams at key points in the measure development process

• Account for additional requirements for QI initiatives,⁴ such as the need to select measures that: (a) can show improvement over time (therefore avoid using measures with high baseline levels); (b) are responsive to the types of strategies and interventions clinical teams may employ; (c) minimize the burden of data collection for state teams; and (d) are able to demonstrate improvement within the CMC CoIIN timeframe.

The data sources used in this paper were primarily to describe our processes, our learnings, and the adaptations we made as we selected and identified QI measures for the CMC CoIIN. These sources included: the CMC CoIIN Notice of Funding Opportunity, the Measurement Workgroup’s charter, agendas, meeting minutes, and supporting documents; select qualitative and quantitative data during the CMC CoIIN including the related measurement specification tables; relevant National Advisory Committee and project team meetings minutes; and family focus group workgroup (described in Results Section) notes. We also reviewed presentations and planning meeting notes for the National Convening on the Future of Care for CMC, which was the capstone event for the CMC CoIIN project and focused on synthesizing the learnings from the initiative.

The BU Institutional Review Board reviewed the CMC CoIIN plans and procedures and determined the CMC CoIIN was not human subjects research and thereby exempt.

To develop QI measures for state teams that measure what matters most to families, the CMC CoIIN, and specifically the Measurement Workgroup, faced numerous challenges and had to make numerous adaptations in the measurement development process. Below we describe some of the challenges and the major adaptations made.

To guide the QI activities of the states, the Measurement Workgroup created a standardized quantitative survey of families (whose children were enrolled in each state’s cohort) that aligned with the CMC CoIIN goals and objectives. State teams collected and submitted these data and received results every 6 months to track progress toward their objectives. Data collection and feedback continued for a total of 4 cycles of results.

Beginning in the first months of the CMC CoIIN, the Measurement Workgroup met 5 times over an 8-month period, and then as needed when measurement challenges arose during the CMC CoIIN. The workgroup identified dozens of existing, validated measures relevant to the CMC CoIIN goals and objectives that could potentially be asked of families, using a focused literature search conducted by BU staff. The search focused on existing validated measures that were potentially relevant to the varied approaches and populations used by the CMC CoIIN states in their QI plans. We identified the following potential sources of survey questions: Pediatric Integrated Care Survey, National Survey of Children with Special Health Care Needs, Family Experiences with Care Coordination, PROMIS Pediatric Item Bank, The Medical Home Index: Pediatric, The Medical Home Family Index, PedsQL Parent Family Impact, National Survey of Children’s Health, and the Family-Centered Care Assessment for Families. Using these sources, the Workgroup attempted to identify survey questions in the following domains, aligned with the HRSA QI goals and objectives: medical home, shared plans of care, unmet needs, child quality of life, family well-being, and patient and family engagement.

Measuring child quality of life and family well-being was identified by the Measurement Workgroup as the central challenge. When reviewing the existing measures identified in the literature search, family leaders in the workgroup were unified in pointing out their shortcomings. Family leaders repeatedly noted that CMC can and do live full, meaningful lives irrespective of functional disabilities. However, existing measures almost exclusively focused on physical functional status, health care utilization, and the related perceived burdens on families. Measures of physical function were additionally recognized as a deficit-based approach that failed to capture the lived experiences of family leaders. These challenges related to measuring child quality of life and family well-being led to the need for the majority of adaptations described below.

A secondary challenge the Measurement Workgroup needed to address was the usefulness of existing measures related to medical home, shared plans of care, and patient and family engagement for a QI initiative. When taking the perspective of measuring what is important to families, assessing these concepts is complex, and typically requires composite or multicomponent measures. Data from prior research studies and national surveys using such composite measures often had high proportions of respondents reporting positive responses on individual components of these measures. Utilizing questions with high levels of positive survey responses at baseline provide little benefit for QI initiatives, since it is difficult to demonstrate meaningful improvement over time when baseline levels of a measure are already high and suffer from a ceiling effect.⁴ 

As the Measurement Workgroup developed and implemented the CMC CoIIN quantitative QI measures via the family survey, the project team needed to make numerous adaptations to the original measurement strategy. We describe in detail below the major adaptations, including the addition of qualitative family focus groups, creation of new child quality of life measures, changes to the unmet health care needs measure, and use of all-or-nothing measures.

As the family surveys were finalized, it became clear, particularly among CMC CoIIN family leaders, that the identified measures related to children’s quality of life and family well-being were inadequate to capture the lived experience of families of CMC. Family leaders shared that measures of children’s quality of life and family well-being must reflect the whole child and whole family, not just their experiences related to the health care sector and medically defined functional status. Family leaders stressed that a CMC is not just a patient, they are a child first, always. These gaps in quality of life and well-being measures led the CMC CoIIN project team to add family focus groups as a necessary and complementary approach to better capture and understand the lived experience of families.

The CMC CoIIN project team convened a focus group workgroup (FGWG) to identify how best to probe for CMC quality of life and family well-being in depth. Recognizing the importance of family leadership, the group was chaired by a family leader and each state team was required to provide at least 1 family leader member. FGWG discussions explored what was missing from the CMC CoIIN quantitative survey, identified definitions or shared understandings about CMC quality of life and family well-being, and described what aspects of quality of life and well-being are most meaningful to families to support wellness and flourishing. The FGWG identified domains with descriptive elements and ranked them in terms of priority. Notably, the FGWG yielded compelling data about measuring what matters to families, primarily because of the central role of family leadership. Table 1 lists the output of the prioritized core elements of CMC quality of life and family well-being generated by the FGWG, with a few illustrative examples. These categories provided the overarching framework to create all family focus group questions.

Utilizing this framework, BU staff and family leaders developed the focus group interview guide used in all states (see Appendix 2 and Appendix 3). Focus groups held by all 10 state teams were conducted either in person or online using Zoom, with 2 rounds conducted approximately 1 year apart (some states chose to do an optional third round, and several chose to include focus groups in Spanish).

Analysis of family focus group data resulted in the emergence of a number of themes that were critical to the states’ CMC CoIIN improvement efforts, shown in Table 2.

And, importantly, the findings from the focus groups led to important information and ideas for potential improvements in states. For example, 1 state team tested and implemented virtual support groups after focus group participants expressed the benefits of the shared support they found within the focus group itself. Another state redesigned the entire patient visit in their complex care clinic based on detailed focus group input. In addition, state teams repeatedly noted that family focus group data helped them better understand and interpret their quantitative data from the family surveys.

The Measurement Workgroup could not identify any existing quantitative measures for child quality of life that would allow state teams to effectively monitor their QI work over time. The decision to create new measures was primarily driven by the desire to measure quality of life more holistically as family leaders desired and have measures that were responsive to the customized approaches in each state. Family leaders were vital in the process of creating these new measures as well as conceptualizing the preambles necessary to frame these questions for survey respondents to ensure a shared understanding of each concept. The newly created questions and preamble are shown in Fig 1.

Although validation of these new quality of life and well-being measures was outside the scope of the project, they performed well as QI measures in terms of responsiveness to the implementation of improvements by state teams. The proportion of those responding “almost always” on both child quality of life questions 1 and 2 (a combined “all-or-nothing” measure; see subsection below for more details on all-or-nothing measures) increased from 65% to 77% across all 10 states during the CMC CoIIN. Similarly, the proportion of those responding “quite a bit” on both questions 3 and 4 increased from 41% to 48%. These improvements occurred despite the substantial disruptions caused by the coronavirus disease 2019 pandemic in the middle of the CMC CoIIN.

To address the CMC CoIIN objective of reducing unmet needs, we used a 2-part question with skip logic from the 2016 National Survey of Children's Health, shown in Fig 2. As a result of the survey skip logic, only participants who answered “yes” that their child had any unmet health care needs could see and select the unmet needs for their child. Initial results from the unmet need question indicated 89% of family respondents across the 10 states reported no unmet health care needs for their child, with some states having nearly 100% of parents reporting all health care needs being met.

The project team, and especially family leaders, were skeptical of the validity of these findings. Focus group participants and CMC CoIIN family leaders alike recounted frequent unmet health care needs. Notably, focus group data identified that families may not recognize that they had unmet health care needs because they did not realize there are resources or supports to address these needs. Thus, the project team concluded that the original unmet need survey question was not eliciting information that was helpful to the state teams’ QI efforts. The project team consulted with family leaders to redesign the question and pilot test it in 3 states midway through the CMC CoIIN. The adapted question is shown in Fig 2.

The new question asked about unmet health care needs for the child upfront, with a checkbox option of “None, my child always received needed health care.” The pilot states showed a dramatic change in their results, with a mean of a 30% (range 15% to 46%) decrease in families reporting their child’s health care needs were met. The revised question would have been a much more useful QI measure for the state teams.

In addition to the newly created children’s quality of life questions mentioned above, the Measurement Workgroup decided to use “all-or-nothing measures” for medical home, shared plans of care, and 1 of the patient and family engagement measures. All-or-nothing measures are created by combining a set of related measures and determining the proportion by applying an all-or-none rule at the respondent’s level.⁵ For example, the medical home measure consisted of 7 questions related to 7 domains of medical home for families. A positive response was counted only for respondents who answered usually or almost always on all 7 questions.

Using all-or-nothing measures avoided measures having extremely high baseline levels, which make it very difficult for teams to determine if the changes they make are an actual improvement. Perhaps more importantly, all-or-nothing measures are more reflective of a full system of care and are better aligned with what patients and families want and need.⁵ Many state teams shared that the all-or-nothing measures, although sometimes disappointing early on because of the lower baseline levels, ultimately challenged their teams to take a more comprehensive systems approach to their improvements.

The all-or-nothing measures served the state teams well. The baseline levels of all-or-nothing measures for medical home, shared plans of care, and patient and family engagement ranged from 42% to 65%, much lower than the individual items, had they been used. These measures were also responsive to the improvements teams made. For example, the medical home scores across the 10 states increased from 47% to 60% during the CMC CoIIN, despite the coronavirus disease 2019 pandemic.

The CMC CoIIN provided a unique opportunity for authentic codesign of a QI initiative with families of CMC as full participants and leaders at all levels of the initiative, including our QI measurement strategy. We believe this partnership provided tremendous value and learning opportunities that greatly enhanced the initiative with a fuller, more complex view of flourishing for CMC and their families—and specifically, in this initiative, greatly enhanced our measurement approach.

One of the most important lessons we learned was the need for new, family-driven measures for children’s quality of life and family well-being. We struggled to find existing validated measures of children’s quality of life and family well-being that were both suitable for QI, and meaningful to family leaders. Family leaders desired metrics that were able to capture relationships, values, and cultural contexts that would indicate whether systems of care were supporting health and flourishing. The existing measures we identified (from some of the most commonly used surveys in the field) were deemed to miss the mark by our family leaders — primarily because they were mostly focused on health care related outcomes and physical functional status, failing to capture the fuller lived experience of CMC and their families. Some of the measures were even referred to as “ableist” by our family leaders, using a typically developing child’s development and outcomes as a benchmark against which to measure quality of life and well-being for their children.

Although we were able to create and successfully use a few new family-driven measures for child quality of life that were relevant to the QI strategies of our 10 states, they are certainly just a first step — much more work is needed to create and validate children’s quality of life measures that can be used to improve systems for and the lives of CMC. Indeed, our experience is consistent with prior health literature on quality of life measurement focusing too much on deficit-based indicators, based on relatively narrowly defined measures of physical and cognitive impairments.^6,7,8 

Recognizing the gaps in the quantitative children’s quality of life and family well-being measures we employed, we added qualitative family focus groups. The addition of these qualitative data driven by family leaders was an incredibly valuable addition to the quantitative data in this QI initiative. These data not only generated ideas for improvements in the states, but also helped state teams better understand and interpret their quantitative data.

We also learned that asking a simple yes or no about whether health care needs were unmet, without being specific upfront about the range of needs to consider, markedly missed capturing the unmet needs for many families. Again, the qualitative data from the family focus groups was a much-needed addition to our measurement strategy, including shining a light on these missed needs. It’s also important to note that the unmet needs question we used focused almost entirely on health care needs, yet the family focus groups uncovered many broader unmet needs that we were unable to explore quantitatively in the CMC CoIIN, though certainly important to explore in the future.

Finally, we created a number of all-or-nothing measures for systems issues that typically have high baseline levels for their individual components, such as medical home and shared plans of care measures. These all-or-nothing measures were much more helpful for a QI initiative because they provided teams with more room to demonstrate improvement over time and challenged them to think more about their systems, and how to improve them.

The generalizability of this study is somewhat limited. For example, the purpose of measurement in the CMC CoIIN was to drive improvement; thus, the findings may not apply to measurement for other purposes, like measurement for accountability, or establishing the prevalence of conditions. Nevertheless, the need to use metrics to drive improvement for the CMC population is both common and vitally important. Generalizability may also be limited somewhat to these specific states’ context and these specific family leaders and interdisciplinary state teams, as they were voluntary participants in an innovative initiative, and thus may differ from the general population. Finally, it was beyond the scope of the CMC CoIIN to do a systematic review of all potentially relevant existing measures to drive improvement for the CMC population — we may have missed some useful existing potential measures. It was also beyond our scope to thoroughly validate or assess the reliability of the new measures we created though it will be important to do so if they are to be used in future initiatives.

Codesigning the CMC CoIIN measurement strategy with family leaders greatly enhanced our measurement approach and provided numerous unique learning opportunities. Accordingly, we believe that providing family leaders with opportunities for professional development is a crucial workforce development strategy in the future, including opportunities to learn more about measurement and evaluation. Support for family leaders as codesigners of improvement initiatives is consistent with the success of human-centered design principles found in many other industries.⁹ For example, a fundamental takeaway we have is that family leaders helped us shift our conversations to the broader principles of health and wellness, and flourishing, rather than zeroing in on deficits.

Additionally, we believe the CMC CoIIN demonstrates the pressing need for funders to support more opportunities to develop family-driven measures that can drive systems improvement, including validating and assessing their reliability.

In closing, the recently published National Research Agenda for Children and Youth With Special Health Care Needs, designed in partnership with family leaders, has noted research priorities that focus on family health and caregiving, including social determinants of health (SDoH), home supports for caregivers, and the impact of family health and adaptability.¹⁰ Future CMC initiatives centering family codesign from the onset can further elucidate the mechanisms that are likely to lead to health and wellness, with the downstream impact of improved health care utilization – not the other way around.

The authors are grateful for the thoughtful review and feedback on this manuscript from Maureen Benschoter, Meg Comeau, Kerri Deloso, Bethlyn Houlihan, and Christopher Stille, which helped substantially improve the paper.

BU

Boston University

CMC

children with medical complexity

CoIIN

Collaborative Improvement and Innovation Network

FGWG

focus group workgroup

HRSA

Health Resources and Services Administration

QI

quality improvement