OBJECTIVES:

The objectives of this study were to design and validate a survey measuring the parents’ and caregivers’ experiences of integration of their child’s care across providers.

METHODS:

After review of the literature on care coordination and integration, we solicited input regarding care experiences from focus groups of families with children with chronic conditions. These data informed a 95-item pilot survey that included elements from a care integration measure designed for adult care experiences. The survey was then administered to parents of children who had had at least 1 primary care appointment and 2 specialty care appointments in the previous 12 months. Psychometric analyses were used to establish scales through exploratory factor analysis, internal consistency using Cronbach’s α, test–retest reliability using Spearman’s rank correlation coefficient, and known-group validity according to χ2 tests. All research activities were institutional review board approved.

RESULTS:

The pilot survey was completed as either a Web or mail survey by 255 participants. After excluding nonrating or screening questions and items not applicable to a large percentage of participants, 26 experience items were included in the exploratory factor analysis. The final survey contained 19 experience items in 5 scales: access, communication, family impact, care goal creation, and team functioning. Psychometric analyses supported these 5 scales.

CONCLUSIONS:

This project developed and validated a survey with 19 experience items, plus additional demographic and health needs and usage items. The Pediatric Integrated Care Survey can be used in quality improvement efforts to measure family-reported experience of pediatric care integration.

Care delivery for children and youth with special health care needs (CYSHCN) is often experienced by families as being fragmented, resulting in outcomes not meeting their needs.120,2 The implementation of standardized activities of care coordination (CC) is often cited as a solution. CC addresses interrelated medical, social, developmental, behavioral, educational, and financial needs to achieve optimal health and wellness outcomes.3 It is unknown if broadly implemented CC activities influence the outcome of family experience of care integration across providers, nor how measures of CC relate to measures of family experience of integrated care. Singer et al4 have proposed a definition of integrated patient care, describing it as “care that is coordinated across professionals, facilities and support systems; continuous over time and between visits; tailored to the patients’ needs and preferences; and based on shared responsibility between patient and caregivers for optimizing health.” The Patient Perceptions of Integrated Care Survey was developed to measure integrated care from the perspective of the adult patient, capturing the notions of continuity over time and alignment of efforts by professionals, patients, and family members across settings and systems. Although this definition is foundationally sound for child health care delivery, it is important to emphasize the fundamental dependence of child health outcomes on additional factors, including family functioning, and on the provision of both medical and nonmedical services, such as education, home care, and family support, across the care continuum.5 For children, integration results from coordinating the efforts of all providers, irrespective of institutional, agency, or community-based organizational boundaries.1 

A recent policy statement on Family-Centered Care Coordination by the American Academy of Pediatrics emphasizes aspects of care integration beyond the boundary of the primary care medical home and, indeed, across the broader community, including both medical and nonmedical services. It calls for implementation of community-wide CC strategies, making comprehensive measurement of the family experience of care integration more important than ever.1 The present article reports on the development and validation of the Pediatric Integrated Care Survey (PICS), an instrument that measures family-reported experiences of care integration.

Boston Children’s Hospital (BCH) Institutional Review Board approved all parts of this study (P00008144; P00007912). The Lucile Packard Children’s Hospital Institutional Review Board approved the study protocol through a reliance agreement.

A pilot survey of parent-reported experiences with pediatric care integration was developed with substantial input from parents. Between July and December 2013, six focus groups with a total of 36 parents and 4 additional semi-structured interviews were conducted with parents of children receiving care from multiple medical and nonmedical providers. Interviews were conducted in English except for 1 interview, which was conducted in Spanish using a language interpreter. The families were recruited through the BCH Department of Psychiatry, BCH Primary Care Center, and private, community-based, pediatric primary care practices. The Massachusetts Federation for Children with Special Needs also recruited families by sending out study information to their listserv. Families were eligible to participate if they reported having a child who, in the past 12 months, had had at least 1 primary care appointment and at least 2 specialty appointments (with ≥1 specialist), defined broadly as medical, surgical, behavioral specialties, and allied health professions. All participants were parents of CYSHCN with various medical, behavioral health, and educational needs; various institutional affiliations of their primary and specialty care providers; and varying degrees of dependence on allied health and durable medical equipment providers (Supplemental Table 5). Participants were asked to describe what integrated care meant to them, how they would characterize care that is integrated across all care team members, and how care should be integrated from the perspective of families. Thematic analyses of focus group transcripts resulted in a framework of family-centered care integration (Fig 1).

FIGURE 1

Framework demonstrating domains and elements of family-centered, integrated care.6 

FIGURE 1

Framework demonstrating domains and elements of family-centered, integrated care.6 

Close modal

Families reported that, essentially, integrated care is an approach that treats their children and youth “holistically” and which includes 3 domains: team-based care; connection to life/community; and the future (care planning). Team-based care encompasses elements of team composition, methods of efficient communication, and purposeful information sharing. The second domain, linkage to the broader community, was described by families as connection to aspects of “life” beyond medical care. Families expressed that assessment of both their strengths and needs is essential to the process by which appropriate resources are identified for them. Assisting families in accessing indicated medical and nonmedical resources is vital to achieving optimal outcomes. The third domain, the future, encompassed a formal, inclusive process of care planning. Families described the need for time-bound goals, taking into account the multiplicity of tactics and skills necessary to achieve both short- and long-term outcomes. Families and care team members must collaborate to prioritize goals and ensure that the necessary resources can be secured to achieve those goals. Underlying each of these elements is an expectation of accountability for organizing and coordinating care, with intentionality to improve care integration efforts and outcomes.

A first draft of the pilot survey was based on the Patient Perceptions of Integrated Care Survey, the data collected in the focus groups, and a literature review of existing parent-reported surveys on care experiences as well as measures of CC and care integration.6,12 In addition, an expert advisory panel provided input on the structure and content of the survey during the initial survey development. The 17 panelists were selected based on their experience and knowledge related to family advocacy, care of CYSHCN, health policy, quality measurement, CC, care integration, and survey methodology. Representatives of family organizations were recruited to ensure that the perspectives of families were included during all phases of this project.

Between December 2013 and February 2014, eight cognitive interviews were conducted to assess terminology and sentence structure, and to identify gaps and vulnerabilities in respondents’ ability to retrieve experiences from memory. Survey questions were revised based on the feedback.

The final pilot survey contained 95 questions, organized into 8 thematic sections (Supplemental Table 6). Responses for the experience items were generally 6-point Likert scales with the response options “never,” “rarely,” “sometimes,” “usually,” “almost always,” and “always.”

Participants for pilot testing of the survey were recruited between July 2014 and June 2015. Recruitment procedures and eligibility criteria for the pilot test were the same as for the focus groups. In-person recruitment took place at 4 sites: BCH; Lucile Packard Children’s Hospital, Palo Alto, California; and at state and regional meetings of the Massachusetts Federation for Children with Special Needs in 2014 and 2015. Four collaborating organizations (the Massachusetts Federation for Children with Special Needs, the Parent/Professional Advocacy League of Massachusetts, Westwood-Mansfield Pediatrics [a community-based primary care setting], and the Crotched Mountain School, Greenfield, New Hampshire) electronically sent recruitment leaflets to their families of CYSHCN.

All participants in the pilot study were invited to complete the PICS at least once. The 259 participants recruited before March 2015 were also asked to complete the PICS a second time ∼2 weeks after completing the first survey to assess test–retest reliability. BCH study staff sent out all surveys and conducted all follow-up activities.

PICS were administered either by paper through mail or electronically through Qualtrics (Provo, UT),13 depending on participant preference. Each participant received up to 3 reminder e-mails and/or 2 reminder telephone calls if they had not completed the survey. Monetary stipends were sent for each completed survey.

All statistical analyses were performed by using Stata version 14.0 (Stata Corp, College Station, TX).13 Item-level descriptive statistics were evaluated by using frequencies, percentages, and univariate measures of location and variability. Exploratory factor analysis (EFA) was used to determine which of the experience items should be included in the final questionnaire and the underlying measurement domains each item represented. Psychometric properties of the final questionnaire were evaluated by using tests of construct validity, internal consistency, and test–retest reliability.14,15 

The experience items in the EFA for the construction of the core instrument needed to have a minimum sample size to allow for the factor analytic statistics to be estimated. Because answers to a number of items, including experience questions relating to school, transition/transfer to adult care, and care planning, were dependent on applicability of the screening questions, only experience items that were reportedly applicable to at least two thirds of the respondents were included in the EFA. Furthermore, items with >10% of respondents choosing either “don’t know” or “not applicable” were also excluded from the factor analysis. However, “don’t know” responses of experience items for which this option represented a lack of awareness of communication or information sharing were recoded as the lowest Likert scale response option. The justification for this choice was that the focus groups described awareness of these activities of communication and information sharing as being important to family experience of care integration. The 26 remaining items were then eligible to be included in the EFA.

There were no significant differences between respondents using electronic versus paper survey results (Supplemental Table 7).

Exploratory Factor Analysis

Principal factor analysis with varimax rotation was used to explore the structure underlying the 26 experience items. Factor loadings guided which items measured the same underlying construct. The inclusion or exclusion of an item in a construct was determined iteratively by examining factor loadings (>0.3) and Cronbach’s α to identify redundant items or items that did not sufficiently measure the same underlying construct.

Composite Score Calculation

The composite score calculation method followed the proportional scoring method used for Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys. Detailed instructions can be found in the Supplemental Materials “PICS Composite Score Calculation.”

Reliability

Internal consistency of the items within each measurement domain was tested by using Cronbach’s α. Values of α ≥0.7 were acceptable. For demonstration of discriminant validity, pairwise Spearman rank correlation coefficients between each item, their own composite, and all other composites were examined. Evidence of internal consistency between an item and its own composite scale was defined as a correlation coefficient ≥0.40. Correlations between an item and the other composite scales should, on average, be lower than between the item and its own composite scale.

For test–retest reliability, a weighted Cohen’s κ was used as an indicator for agreement between the individual items assessed by the same respondent measured at the 2 time points. Values >0.80 represent almost perfect agreement, those between 0.61 and 0.80 substantial agreement, values between 0.41 and 0.60 moderate agreement, and those <0.41 fair agreement.16 

Validity

Construct validity tests were conducted by using known group comparisons. Composite measures should be able to discriminate between groups that are known to be different with regard to at least some of the constructs measured by using the survey tool. Children with characteristics that predict increased health care needs and usage are more likely to exhibit fragmented care2 and therefore should have, on average, lower scores on the constructs. We used t tests adjusting for unequal variances to compare the mean composite scores between the identified groups.

In total, 442 parents and guardians agreed to receive the survey. The majority preferred to receive the survey electronically (n = 326 [73.8%]) compared with mail (n = 116 [26.2%]). Of those sent, 255 surveys were returned, for a 57.7% response rate; responses were higher for Web mode than for mail administration (electronic: n = 198 [60.7%]; mail: n = 57 [49.6%]; P = .04).

The first 259 participants were also invited to complete the PICS a second time after having returned the first survey. From this group of participants, 145 returned the first survey (56.6%) and of those, 81.3% returned the second survey (n = 118). There were, however, no significant differences between how participants returned the second survey (electronic: 84 of 103 [81.6%]; mail: 34 of 42 [81.0%]; P = .93).

Table 1 displays the demographic characteristics of the respondents, households, child, and the child’s health care needs. The majority of the respondents were female (92.1%) and had at least a 4-year college degree (63.7%). Nearly all children were reported to have a primary care provider (96.5%). Two-thirds of children were seen by medical or surgical specialists (65.1%) and/or allied health professionals, such as occupational, physical, and speech therapists (65.5%). Approximately one-half of the children used school care providers, such as school nurse or school counselor, and/or behavioral health care providers (53.3% and 45.1%, respectively). During PICS development, families reported that their experience of patient- and family-centered care integration was critically dependent on the designation of all professionals as “care providers” who offer services to their children. This label includes professionals from multiple disciplines including, but not limited to, medical, surgical, behavioral health, allied health, and educational providers. Therefore, the instrument encouraged families to define care provider as anyone whom they perceived to have delivered services to their child in the previous 12 months, with the expectation of having an impact on the child’s health.

TABLE 1

Demographic Characteristics of Survey Pilot Test Respondents

CharacteristicN%
Respondent characteristics 
 Female (n = 252) 232 92.1 
 Education (n = 253)   
  Some high school 2.8 
  High school graduate 24 9.5 
  Some college 61 24.1 
  4-y college graduate 69 27.3 
  More than 4-y college 92 36.4 
Household characteristics 
 No. of adults in household, including respondent (n = 250)   
  1 29 11.6 
  2 166 66.4 
  3 36 14.4 
  4 15 6.0 
  ≥5 1.6 
 No. of children aged <18 y in household, including child the survey asks about (n = 252)   
  0 18 7.1 
  1 84 33.3 
  2 99 39.3 
  3 43 17.1 
  4 2.4 
  ≥5 0.8 
 No. of children with special health care needs in household (n = 252)   
  0 170 67.5 
  1 62 24.6 
  2 16 6.4 
  3 1.2 
  4 0.4 
  ≥5 
Child characteristics 
 Female (n = 252) 120 47.6 
 Age (n = 252)   
  Infant (aged <1 y) 13 5.2 
  1–3 y 45 17.9 
  4–12 y 107 42.5 
  13–17 y 63 25.0 
  ≥18 y  24 9.5 
 Race (n = 251)   
  White 212 84.5 
  African American/black 17 6.8 
  Asian 2.4 
  Native American/Alaskan Native 1.2 
  Native Hawaiian/Pacific Islander 
  Other 13 5.2 
 Hispanic (n = 252) 29 11.5 
 Insurance (n = 250)   
  Medicare/Medicaid 63 25.2 
  Private/commercial 181 72.4 
  No insurance 0.4 
  Don’t know 2.0 
Health care needs of child 
 No. of health care providers contributing to child’s care (n = 253)   
  2–5 111 43.9 
  6–10 91 36.0 
  11–15 26 10.3 
  16–20 11 4.4 
  >20 14 5.5 
 Types of health care provider caring for childa (n = 255)   
  Primary care provider 246 96.5 
  Specialists 166 65.1 
  Home health 54 21.2 
  School health 136 53.3 
  Behavioral health 115 45.1 
  Therapists 167 65.5 
  Social work 57 22.5 
  Alternative medicine 24 9.4 
  Other types 35 13.7 
 No. of emergency department visits (n = 255)   
  Never 124 48.6 
  Once or twice 89 34.9 
  3–5 34 13.3 
  ≥6 3.1 
 Special education service (n = 203) 133 65.5 
 Early education service (n = 50) 39 78.0 
 Medical conditionsa (n = 253)   
  None 25 9.9 
  Behavioral conditions 120 47.4 
  Developmental conditions 117 46.3 
  Breathing conditions 62 24.5 
  Neurologic conditions 138 54.6 
 Child had usual place of care (n = 255) 247 96.9 
 Use of prescription medicine (n = 254) 224 88.2 
 Use of special therapy (n = 254) 181 71.3 
 Use of counseling (n = 255) 160 62.8 
 Use of medical equipment (n = 255) 98 38.4 
CharacteristicN%
Respondent characteristics 
 Female (n = 252) 232 92.1 
 Education (n = 253)   
  Some high school 2.8 
  High school graduate 24 9.5 
  Some college 61 24.1 
  4-y college graduate 69 27.3 
  More than 4-y college 92 36.4 
Household characteristics 
 No. of adults in household, including respondent (n = 250)   
  1 29 11.6 
  2 166 66.4 
  3 36 14.4 
  4 15 6.0 
  ≥5 1.6 
 No. of children aged <18 y in household, including child the survey asks about (n = 252)   
  0 18 7.1 
  1 84 33.3 
  2 99 39.3 
  3 43 17.1 
  4 2.4 
  ≥5 0.8 
 No. of children with special health care needs in household (n = 252)   
  0 170 67.5 
  1 62 24.6 
  2 16 6.4 
  3 1.2 
  4 0.4 
  ≥5 
Child characteristics 
 Female (n = 252) 120 47.6 
 Age (n = 252)   
  Infant (aged <1 y) 13 5.2 
  1–3 y 45 17.9 
  4–12 y 107 42.5 
  13–17 y 63 25.0 
  ≥18 y  24 9.5 
 Race (n = 251)   
  White 212 84.5 
  African American/black 17 6.8 
  Asian 2.4 
  Native American/Alaskan Native 1.2 
  Native Hawaiian/Pacific Islander 
  Other 13 5.2 
 Hispanic (n = 252) 29 11.5 
 Insurance (n = 250)   
  Medicare/Medicaid 63 25.2 
  Private/commercial 181 72.4 
  No insurance 0.4 
  Don’t know 2.0 
Health care needs of child 
 No. of health care providers contributing to child’s care (n = 253)   
  2–5 111 43.9 
  6–10 91 36.0 
  11–15 26 10.3 
  16–20 11 4.4 
  >20 14 5.5 
 Types of health care provider caring for childa (n = 255)   
  Primary care provider 246 96.5 
  Specialists 166 65.1 
  Home health 54 21.2 
  School health 136 53.3 
  Behavioral health 115 45.1 
  Therapists 167 65.5 
  Social work 57 22.5 
  Alternative medicine 24 9.4 
  Other types 35 13.7 
 No. of emergency department visits (n = 255)   
  Never 124 48.6 
  Once or twice 89 34.9 
  3–5 34 13.3 
  ≥6 3.1 
 Special education service (n = 203) 133 65.5 
 Early education service (n = 50) 39 78.0 
 Medical conditionsa (n = 253)   
  None 25 9.9 
  Behavioral conditions 120 47.4 
  Developmental conditions 117 46.3 
  Breathing conditions 62 24.5 
  Neurologic conditions 138 54.6 
 Child had usual place of care (n = 255) 247 96.9 
 Use of prescription medicine (n = 254) 224 88.2 
 Use of special therapy (n = 254) 181 71.3 
 Use of counseling (n = 255) 160 62.8 
 Use of medical equipment (n = 255) 98 38.4 
a

Multiple selections possible.

Care teams ranged from 2 to 5 care providers (43.9%) or 6 to 10 care providers (36.0%). Nearly two-thirds of the families reported their providers were part of the same health system (62.0%). Respondent, household, and child characteristics did not differ across the recruitment sites (Supplemental Table 8). Significant differences were, however, found with regard to the size of the health care teams, the types of health care providers caring for the child, the type of medical conditions, and the use of services such as counseling, medical equipment, special therapy, and special education services. There were no significant differences in experience items or the composite mean scores as a function of survey administration mode (Supplemental Table 7).

Exploratory Factor Analysis

A 5-factor model was chosen as the best analytic solution based on the scree plot of the 19 experience items, the factor loadings, parallel analysis (Supplemental Fig 2), and conceptual knowledge from the focus groups (Supplemental Tables 14-17 for models with factors 1–4). Table 2 displays the descriptive statistics for each item according to constructs. Cronbach’s α (Table 3) and the factor loadings excluded 7 of the 26 initial items from the core instrument. The factor loadings, the uniqueness of each item, and the proportion of the common variance of the item not associated with the factors, are shown in Supplemental Table 9. The 5 constructs underlying the 19-item core instrument are:

TABLE 2

Item Descriptive Statistics and Test–Retest Reliability

FactorNResponse Category number% Responses in Bottom Boxa% Responses in Bottom 2 Boxesb25% PercentileMean ± SDMedian75% Percentile% Response in Top 2 Boxesc% Responses in Top Boxdκe
Factor 1: access to care            
 Delays: lack of sufficient servicesf 255 1.6 10.2 4.4 ± 1.26 49.0 23.1 0.61 
 Delays: lack of informationf 255 1.6 3.5 5.1 ± 1.10 76.5 46.3 0.47 
Factor 2: communication with CTMs            
 CTMs explained things in understandable manner 255 0.5 0.8 5.2 ± 0.90 77.6 40.8 0.58 
 Family comfortable voicing concerns with CTMs 246 1.2 2.0 5.4 ± 1.03 83.3 63.4 0.55 
 CTMs listened carefully to what family had to say 255 0.8 1.2 5.0 ± 0.99 74.5 39.2 0.70 
 CTMs treated family as a full partner 253 1.2 2.8 5.2 ± 1.12 76.3 52.6 0.64 
Factor 3: family impact            
 CTMs discussed things that cause family stress 252 40.1 66.7 2.2 ± 1.34 7.5 3.6 0.50 
 CTMs discussed things that made it hard to care for child 253 45.1 70.4 2.1 ± 1.29 6.3 3.2 0.62 
 CTMs took the whole family into account 253 18.6 40.7 3.5 ± 1.84 36.0 21.3 0.59 
 CTMs offered other than in-person communication 255 20.8 31.0 3.5 ± 1.77 32.9 18.8 0.52 
 CTMs offered peer connections 254 60.6 76.4 1.9 ± 1.34 7.5 3.9 0.42 
Factor 4: care goal creation/planning            
 CTMs created short-term care goals 251 51.8 NA 0.5 ± 0.50 NA 48.2 0.53 
 CTMs created long-term care goals 252 63.5 NA 0.4 ± 0.48 NA 36.5 0.49 
Factor 5: team functioning/quality            
 CTMs knew about advice from other CTMs 255 5.1 19.2 3.9 ± 1.49 38.8 19.2 0.65 
 CTMs assigned and explained responsibility 253 20.6 37.2 3.4 ± 1.78 34.4 16.2 0.63 
 CTMs aware of tests and evaluations 255 9.4 19.6 4.1 ± 1.64 47.1 27.1 0.65 
 CTMs followed through on responsibilities 255 2.8 4.3 4.8 ± 1.23 69.0 36.5 0.65 
 CTMs considered “big picture” 254 3.9 14.6 4.1 ± 1.45 46.9 22.4 0.68 
 CTMs had access to same medical record 254 38.2 NA 0.6 ± 0.49 NA 61.8 0.62 
FactorNResponse Category number% Responses in Bottom Boxa% Responses in Bottom 2 Boxesb25% PercentileMean ± SDMedian75% Percentile% Response in Top 2 Boxesc% Responses in Top Boxdκe
Factor 1: access to care            
 Delays: lack of sufficient servicesf 255 1.6 10.2 4.4 ± 1.26 49.0 23.1 0.61 
 Delays: lack of informationf 255 1.6 3.5 5.1 ± 1.10 76.5 46.3 0.47 
Factor 2: communication with CTMs            
 CTMs explained things in understandable manner 255 0.5 0.8 5.2 ± 0.90 77.6 40.8 0.58 
 Family comfortable voicing concerns with CTMs 246 1.2 2.0 5.4 ± 1.03 83.3 63.4 0.55 
 CTMs listened carefully to what family had to say 255 0.8 1.2 5.0 ± 0.99 74.5 39.2 0.70 
 CTMs treated family as a full partner 253 1.2 2.8 5.2 ± 1.12 76.3 52.6 0.64 
Factor 3: family impact            
 CTMs discussed things that cause family stress 252 40.1 66.7 2.2 ± 1.34 7.5 3.6 0.50 
 CTMs discussed things that made it hard to care for child 253 45.1 70.4 2.1 ± 1.29 6.3 3.2 0.62 
 CTMs took the whole family into account 253 18.6 40.7 3.5 ± 1.84 36.0 21.3 0.59 
 CTMs offered other than in-person communication 255 20.8 31.0 3.5 ± 1.77 32.9 18.8 0.52 
 CTMs offered peer connections 254 60.6 76.4 1.9 ± 1.34 7.5 3.9 0.42 
Factor 4: care goal creation/planning            
 CTMs created short-term care goals 251 51.8 NA 0.5 ± 0.50 NA 48.2 0.53 
 CTMs created long-term care goals 252 63.5 NA 0.4 ± 0.48 NA 36.5 0.49 
Factor 5: team functioning/quality            
 CTMs knew about advice from other CTMs 255 5.1 19.2 3.9 ± 1.49 38.8 19.2 0.65 
 CTMs assigned and explained responsibility 253 20.6 37.2 3.4 ± 1.78 34.4 16.2 0.63 
 CTMs aware of tests and evaluations 255 9.4 19.6 4.1 ± 1.64 47.1 27.1 0.65 
 CTMs followed through on responsibilities 255 2.8 4.3 4.8 ± 1.23 69.0 36.5 0.65 
 CTMs considered “big picture” 254 3.9 14.6 4.1 ± 1.45 46.9 22.4 0.68 
 CTMs had access to same medical record 254 38.2 NA 0.6 ± 0.49 NA 61.8 0.62 

CTM, care team member; NA, not applicable.

a

Bottom box: Least favorable response option.

b

Bottom 2 boxes: 2 least favorable response options.

c

Top 2 boxes: 2 most favorable response options.

d

Top box: Most favorable response option.

e

Sample size for test–retest reliability, n = 118.

f

This item is reverse coded; thus, higher values indicate better performance.

TABLE 3

Item-to-Composite Scale Correlations and Internal Consistency Reliability

CompositeItem-to-Own Composite Scale CorrelationsaItem-to-Other Composite Scale Correlationsaα
MeanRange
Composite 1: access to care    0.69 
 Delays: lack of sufficient servicesb 0.85 0.31 0.11 to 0.47 
 Delays: lack of informationb 0.65 0.27 0.12 to 0.38 
Composite 2: communication with CTMs    0.80 
 CTMs explained things in understandable manner 0.68 0.37 0.16-0.55 
 Family comfortable voicing concerns with CTMs 0.56 0.25 0.16 to 0.38 
 CTMs listened carefully to what family had to say 0.72 0.38 0.12 to 0.55 
 CTMs treated family as a full partner 0.70 0.38 0.19 to 0.56 
Composite 3: family impact    0.72 
 CTMs discussed things that cause family stress 0.33 0.13 −0.03 to 0.23 
 CTMs discussed things that made it hard to care for child 0.28 0.12 −0.05 to 0.21 
 CTMs took the whole family into account 0.75 0.38 0.26 to 0.55 
 CTMs offered other than in-person communication 0.63 0.27 0.15 to 0.37 
 CTMs offered peer connections 0.30 0.13 0.02 to 0.23 
Composite 4: care goal creation/planning    0.64 
 CTMs created short-term care goals 0.89 0.13 0.05 to 0.19 
 CTMs created long-term care goals 0.84 0.18 0.09 to 0.28 
Composite 5: team functioning/quality    0.84 
 CTMs knew about advice from other CTMs 0.78 0.39 0.23 to 0.52 
 CTMs assigned and explained responsibility 0.71 0.40 0.31 to 0.54 
 CTMs aware of tests and evaluations 0.73 0.31 0.23 to 0.52 
 CTMs followed through on responsibilities 0.71 0.40 0.23 to 0.52 
 CTMs considered “big picture” 0.78 0.45 0.27 to 0.56 
 CTMs had access to same medical record 0.60 0.24 0.10 to 0.32 
CompositeItem-to-Own Composite Scale CorrelationsaItem-to-Other Composite Scale Correlationsaα
MeanRange
Composite 1: access to care    0.69 
 Delays: lack of sufficient servicesb 0.85 0.31 0.11 to 0.47 
 Delays: lack of informationb 0.65 0.27 0.12 to 0.38 
Composite 2: communication with CTMs    0.80 
 CTMs explained things in understandable manner 0.68 0.37 0.16-0.55 
 Family comfortable voicing concerns with CTMs 0.56 0.25 0.16 to 0.38 
 CTMs listened carefully to what family had to say 0.72 0.38 0.12 to 0.55 
 CTMs treated family as a full partner 0.70 0.38 0.19 to 0.56 
Composite 3: family impact    0.72 
 CTMs discussed things that cause family stress 0.33 0.13 −0.03 to 0.23 
 CTMs discussed things that made it hard to care for child 0.28 0.12 −0.05 to 0.21 
 CTMs took the whole family into account 0.75 0.38 0.26 to 0.55 
 CTMs offered other than in-person communication 0.63 0.27 0.15 to 0.37 
 CTMs offered peer connections 0.30 0.13 0.02 to 0.23 
Composite 4: care goal creation/planning    0.64 
 CTMs created short-term care goals 0.89 0.13 0.05 to 0.19 
 CTMs created long-term care goals 0.84 0.18 0.09 to 0.28 
Composite 5: team functioning/quality    0.84 
 CTMs knew about advice from other CTMs 0.78 0.39 0.23 to 0.52 
 CTMs assigned and explained responsibility 0.71 0.40 0.31 to 0.54 
 CTMs aware of tests and evaluations 0.73 0.31 0.23 to 0.52 
 CTMs followed through on responsibilities 0.71 0.40 0.23 to 0.52 
 CTMs considered “big picture” 0.78 0.45 0.27 to 0.56 
 CTMs had access to same medical record 0.60 0.24 0.10 to 0.32 

CTM, care team member.

a

All correlations are significant at P < .05.

b

This item is reverse coded so that higher values indicate better performance.

  1. Access: access to care (2 items);

  2. Communication: communication between care team members and parents/guardians (4 items);

  3. Family impact: assessment and remediation of the impact the child’s health care needs have on the family (5 items);

  4. Care goal creation: creation of short- and long-term care goals (2 items); and

  5. Team functioning: team functioning and quality of the care team (6 items).

Except for 2 items in the “family impact” domain, all items showed factor loadings ≥0.5 on their respective factor. The Flesch-Kincaid Reading Ease Score of the final core instrument, the demographic background questions, and the health care characteristics and utilization questions was calculated to be 70.5 or a grade level of 8.6.17 

Reliability

Cronbach’s α values were close to or >0.7 for 4 of the 5 constructs (Table 3). Internal consistency was also established through strong correlation coefficients with values ≥0.4 between items and their own composite for 4 of the 5 constructs. Three family impact items showed moderate or weak correlation coefficients (≤0.33). However, the range and mean of the correlation coefficients between items and the other composite scales were always lower than those between the items and their own composite scale, demonstrating strong discriminant validity.

With regard to test–retest reliability, 10 items agreed substantially between the 2 survey administrations, with κ values >0.60. The other 9 items showed moderate agreement, with κ values between 0.41 and 0.60 (Table 3).16 

Respondents were expected to have, on average, lower mean composite scores if they had the following: a child with increased health care needs because of ≥2 types of medical conditions; those with higher health care usage such as services from allied health professionals, counseling, medical equipment, and additional services; those with >2 health care providers or >1 medical specialist; those reported by parents to have needed more care than usual; and those reported by parents to have changing care needs. The distribution of the composite scores for each construct is shown in Table 4. Supplemental Table 10 presents the composite-to-composite correlation.

TABLE 4

Descriptive Statistics of Composite Scales (n = 255)

CompositeMinimum25% PercentileMean ± SDMedian75% PercentileMaximum
Composite 1: access to care 50 62.7 ± 39.0 50 100 100 
Composite 2: communication with care team members 50 78.1 ± 30.5 100 100 100 
Composite 3: family impact 18.3 ± 23.9 20 20 100 
Composite 4: care goal creation/planning 42.4 ± 42.4 50 100 100 
Composite 5: team functioning/quality 16.7 49.7 ± 34.4 50 83.3 100 
CompositeMinimum25% PercentileMean ± SDMedian75% PercentileMaximum
Composite 1: access to care 50 62.7 ± 39.0 50 100 100 
Composite 2: communication with care team members 50 78.1 ± 30.5 100 100 100 
Composite 3: family impact 18.3 ± 23.9 20 20 100 
Composite 4: care goal creation/planning 42.4 ± 42.4 50 100 100 
Composite 5: team functioning/quality 16.7 49.7 ± 34.4 50 83.3 100 

Construct validity was established by assessing the differences in mean composite scores between groups known to be different with regard to at least 1 of the 5 constructs. Questions from the PICS about health care needs and usage were used to differentiate the respondents into groups. If a child’s health care providers all have access to the same medical record, this access should, in theory, facilitate information-sharing, CC, and integration, and therefore yield, on average, higher composite scores. Supplemental Table 11 generally confirms these expectations. Nearly all health care needs and usage indicators show lower composite scores for those indicating higher needs or usage. Although the differences were significant for the composite scales access and team functioning, only about one-half of the indicators for the composite scales of communication and family impact reached statistical significance. However, differences in the mean scores were directionally as expected. If all health care providers had access to the same medical records, composite scores were, as expected, significantly higher for these 4 domains. For the composite scale of care goal creation, none of the groups exhibited significant differences, and the direction of the composite score means was not always as expected. Having services from allied health professionals, medical equipment, and using >2 health care providers showed increased mean scores. Parents of children using behavioral health care showed significantly lower mean scores on all composites and most individual items (Supplemental Table 12). Care received from providers within 1 health system was rated significantly higher with regard to 4 composites and most of the related items. The only composite showing no difference between the health care systems was care goal creation. Insurance type did not show any significant differences for composite and most individual item ratings.

Although frameworks for care coordination1,3,18 and care integration4 have been described, assessing care integration as a patient- or family-reported experience measure has yet to be adopted as a critical outcome of delivery systems taking on accountability for both quality and cost.19,20 

Due to the number of providers involved in their care, CYSHCN are at risk for fragmented care. The results of the family focus groups in the present study shed light on key aspects of care integration for this population and vulnerabilities that may result in poor integration. Focus groups reported that families themselves were most likely to be the care integrator for children and youth with multiple chronic conditions because these children often do not fit within the purview of any single, traditional clinical care model. In addition, families whose children have significant behavioral health needs, as well as those with unknown diagnoses, were often left on their own to integrate care across providers.

The PICS offers an instrument to complement both the Family Experience with Coordination of Care Survey measure (a tool for assessing families’ experience of care coordination for their children with medical complexity)21 and the Child CAHPS surveys.22 The Family Experience with Coordination of Care Survey measures family experience of CC at a tactical level, such as in response to providing access to a care coordinator or the availability of a care plan. The Child CAHPS surveys are important measures of care experiences received by children, focusing on interactions with specific care providers but not necessarily how the providers integrate their efforts in addressing the holistic needs of the child and family. In contrast, the PICS assesses a family’s experience of care integration across a team of providers, irrespective of institutional affiliation, the discipline of the provider, the type of intervention, or the location of care. As such, the PICS may prove valuable for those stakeholders who are approaching care redesign more broadly. Although the PICS can measure the impact of the implementation of a care plan, it may also be effective in assessing the family experience outcome of a broad array of interventions, such as the implementation of innovative collaborative care models. These models aim to transform how primary and specialty care providers collaborate, and early experience has been favorable with conditions such as constipation,23 behavioral health,24 and obesity.25 Furthermore, because the PICS enables the family to identify all members of a multidisciplinary team, it can be used to evaluate care integration between medical, educational, behavioral, and social service providers.

The PICS can be used to assess family experience of care integration within existing care delivery models and systems and therefore guide quality improvement efforts in those settings. Given that medically complex children are increasingly identified as a key population for care management, the PICS can be used to assess baseline integration experience, as new care models are designed and implemented. Due to its ability to assess integration across disciplines (eg, medical, behavioral, educational, family support), the PICS can be used to identify gaps in care delivery. As delivery systems begin to take on accountability for populations with medical and social determinant risk factors, the PICS might serve a critical role in evaluating broader care integration needs.

The PICS should be viewed in light of its limitations. Although we received family input into the design of the PICS instrument from families in California and Massachusetts, to date it has only been validated for families living in the Boston area. In addition, our family focus groups suggested a number of tactics and tools that they felt would improve their perception of care integration, such as care planning. However, in our pilot survey, the number of families who reported having experience with care planning was sufficiently small that we were not able to include them in the EFA. Despite this limitation, we have chosen to honor the wisdom of our family collaborators by retaining items that families felt would be of high potential value as additional modules, even if not yet mapping to a psychometric domain. The final PICS includes 26 rating items and 29 demographic and background questions that can be selectively included depending on how much information is already known about the respondent’s child. Our expectation is that further implementation of the PICS will enable us to gather the necessary data to expand the validated core set of questions. We have translated the PICS into Spanish, but its psychometric properties have not yet been determined.

Future research should focus on assessing the predictive validity of PICS that could not be tested in this study because collecting outcome measures beyond parent-reported experience of care integration was beyond the scope of the project. Correlational analyses with other existing instruments measuring the quality of care experiences should also be undertaken to assess concurrent validity. The inclusion of additional items that could not be included in the psychometric analyses in this study due to low prevalence in traditional health care, such as existence of a written care plan, might also further improve the psychometric characteristics of PICS. Administration of PICS in other settings and subsequent confirmatory factor analysis is therefore recommended. Given the nascent state of care integration, PICS is expected to evolve to meet the performance evaluation needs of a transforming health care ecosystem, as family experience of care integration justifiably emerges as a normative outcome.

The PICS is a validated instrument designed to measure parent-reported experience of care integration for children and youth who, in the past 12 months, had 1 primary care encounter and at least 2 specialty encounters. It was designed to broadly assess family experience across the care continuum. As a result of its strong psychometric properties, the PICS may be used to inform quality improvement efforts within and across delivery systems and communities, reflecting families’ experience of care among medical and nonmedical providers, irrespective of organizational affiliations. Broad adoption of the PICS by delivery systems will depend on additional research to assess implementation experience. Using this measure of authentic, family-reported care integration may serve to advance performance measurement of the patient and family experience domain of the Institute for Healthcare Improvement's Triple Aim.26 

Dr Ziniel conceptualized and designed the study, conducted all statistical analyses, and drafted the majority of the initial manuscript; Ms Rosenberg coordinated and supervised data collection during all phases of the project, drafted parts of the initial manuscript, and critically reviewed the manuscript; Ms Bach coordinated and collected data during the pilot phase of the project, drafted parts of the initial manuscript, and critically reviewed the manuscript; Dr Singer conceptualized and designed the study and critically reviewed the manuscript; Dr Antonelli conceptualized and designed the study, drafted parts of the initial manuscript, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

BCH

Boston Children’s Hospital

CAHPS

Consumer Assessment of Healthcare Providers and Systems

CYSHCN

children and youth with special health care needs

EFA

exploratory factor analysis

CC

care coordination

PICS

Pediatric Integrated Care Survey

The authors are grateful to Dr Edward Schor, Senior Vice President at the Lucile Packard Foundation for Children’s Health, for his support, inspiration, and guidance throughout this project. They acknowledge the contributions of the expert advisors, family and community groups, health care professionals, and families who have contributed to the development and validation of the PICS. The authors especially thank Ms Cathy Hickey and Mr Rich Robison at the Massachusetts Federation for Children with Special Needs for their collaboration throughout this project.

1
Council on Children with Disabilities and Medical Home Implementation Project Advisory Committee
.
Patient- and family-centered care coordination: a framework for integrating care for children and youth across multiple systems.
Pediatrics
.
2014
;
133
(
5
). Available at: www.pediatrics.org/cgi/content/full/133/5/e1451
[PubMed]
2
US Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau
. The National Survey of Children with Special Health Care Needs Chartbook 2005–2006. 2008. Available at: http://mchb.hrsa.gov/cshcn05. Accessed March 8, 2016
3
Antonelli
RC
,
McAllister
JW
,
Popp
J
. Making care coordination a critical component of the pediatric health system: a multidisciplinary framework. The Commonwealth Fund. Available at: www.commonwealthfund.org/publications/fund-reports/2009/may/making-care-coordination-a-critical-component-of-the-pediatric-health-system. Accessed March 8, 2016
4
Singer
SJ
,
Burgers
J
,
Friedberg
M
,
Rosenthal
MB
,
Leape
L
,
Schneider
E
.
Defining and measuring integrated patient care: promoting the next frontier in health care delivery.
Med Care Res Rev
.
2011
;
68
(
1
):
112
127
[PubMed]
5
Stille
C
,
Turchi
RM
,
Antonelli
R
, et al;
Academic Pediatric Association Task Force on Family-Centered Medical Home
.
The family-centered medical home: specific considerations for child health research and policy.
Acad Pediatr
.
2010
;
10
(
4
):
211
217
[PubMed]
6
Qualtrics [computer program]. Provo, UT: Qualtrics;
2015
7
Singer
SJ
.
Patient Perceptions of Integrated Care Survey
. Available at: www.integratedpatientcare.org/the-survey.html. Accessed March 8,
2016
8
National Center for Family Professional Partnerships
. Family-centered care assessment for families. Available at: www.fv-ncfpp.org/activities/fcca/. Accessed March 8,
2016
9
Data Resource Center for Child and Adolescent Health
. National Survey of Children with Special Health Care Needs, 2009-2010 Survey. Available at: www.childhealthdata.org/learn/NS-CSHCN. Accessed March 8, 2016
10
Ziniel
SI
,
Connor
JA
,
Graham
D
, et al
.
Development and psychometric characteristics of the pediatric inpatient experience survey (PIES).
Int J Qual Health Care
.
2016
;
28
(
2
):
191
199
[PubMed]
11
McDonald
KM
,
Schultz
E
,
Albin
L
, et al;
Agency for Healthcare Research and Quality
. Care Coordination Measures Atlas. AHRQ Publication No. 14-0373-EF.
2011
. Available at: www.ahrq.gov/sites/default/files/publications/files/ccm_atlas.pdf. Accessed March 8, 2016
12
Rosenberg
H
,
Ziniel
S
,
Singer
S
,
Antonelli
R
.
What does integrated care mean to you and who is integrating It? A qualitative study of families with children seeing multiple health care providers.
Poster presented at:
Pediatric Academic Societies Annual Meeting
; May
3-6, 2014
;
Vancouver, British Columbia, Canada
13
STATA [computer program]. Release 13. College Station, TX: StataCorp LP;
2012
14
Streiner
D
,
Norman
G
.
Health Measurement Scales: a Practical Guide to Their Development and Use
.
New York, NY
:
Oxford University Press
;
2008
15
Raykov
T
,
Marcoulides
G
.
Introduction to Psychometric Theory
.
New York, NY
:
Routledge, Taylor and Francis Group
;
2011
16
Landis
JR
,
Koch
GG
.
The measurement of observer agreement for categorical data.
Biometrics
.
1977
;
33
(
1
):
159
174
[PubMed]
17
Readability Score
. Improve your writing! Available at: https:// readability-score.com. Accessed July 8, 2016
18
Clark
J
,
Singer
S
,
Kane
N
,
Valentine
M
.
From striving to thriving: systems thinking, strategy, and the performance of safety net hospitals.
Health Care Manage Rev
.
2013
;
38
(
3
):
211
223
[PubMed]
19
Singer
S
,
Shortell
SM
.
Implementing accountable care organizations: ten potential mistakes and how to learn from them.
JAMA
.
2011
;
306
(
7
):
758
759
[PubMed]
20
Shortell
SM
,
Singer
SJ
.
Improving patient safety by taking systems seriously.
JAMA
.
2008
;
299
(
4
):
445
447
[PubMed]
21
Seattle Children’s Hospital Research Foundation, Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN)
. Family Experiences with Coordination of Care (FECC) measurement set. Available at: www.seattlechildrens.org/research/child-health-behavior-and-development/mangione-smith-lab/measurement-tools/. Accessed March 8, 2016
22
US Department of Health and Human Services, Agency for Healthcare Research and Quality
. Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey. 2012. Available at: www.ahrq.gov/cahps/surveys-guidance/hp/instructions/version4.html. Accessed July 8, 2016
23
Mallon
D
,
Vernacchio
L
,
Trudell
E
, et al
.
Shared care: a quality improvement initiative to optimize primary care management of constipation.
Pediatrics
.
2015
;
135
(
5
). Available at: www.pediatrics.org/cgi/content/full/135/5/e1300
[PubMed]
24
Asarnow
JR
,
Rozenman
M
,
Wiblin
J
,
Zeltzer
L
.
Integrated medical-behavioral care compared with usual primary care for child and adolescent behavioral health: a meta-analysis.
JAMA Pediatr
.
2015
;
169
(
10
):
929
937
[PubMed]
25
Ebbeling
CB
,
Antonelli
RC
.
Primary care interventions for pediatric obesity: need for an integrated approach.
Pediatrics
.
2015
;
135
(
4
):
757
758
[PubMed]
26
Berwick
DM
,
Nolan
TW
,
Whittington
J
.
The triple aim: care, health, and cost.
Health Aff (Millwood)
.
2008
;
27
(
3
):
759
769
[PubMed]

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (https://creativecommons.org/licenses/by-nc-nd/4.0/), which permits noncommercial distribution and reproduction in any medium, provided the original author and source are credited.