Development of the Pediatric Hospitalization Admission Survey of Experience (PHASE) Measure

We undertook a 3-phase design to develop and establish validity evidence for the Pediatric Hospital Admission Survey of Experience (PHASE) measure (Fig 1). In phase 1, we developed an initial survey with 33 preliminary items. In phase 2, we undertook psychometric testing and instrument refinement to develop a preliminary quality measure. In phase 3, we performed confirmatory testing including a confirmatory factor analysis and additional validity testing.

The survey was administered as part of a stepped-wedge clinic-randomized trial comparing the effectiveness of pediatric direct admissions to emergency department admissions for hospitalized children.¹⁰  The trial, currently ongoing, involves 70 primary and urgent care clinics in 3 health systems in the Northeast, Midwest, and West Coast of the United States. Two of the sites are referral children’s hospitals and the third is a community hospital; 34 clinics were randomized to participate at 1 site with 18 clinics randomized at each of the other sites. The study was reviewed, approved, and monitored by the Dartmouth College Committee for the Protection of Human Subjects. All sites developed institutional review board authorization agreements with Dartmouth to serve as the single institutional review board of record.

Survey development took place from October 2019 through January 2020, following Gehlbach’s 7 steps for survey development.¹¹  Initially, we reviewed 3 hospital-based surveys with established validity evidence, including the adult-oriented Consumer Quality Index (CQI admission module, CQI Accident and Emergency, CQI General Practice, and CQI in the Dutch Social Insurance System), HCAHPS, and Child HCAHPS.^8,9,12–16  From these, we identified 77 items that mapped to 5 domains identified in a previous qualitative analysis of families’ experiences and priorities about their child’s hospital admission.⁵  As a research team, we then conducted 4 rounds of item revisions, in which we combined similar items, edited items to be pediatric-specific in relevance and content, and made them specific to the hospital-admission process. Following these steps, 33 items specific to pediatric hospital admission were candidates for the survey.

A 12-member multidisciplinary advisory panel reviewed these 33 potential survey items to prioritize items for inclusion. The participants on the panel, convened to support implementation of the previously described trial, included hospitalists, a primary care physician, urgent care physician, emergency department physician, nurse, federal payer representative, pediatric subspecialists, and parents. To establish content validity, panel participants anonymously rated each survey item for its relevance to the hospital admission process using an ordered response scale of 0 to 4, with 4 = extremely relevant and 0 = not relevant at all. We elicited open-ended feedback on the survey structure (eg, question order, grouping, flow), readability, and missing or duplicative content. Panel members then discussed the items and survey feedback during a web-based video conference session. From this process, 4 questions with a mean relevance ranking below 2.5 were eliminated, 24 questions were revised for clarity and specificity, and 2 items were added on error and mistake reporting from the Patient and Family Centered I-PASS study.¹⁷ 

The resulting 31 items went through 3 rounds of think-aloud cognitive interviews with 16 parents to establish response process validity. Rounds 1 and 2 included parent volunteers from advisory councils at the 3 participating hospitals; round 3 was conducted with parents of children hospitalized for general medical conditions. Cognitive interviews led to the subsequent elimination of 5 items and modification of 15 items, leaving 26 items. We then pilot tested the 26-item survey with 27 caregivers of hospitalized children. After pilot testing, we assessed the distribution of responses for each question, the range and median time to survey completion, and elicited qualitative feedback from respondents; a single item was eliminated after this process.

After completion of this process, the initial PHASE survey included 25 items assessing the family experience of hospital admission along with 6 additional patient and family demographic questions (Supplemental Table 6). Race and ethnicity were included as self-report by the parent only. We included these social constructs given known disparities in health care access and quality based on the constructs.^18,19  Items were primarily evaluated using a 4-point ordinal response scale with inclusion of a single open-ended question to provide opportunity for additional comments. The survey was translated into Spanish, Arabic, Nepali, and Somali (the most frequently spoken languages other than English at our implementation sites).

We launched prospective data collection in February 2020, at the beginning of the previously described trial. Parents of children and adolescents <18 years of age were eligible to participate if their child was a patient in a participating primary care practice or referred from a participating urgent care center for either direct admission or admission through the emergency department to a general pediatric bed. Interfacility transfers and ICU admissions were excluded. Eligible admission diagnoses included pneumonia, gastroenteritis, dehydration, influenza, skin and soft-tissue infections, urinary tract infections, fever or other viral infection not yet specified, as well as those admitted directly with failure to thrive or hyperbilirubinemia. These diagnoses were chosen to minimize heterogeneity of the study population; together, they account for approximately 25% of unscheduled pediatric hospitalizations in the United States.¹⁰  Eligibility criteria for caregivers included ability to complete the written survey in 1 of the previously described languages, presence at the time of hospital admission, and availability between 6 and 72 hours after admission to engage with a research coordinator. Surveys were completed during the hospitalization so that responses were not influenced by the hospital-to-home discharge transition.

Caregivers of hospitalized children were approached by a research coordinator after hospital admission. Each child who was eligible for the study had electronic health record (EHR) information extracted including demographic characteristics (eg, age at time of hospitalization), primary diagnosis, measures of time-to-care, medical complexity (assessed using the Pediatric Medical Complexity Algorithm),²⁰  and length of stay. After providing verbal consent, participants completed the survey on a tablet using REDCap.²¹  Initially, no financial incentives were provided. Following a pause in survey collection for 4 months, at the start of the COVID-19 pandemic (March–June 2020), during which time in-person research activities were halted at the parent institution and protocols were developed for remote survey administration via caregivers’ personal phones or tablets, participants were offered a $10 gift card after survey completion. Surveys analyzed here were collected from February 2020 through November 2021.

Psychometric analyses were conducted using Stata 17.0 and R (version 4.3.1). Items with >10% of respondents choosing “not applicable” were excluded. To evaluate individual item performance, we examined response distribution characteristics, observing ceiling effects in 14 of 25 items (defined as >80% of respondents chose the maximal score). The number of items with ceiling effects prohibited the running of the exploratory factor analysis (EFA). Subsequently, we reengaged the multidisciplinary panel to rank items with ceiling effects in order of importance to the admission process. From this, we stratified each item based on theoretical relevance to the outpatient-to-inpatient transition, ceiling effects, and original question mapping to create a prioritization order for inclusion of items into the EFA. Subsequently, 4 items were eliminated based on the prioritization order for the EFA to run.

EFA with varimax rotation was then used to explore the structure underlying the survey items. Surveys from February 2020 to July 2021 were included. Factor loadings guided decisions regarding which items measured the same underlying construct. Inclusion or exclusion of an item in a construct was determined iteratively by examining factor loadings (>0.3) and Cronbach α to identify redundant items or items that did not sufficiently measure the same construct. We used Eigenvalues via a scree plot and parallel analysis to determine the best number of factors for the data and conceptual viability (Supplemental Fig 2). We eliminated 7 additional items through this process.

Based on the results of the EFA, 14 items representing 4 domains were included in the final instrument. We then calculated item and domain scores using a top-box approach, where the total score and the score for each domain represented the proportion of items for which the top answer was chosen by respondents. Possible scores ranged from 0 to 100, with high scores indicating higher quality of care. For example, a domain score of 80% means that 4 questions of 5 on average within that domain were answered as the top choice. Item scores represent the proportion of respondents that chose the top answer for the individual item.

In phase 3, we undertook a confirmatory factor analysis (CFA) using categorical correlations from surveys collected from August through November 2021. Known-group validity was assessed comparing extracted EHR timeliness of care data from individual patients linked to their survey, as well as by hospital type and medical complexity using Spearman correlations for all objective measures except hospital type, which used polychoric correlation. Model fit was evaluated using comparative fit index (>0.95 excellent), root mean square error of approximation (<0.05 excellent), and goodness of fit indices (>0.9 acceptable).

The survey was completed by 910 caregivers of hospitalized children; 62 (7%) were excluded because of incomplete surveys, leaving 848 (93%) completed surveys. Of those, the first 350 surveys were used for the EFA and the final 498 from the data collection period for the CFA. Patient, caregiver, and hospital demographic characteristics are presented in Table 1.

In the EFA, we identified that a 4-factor solution was the best factor structure with similar items loading together on distinct factors without cross loading (Supplemental Table 7). The 4-factor solution was corroborated by the CFA. Model fit indices demonstrated satisfactory fit using the comparative fit index (0.983), root mean squared error of approximation (0.043), and goodness of fit (0.990). Acceptable internal consistency within the identified factors was established with Cronbach α scores for each domain >0.7 (Table 2). The final PHASE instrument, consisting of 14 items within 4 domains, is presented in Table 3.

Top-box domain scores for the exploratory sample ranged from 62% (95% confidence interval [CI], 58–67) for timeliness of care to 84% (95% CI, 81–86) for Patient and Family Engagement (Table 2). Confirmatory sample domain scores followed a similar pattern, ranging from 49% (95% CI, 46–53) for Timeliness of Care to 81% (95% CI, 79–84) for Patient and Family Engagement. Domain scores for Timeliness of Care were the lowest in both samples, although Effectiveness of Care Delivery also dropped from 76% (95% CI, 73–80) in the exploratory sample to 70% (95% CI, 67–72) in the confirmatory sample.

Correlations between domains and medical complexity, length of stay, and hospital type are shown in Table 4. Timeliness of Care correlated, as expected, with HER-extracted variables of time to first vital sign assessment at the hospital and time to first clinical intervention (ie, first medication administration, laboratory testing or imaging). Information sharing weakly negatively correlated with length of stay (Spearman ρ, –0.203) and moderately with hospital type (children’s hospital versus general hospital, Spearman’s ρ of 0.323). Individual item correlations are in Supplemental Table 8. The strongest correlation was demonstrated between the time to first vital sign assessment from the EHR and the item, “How soon after arriving to the hospital were vital signs checked?” with a Spearman ρ of –0.439. Similarly, hospital type demonstrated a moderate correlation with the item, “Did the care team understand your child’s complaint or condition?” (Spearman ρ, 0.422).

Given the differences in correlations by hospital type, we examined domain scores across hospitals (Table 5). Significant differences by domain were noted between hospitals in Information Sharing (P = .03) with the community hospital having the highest domain score (92% [95% CI, 85–100]) and the 2 children’s hospitals having similar scores (hospital A, 77% [95% CI, 73–81] and hospital C 80% [95% CI, 76–85]). Effectiveness of Care Delivery also had differences by hospital type (P = .047), although all 3 hospitals had overlapping CIs.

A 4-domain, 14-item caregiver-reported hospital admission experience measure demonstrated acceptable validity and psychometric properties across diverse hospital settings. The 4 domains, Patient and Family Engagement, Information Sharing, Effectiveness of Care Delivery, and Timeliness of Care, provide a global assessment of the transition process from outpatient-to-inpatient for pediatric patients requiring hospitalization.

Substantial quality improvement work within pediatric care has been targeted toward improving the quality of hospital discharge, including improving communication with the primary care provider,²²  efficiency and flow,^23,24  family readiness for discharge,²⁵  and reducing readmissions.²⁶  Furthermore, entire curricula and programs have been developed to support the daily discussion of discharge criteria,²⁷  improve the hospital-based follow-up of tests pending at discharge,²⁸  and better training of families for complex discharges.²⁹  Although there has been some quality improvement work around primary care physician communication for incoming hospital admissions,³⁰  most quality and safety efforts for hospitalized patients have focused on improving care during the hospital stay rather than the admissions process itself. The development and validation of the PHASE measure illustrates where there are opportunities to improve quality of care during the admissions process and to strengthen communication with families. The PHASE measure correspondingly then can be used to evaluate the quality of transitions into the hospital and to focus admission quality improvement efforts.

One of the strengths of the PHASE measure is the establishment of substantial validity evidence to support its use in diverse hospital settings. In addition to establishing content and response process validity through an expert-consensus process and cognitive interviews, we demonstrated construct validity with the EFA, CFA, and the model-fit statistics. We also demonstrated concurrent criterion validity by comparing parent perceptions of wait times with the EHR timestamps, with longer wait times before a clinical intervention as measured by EHR time stamps correlating with lower scores in family engagement, perception of effectiveness of care delivery, and perception of timeliness of care. Similarly, we established known-group validity because longer length of stay was correlated with lower scores in family engagement, information sharing, and effectiveness of care delivery. Interestingly, hospitalization at a children’s hospital correlated with lower scores on information sharing, perhaps because of larger health care teams providing care. However, when looking across domain scores by medical complexity, we found no difference in scores by domains.

The development of the PHASE measure across 3 hospital systems that care for diverse populations improves the generalizability of the instrument. Frequently, representation of Black and Hispanic children within clinical trials is substantially below their proportion within the US population.³¹  During instrument development, we specifically designed the instrument to be accessible (eg, limiting the reading level to below sixth grade) and lowered the burden of participation (eg, QR codes for direct scanning using a parent’s own device versus using a study-provided tablet). Race and ethnicity were self-reported from caregivers, we added financial compensation for participation, and study materials were available in multiple languages.³²  Although we were successful recruiting a diverse sample across race, ethnicity, medical complexity, insurance type, and caregiver education, future iterations of the instrument could be piloted within other languages and for verbal administration for those unable or learning to read.

With the growth of pediatric hospital medicine as a specialty and decreases in the proportions of primary care clinicians and clinic-based specialists caring for their patients during hospitalization, understanding the quality of the transition from home-to-hospital is critical. Improving processes for handoffs, medical history and medication validation, and communication with families at the time of admission may reduce risk of harm to patients such as medication errors, unnecessary repeated tests or evaluations, and other adverse events.³³  Future applications may use the PHASE measure as a measure of effectiveness for cost-benefit and cost-effectiveness analyses. Additional work may also focus upon the testing of the measure as a patient-reported experience measure, particularly for 12- to 18-year-old children who are able to reflect on their admission experience to further promote patient-centered initiatives.

We note several limitations. First, 14 of the 25 original items showed ceiling effects. Future validation iterations could consider the use of more than 4 response options. Second, to enhance feasibility, the survey was administered up to 72 hours after admission. This timing may have influenced recall of the admissions process relative to the remainder of the hospital stay. Third, because of the COVID-19 pandemic, we were required to change the data collection method over the time of survey administration, with an increasing reliance on remote methodologies. Fourth, because of time constraints during phase 1, we were unable to perform cognitive interviews or pilot the instrument in languages other than English. This introduces the potential that cultural nuances were overlooked or misunderstood. Last, we administered the survey only to caregivers of patients admitted to a medical pediatric ward. Additional validation needs to be performed for the PHASE measure in other populations, including admissions to ICUs and patients with surgical diagnoses.

We developed a 14-item caregiver-reported hospital admission experience measure with acceptable validity across both children’s hospitals and general community hospitals. This measure can be used to evaluate the quality of transitions into the hospital and to focus admission quality improvement efforts.

CFA

confirmatory factor analysis

CI

confidence interval

CQI

Consumer Quality Index

EFA

exploratory factor analysis

EHR

electronic health record

HCAHPS

Hospital Consumer Assessment of Healthcare Providers and System

PHASE

Pediatric Hospital Admission Survey of Experience