Parents Have More to Say: Comments From the Child HCAHPS Single Question Versus a Narrative Item Set

We partnered with a medium-sized 131-bed children’s hospital-within-a-hospital at an urban, academic medical center on the West Coast serving a racially and economically diverse patient population. In July 2017, the hospital began using the Child HCAHPS survey and its single open-ended question to collect patient experience data from families of hospitalized children.^6,20  We previously published results of our analysis of the valence (ie, hedonic tone), content, and actionability of the comments in response to the single open-ended question from the Child HCAHPS survey administered by mail in English and Spanish from July 2017 to December 2020; 546 respondents provided a comment (comment response rate of 58%, 546 of 945).¹⁹  Between May 2021 and April 2022, the 6-item beta version of the Child HCAHPS NIS was included (instead of the single question) on the Child HCAHPS survey. Table 1 lists the 6-item NIS for Child HCAHPS. The Child HCAHPS survey was administered by mail and e-mail in both English and Spanish to parents or guardians of children who had an overnight stay in the hospital.

We coded NIS comments using a conventional content analysis approach.^21–23  We developed a code structure and codebook using systematic, inductive, and deductive procedures. We started with our previously developed code structure,¹⁹  which contained codes for the valence of the comment (ie, positive versus negative, or a comment with both positive and negative aspects that is coded as mixed), the frequency of the topics discussed (ie, single event, something that happened sometimes, or something happening for the entire hospital stay), staff mentioned (ie, doctors, nurses, child life specialists, “everyone”), location in the hospital (ie, hospital room, surgery, emergency department), and the actionability of the comment, including whether the comment was actionable for an individual provider or actionable for the organization. We defined comments as actionable if they contained information that would allow a reader to address problems identified in the comment or to highlight particularly good behavior or practices (including the who, what, when, where, and how).^8,14–17  We coded if a comment contained content covered by the Child HCAHPS survey and then created a set of codes for content covered by each of the Child HCAHPS closed-ended questions. For example, we created a code for “Kept informed about what was done for child” corresponding to the Child HCAHPS question “During this hospital stay, how often did providers keep you informed about what was being done for your child?”

When we coded the content of a comment, the code itself either addressed content from the previous codes, (ie, [deductive] using 1 of the codes developed from the single open-ended question) which included both content of items on the Child HCAHPS survey, or not, or pointed to new content that was not raised when coding the content of the single open-ended question (ie, these codes emerged [inductive] from the NIS comment data). We iteratively added codes as they emerged for content not included in initial set of codes.²⁴ 

Our 2-person coding team (D.D.Q. and Z.P.) each initially coded the same set of 5% of comments, yielding a κ coefficient of 0.83, indicating very good agreement. We met, discussed discrepancies, and refined concepts and codes. Then, we each coded an additional 5% of comments and obtained a κ coefficient²⁵  of 0.93, indicating “almost perfect” agreement.²⁶  We split the remaining comments and coded those independently (D.D.Q. coded 23% and Z.P. coded 77% of the remaining comments). Each coder audited 10% of comments coded by the other coder; we met to discuss and resolve any discrepancies.

We linked the coded comment data to the demographic information collected on the Child HCAHPS survey (ie, respondent-report child health status, child’s race and ethnicity, and respondent age, education, and relationship to child).

For the 6-item NIS data, we initially coded each of the comments at the “comment-level” (ie, item-level) and then rolled-up the item-level comment data to the “respondent-level,” so a comment contained the narrative information (and associated codes) provided by each respondent. We reviewed patterns of comments at the item-level and respondent-level by valence, content, and actionability. We also examined whether respondents answered 1 or all of the 6-item NIS. We calculated word count for each comment (item-level and respondent-level) to understand a patient’s engagement with the NIS. We compared the valence, content, and actionability of the respondent-level NIS comments to the single-item comments (respondent-level) using z-tests comparing the proportions of comment content in the NIS comments and the single-item comments. Study protocols were approved by Rand’s Human Subjects Protection Committee (IRB_Assurance_No: FWA00003425; IRB Number: IRB00000051).

From May 2021 to April 2022, 118 parents and guardians completed the Child HCAHPS survey with the 6-item NIS. We excluded 13 comments that did not contain any real content, including comments that were solely laudatory (eg, “Thank you!”) or comments that just described a child’s health problems or the care they received (eg, “My child has an eating disorder and has now been admitted twice”). This yielded 382 individual NIS responses from 77 unique parents and guardians (65% of respondents providing comment; 77 of 118). The completion rate for each NIS item varied: 42 completed all 6-items, 16 completed 5 items, 7 completed 4 items, 2 completed 3 items, 6 completed 2 items, and 3 completed 1 item. Table 1 lists the variation in item-level response rates for each NIS item.

Table 2 describes the demographics of those who provided comments compared with those that did not using z-tests. Respondents who provided comments had higher educational attainment than those that did not; otherwise, respondents were similar in age, language, relationship to child, child’s race and ethnicity, and child’s health status.

Parents and guardians wrote an average of 145 words (median 102 words) in response to all NIS questions. They wrote the most words (mean 36.2, median 22) for question #3: “Next, we’d like to focus on any experiences during your child’s hospital stay that you wish had gone differently. Please explain what happened, how it happened, and how it felt.” and the least words (mean 18.4, median 13) for question# 4: “Please describe how doctors, nurses and other hospital staff interacted with your child, and how they got along with your child.”

Table 3 reports the valence, content, and actionability of NIS responses overall and by item. Table 4 provides illustrative examples of both actionable and nonactionable comments across the 6 items of the NIS set, denoting the specific elements of actionability (ie, who, where, when, how, what).

At the respondent-level, most NIS comments contained both positive and negative content (60%, 46 of 77), some contained only positive content (39%), and only 1 comment contained only negative content (1%,). At the item-level, most comments were positive (77%, 296 of 382), some negative (17%), and a few were mixed (6%). For 5 items of the 6-item NIS, more than 80% of comments were positive. Notably, for NIS question #3, 64% of comments (39 of 61) were negative.

Most respondent-level comments contained experiences occurring at multiple frequencies during the hospital stay (71%, 55 of 77). Nearly all respondent-level comments contained a description of something happening for the entire hospital stay (96%), most described something that happened sometimes (62%), and more than half described a single event (55%). At the item-level, frequencies of described events varied by NIS item; parents and guardians were most likely to describe a single event in response to question #3 (33%, 20 of 61).

Most respondent-level comments (96%, 74 of 77) contained content that was not asked by Child HCAHPS closed-ended questions and, also, most comments (82%, 63/77) contained content covered by Child HCAHPS closed-ended questions. Of the comments containing Child HCAHPS content, almost half (49%, 31 of 63) contained content describing an overall rating of the hospital or care received (eg, “We were impressed by the consistent standard of care.”). For example, this includes 7 comments contained only a descriptive overall rating of the hospital and 24 comments that contained other content and, also, a descriptive overall hospital rating (totaling 31). Another most frequently mentioned Child HCAHPS topic in response to the NIS was being kept informed about care for your child (30%, eg, “Rounding with her care team made me feel in the loop and I appreciated being able to understand the plan”), whether doctors treated the parent and guardian with courtesy and respect (30%, eg, “On our hospital stay there was a cardio surgeon that was pushing us to go home when the primary doctor wasn’t comfortable and even said ‘Well welcome to your hotel’”), and whether doctors explained things to the parent and guardian in a way they understood (27%, eg, “The first doctor that attend[ed] us didn’t explain well why the procedure had to be done…”).

Of the comments that included content not covered by Child HCAHPS closed-ended questions, most (63%, 47 of 74) contained both an overall provider rating and other content, whereas the remaining comments (37%) only contained content other than an overall descriptive provider rating. Of the comments containing non-Child HCAHPS content, the most frequently mentioned content was whether providers were caring, friendly, helpful, or courteous (77%, eg, “All doctors, nurses and staff cared so well for our son. They were gentle, loving, and helpful”), how well staff coordinated care (51%, eg, “The coordination with being admitted through the emergency room could have been better organized”), and general communication (elements of communication not covered by specific Child HCAHPS content) (43%, eg, “attending physician did the best job communicating”).

Most respondent-level comments (69%, 53 of 77) contained some actionable information. This could be either actionable for an individual provider (23%), the organization (29%), or both (17%). At the item-level, only for question #3 were a majority of comments deemed actionable (69%, 42 of 61), with nearly half actionable on the organizational-level (46%) and some actionable on the provider-level (21%), and only 1 comment actionable on both levels (1%).

We compared the content of NIS comments to our previously published analysis of valence, actionability, and content of comments from the single open-ended question using z-tests (Table 5). Respondent-level NIS comments were significantly more likely to have mixed valence (60% NIS vs 16% single-item, P < .001) and significantly less likely to be solely positive (39% NIS vs 57% single-item, P = .005) or solely negative (1% NIS vs 28% single-item), P < .001). NIS comments were also significantly more likely to contain Child HCAHPS content (82% NIS vs 51% single-item, P < .001) and non-Child HCAHPS content (96% NIS vs 83% single-item, P = .006). NIS comments were also significantly more likely to contain some actionable content (69% NIS vs 39% single-item, P < .001), including content that was actionable for individual providers (23% NIS vs 10% single-item, P = .002).

This NIS pilot test provided the opportunity to gain real-world comments from parents and guardians using the newly developed, structured 6-item NIS about actual care experiences of hospitalized children. Given our study team had previously examined comments from the Child HCAHPS single-item questions, “Is there anything else you would like to say about the care your child received during this hospital stay?,”¹⁹  we were in a position to also compare the valence, content, and actionability of comments from the Child HCAHPS single-item and the 6-item NIS.

Previously, we found that most comments to the single open-ended question on Child HCAHPS were positive (57%); the majority (71%) mentioned specific hospital staff; about half (51%) contained content covered by the closed-ended Child HCAHPS questions; and the majority (83%) also included content not covered by the Child HCAHPS survey. Also, thirty-nine percent of the comments were deemed sufficiently specific to make improvements (ie, actionable) in inpatient pediatric care. Specific content covered within the comments varied widely, but interactions between a parent and the child’s providers were written about most frequently, as well as respondents describing an overall rating of providers or whether providers were caring or friendly.¹⁹ 

In this study, we found in response to the structured 6-item NIS that 65% of parents and guardians provided a comment (77 of 118), which is higher than the 58% of respondents that provided a comment to the single-item question (546 of 945). Also, parents and guardians wrote nearly 6 times as many words in response to the 6-item NIS as those who responded to the single item. This implies that respondents were engaged by each of the NIS questions and wrote roughly the same length of narrative to each individual NIS question as they did for the single question. The NIS format consistently elicited more narrative comment from each respondent than the single question.

Single-item comments were more positive (57% vs 39% NIS), yet most NIS comments provided at least 1 negative response within the 6-item NIS (61% vs 43% single-item). This may be the case because the structured probes provided more opportunities for respondents to comment, and the structured questions were written to explicitly elicit things that went wrong, which yielded more text from given respondent and intentionally drew out negative aspects of their experiences so that, overall, an NIS comment (across all 6 items) contained both positive and negative (ie, mixed) experiences.

Most respondent-level comments contained experiences occurring at multiple frequencies during the hospital stay (71%, 55 of 77), with nearly all including a description of something happening for the entire hospital stay (96%) and more than half described a single event (55%). Of note, question #3 most often elicited description of a single event (33%, 20 of 61), as it asks explicitly about any experiences “you wish had gone differently.”

Eighty-two percent of NIS comments included content asked on the Child HCAHPS survey (vs 51% of single-item comments). The NIS comments contained more about communications between parents and the child’s doctors and nurses as well as more non-HCAHPS content specifically related to comments about general communication and whether the providers were caring, friendly, helpful, or courteous. This may possibly reflect the fact that the structured 6-items explicitly ask about communication and about interactions with hospital “doctors, nurses and other hospital staff, and how you got along with them,” whereas the single question does not point to any particular content or aspect of care experiences or specific hospital staff, but rather asks a general question.

Overall, a larger proportion of NIS comments were deemed actionable (69% vs 39% single-item). This would be expected given that we found that the NIS comments are longer, contain more mixed and negative comments (negative comments are typically more actionable), contain descriptions of multiple experiences or events, almost always include mention of specific hospital staff, and explicitly request input from parents or guardians about specific aspects of their experience, all of which increase the level of detail and specificity in a respondent’s narrative text. Notably, the pilot also demonstrated that 1 of the 6-item NIS –NIS question #3– consistently elicited information that was deemed actionable for both provider and health system quality improvement.

Health care providers and systems should consider the added value or costs to administering the 6-item NIS instead of the single item. First, adding the additional items to the Child HCAHPS survey may lower the survey response rate; we were not able to assess overall response rate changes for this pilot study before and after the addition of the 6-item NIS. Previous studies estimated that adding an additional 12 closed-ended item supplement to the Medicare CAHPS survey lowered the overall response rate by 2.5%.²⁷  No evidence exists about survey length and adding open-ended items. Health care providers may be able to offset increased survey length and improve response rates by making surveys more visually attractive²⁸  or using innovative models of survey administration (ie, sequential mixed-mode, web-based forms, point-of-care survey administration).²⁹  Second, the 6-item NIS does yield additional information to that captured by the closed-ended survey questions. Third, analyzing the volume of text from the 6-item NIS will take longer than analyzing the text from a single-item; however, the narrative data review and analytic process does yield robust information and provide specific actions to take to improve care (ie, NIS data were 69% actionable vs 39% from the single-item). Notably, narrative data are supplemental to the quantitative data collected by the closed-ended questions, which are more readily able to be analyzed, synthesized, and reported. Lastly, quality leaders may want to focus their efforts on reading and responding to NIS question #1, given it had the highest response rate, and to NIS question #3, as those responses are likely to have the greatest potential for improvement.

Our study has limitations. We analyzed comments from one hospital with 2 locations within an academic medical center, which may not be representative of larger national hospital samples. We also were not able to compare the demographics of respondents who completed the survey or who provided comments on the survey to those of the general patient population. Future efforts to systematically analyze comment data and use it for quality improvement should explore whether respondents who provide comments are representative of the entire patient population. Also, research is needed that examines whether the review and analytic process used by researchers to code narrative comments can be replicated as well as clearly delineated in a step-by-step process with guidelines concerning pattern recognition, sufficient sample sizes, and actionability for use by quality leaders. Nevertheless, our analysis provides insight into the content and patterns of the newly developed NIS compared with the standard open-ended question used on the Child HCAHPS survey for which there is little published literature.

In sum, the 6-item, structured NIS provided space for parents to elaborate on experiences both asked about on the Child HCAHPS survey but also other topics. The NIS elicited high percentages of positive and negative comments with sufficient detail to make improvements. However, a large-scale, multisite NIS implementation demonstration is needed to establish analytic parameters and guidelines for the use of narrative data, to assess how quality leaders and frontline staff are able to process, interpret, and translate the content of the NIS comments to support and enrich processes for making improvements in inpatient pediatric care, and to determine whether the additional burden associated with administering and analyzing the 6-item NIS provides enough benefit to justify its widespread use.

We acknowledge the time and support of the hospitals leaders who provided the survey data for this study; the parents and guardians that completed the surveys and provided the narrative comments about their pediatric inpatient stays that were analyzed for this study; and Mary Ellen Slaughter, MS, for her contribution in providing the quantitative data in a form to link to the narrative data.