What Parents have to Say: Content and Actionability of Narrative Comments from Child HCAHPS Survey

Child HCAHPS surveys were collected in English and Spanish from parents or guardians of hospitalized children at an urban, medium-sized, 131-bed children’s hospital with 2 facilities nested within an academic medical center on the West Coast. We obtained the de-identified comment data from the hospital’s completed Child HCAHPS surveys from July 2017 to December 2020. We identified comments that contained uninterpretable text and removed these from the counts/data.

We developed a code structure and codebook using systematic, inductive, and deductive procedures. We started with a priori codes based on our objective of examining the content and actionability of the comments. The a priori content coding included the valence (positive, negative, or mixed), whether the comment referred to a teen (versus child), raised a possible patient safety issue, and whether the content was similar to or added to what was known from the closed-ended Child HCAHPS questions. We also created a priori codes based on the content of the Child HCAHPS survey item wording; that is, we created a code for each Child HCAHPS item. For example, we created a code for “Kept informed about child’s care in ER” based on the Child HCAHPS question, “While your child was in this hospital's emergency room, were you kept informed about what was being done for your child?”

When we coded the content of a comment, the code itself either addressed a survey item on the Child HCAHPS survey (ie, [deductive] using one of the a priori codes) or pointed to content that was not specific to items on Child HCAHPS (ie, these codes emerged [inductive] from the comment data). For example, Child HCAHPS items refer to different aspects of patient provider communication, so we created a priori codes for each of these: provider listens carefully, explains things in a way that is easy to understand, encourages questions, and treats you with courtesy and respect. The Child HCAHPS survey, however, does not contain any assessment of the friendliness or caring nature of a provider, and this yielded a non-Child HCAHPS code for the friendly or caring nature of the provider. Our coding categories were based on both a priori codes and those directly developed from the responses. We performed directed content analysis using current research as a guide to establish our initial codes²⁹  and conventional content analysis^30,31  to identify codes to cover content not captured by the Child HCAHPS survey.^32–34  The full set of codes represented an appropriate response to understand the number of times a comment containing certain information was given and the number of circumstances mentioned in totality. This explains what is observable, easily perceived, and apparent in the text.^31,35 

To code the actionability of the information in the comment, we created a priori parent codes for when, where, who, and how, and through the coding process, the child codes for each of these emerged: “when” as which part of the stay (eg, transition from emergency department, discharge); “where” as the location (eg, emergency department, surgery, inpatient room, radiology, etc); “who” as the staff involved; and “how” as the descriptive context (eg, ask my son about his pain level versus basing it on what his dad would say). The “what” is the content of the comment. For context, we also coded the frequency of the experience described in the comment (ie, single event, occurred sometimes, occurred throughout the whole stay, multiple frequencies). We also coded whether, taken altogether, the narrative information was actionable (or not) (i.e., included the specific elements of the commonly used ‘5 Ws and 1 H’ of information gathering and project management: what, why, when, where, who, and how); thus, actionable comments included several codes for when, where, who, or how. Lastly, for actionable codes, we coded whether the comment was actionable for an individual provider or for QI at the organization level.

Our two-person coding team (D.D.Q. and Z.P.) used meetings to reach consensus on codes, identify discrepancies, refine concepts, and define codes.³⁶  We coded comments independently. Each coder initially coded the same randomly selected 5% of comments. We then compared coding differences and agreement; we obtained a pooled κ coefficient of 0.86, indicating “very good” agreement.^37–39  Each coder then coded half the remaining comments. We discussed emerging codes at regular meetings, resolving any differences through discussion to establish the final coding scheme. After all comments were coded, each coder audited every fifth comment coded by the other coder, discussing and resolving the few discrepancies identified to ensure consistency of coding.

We linked the coded comment data to the demographic data collected on the Child HCAHPS survey (ie, respondent-reported child health status, child’s race/ethnicity, and respondent age, education, and relationship to child) and the survey administration variable indicating the type of stay (inpatient, included surgical, included PICU, included both PICU and surgical).

We reviewed patterns of comments by valence, content, and actionability, as well as by the available characteristic information, such as by child race/ethnicity and by respondent characteristics (age, education, relationship to child), respondent-reported child health status, and by type of stay. For example, the Child HCAHPS survey asks respondents, “In general, how would you rate your child’s overall health?” using a 5-point response from poor to excellent. Using this data, we compared comments from parents and guardians of children whose health was reported as “Excellent,” “Very good,” or “Good” (ie, healthier) with those from parents of children whose health was “Fair” or “Poor” (ie, sicker).

From July 2017 to December 2020, 945 parents/guardians completed the Child HCAHPS survey after a hospitalization and 62% (586 of 945) provided a written comment. We excluded 38 comments as “nonsubstantive” (eg, “Thank you!”), yielding 548 comments.

Demographics of the respondents and their children are shown in Table 1. Most comments were provided by the child’s mother (84% of 548), those who spoke English (88%), or those who had a college or graduate degree (87%). Nearly half of respondents were between the ages 35 and 44 (47%). Non-Hispanic Whites provided nearly one-half the comments (46%), and Hispanics one-third (32%). Most respondents reported their child had good-to-excellent health (85%). Four comments referred to a teen.

Three-fourth of the 548 comments were about pediatrics inpatient stays without a surgery or PICU stay (Table 1). We found 5% of these comments involved a possible patient safety issue or near miss (eg, child fall or discharged with inappropriate medication). Four comments (1% of 548) specifically mentioned the COVID-19 pandemic.

Most comments referred to the patient and parent or guardian’s entire experience during the hospital stay (68% of 548) (Table 2) (eg, “Everyone that dealt with [child’s name] was outstanding, including volunteers”). Most comments also did not reference a specific setting or hospital location (79%); of those that did, most mentioned the hospital room or bay (eg, “My daughter had to share a room with another child, which made it extremely uncomfortable”).

Most comments had positive tone and valence (57% of 548) (Table 2). Seventy-one percent of comments (389 of 548) mentioned specific hospital staff members, with 31% mentioning more than 1 staff member. Nurses and doctors were most frequently mentioned (36% and 23%, respectively) (eg, “My baby had excellent care. I really appreciate the nurses and doctor that helped us through this time”). Doctors were mentioned in similar percentages of comments by valence (23% positive, 21% negative, and 27% mixed) (eg, “The night shift doctor I originally spoke to was rude and disrespectful. She rolled her eyes at me and muttered under her breath”), whereas nurses were more frequently mentioned in positive (38%) and mixed (48%) comments than negative ones (26%) (eg, “A big shout out to the evening nurses, [nurse names], and the nurses who took over the morning shift!”).

Most comments (83% of 548) contained information not asked on the Child HCAHPS survey, but half (50% of 548) also had content that mapped directly to aspects of care experiences on the Child HCAHPS survey (Table 2). Specifically, about half the comments had information not asked on the survey, one-third had information both asked about and not asked about on the survey, and one-sixth had information only asked on the survey. The descriptions and illustrative examples of quotes for each code in the coding scheme are provided in Table 3.

Of the comments that included Child HCAHPS content (n = 279) (Table 2), 58% (161 of 279) contained reference to a rating of the overall hospital experience (eg, “The best hospital experience we've ever had”), whereas 34% mapped to other specific content. Doctor communication with parent or guardian (13%; 35 of 279) and nurse communication with parent or guardian (10%; 28 of 279) included such things as “We mostly had a good experience, but one MD intern constantly gave us misinformation” and “…I was completely ignored by the nurse when expressing my concerns.” Being kept informed about child’s care (9%; 24 of 279) included comments such as “Our biggest problem was not knowing what time the surgery would happen” and “Everything was wonderful except wish there was more communication about IV drugs administered/given, communication about side effects of Rx taken home.” Hospital environment (8%; 22 of 279) included issues with cleanliness such as “When a patient is discharged, housekeeping needs to thoroughly clean and sterilize the room. Our room was not patient-ready at all.” Lastly, comments about preparing the child to leave the hospital (6%; 16 of 279) addressed issues such as “The case manager was very ineffective, had poor communication skills, and failed to be proactive in ordering supplies needed for discharge.” Also, 8% (23 of 279) contained information that mapped to both an overall rating and another item or domain on the survey. Of these comments containing Child HCAHPS content, 56% (156 of 279) were positive, 25% (70 of 279) were negative, and 19% (53 of 279) were mixed.

Of the comments that included non-Child HCAHPS content (n = 457) (Table 2), 28% (128 of 457) rated a provider, 44% (201 of 457) contained only other content (no provider rating), and 28% (128 of 457) contained both a rating of the provider and other non-Child HCAHPS content. Non-Child HCAHPS comments a third of the time (31%; 140 of 457) included an assessment of whether a provider was caring or friendly (eg, “Most of the nurses in the pediatric unit were very friendly, caring, and knowledgeable, although one of the young nurses freaked out when my baby cried loud and that appeared to be lack of experience/training”). Other non-Child HCAHPS comments were about: issues with care such as vein management or medical support (eg, “…they kept poking her and repeatedly drawing blood more than 11 times. She ended up with bruises all over her body…”) (11%; 50 of 457); hospital services such as parking or cafeteria (eg, “I had to argue with a parking attendant to get the correct price after I told them I couldn't pay the 60-some dollars”) (9%; 43 of 457); sleeping comfort (eg, “Absolutely need better sleeping chairs for parents!”) (10%; 46 of 457); nonpatient or parent communication (eg, “Communication between the pediatric neuro team and the stroke clinic is weak. This caused confusion during our stay and after discharge”) (9%; 43 of 457); and timeliness of care (eg, “Her care was exceptional. Was aware the ER hurried with the diagnosis to plan for her surgery as fast as possible. We are most grateful”) (9%; 40 of 457). Similar to comments with Child HCAHPS content, 52% of non-Child HCAHPS comments (239 of 457) were positive, whereas 29% (134 of 457) were negative and 18% (84 of 457) were mixed.

Thirty-nine percent of comments (213 of 548) were coded as actionable (ie, supplying specific details about at least some aspects of when, what, where, who, or how) and deemed able to be used to improve pediatric inpatient care. Of the actionable comments, about one-fourth (56 of 213) provided feedback that could only be acted upon by an individual clinician or provider, whereas one-half (105 of 213) required organization-level action and the remaining one-fourth (52 of 213) were actionable by both individual clinicians and the organization (Table 2). Table 3 also provides illustrative examples of both actionable and nonactionable comments for each code, denoting the specific elements of actionability (ie, who, where, when, how, what).

Comments containing Child HCAHPS content were primarily actionable at the organizational level only (15%; 43 of 279) or at both the clinician and organization-level (15%; 42 of 279) (Table 4). Actionable comments referred primarily to communication (eg, “…doctors did not explain things very well and accepted a bed at another hospital for our child without our approval, which was very upsetting”); being kept informed about the child’s care (eg, “Our biggest problem was not knowing what time the surgery would happen”); aspects of hospital discharge (eg, “Also, my son was discharged with a sulfa-based medication despite having a sulfa allergy documented in [the medical record]”); and the hospital environment (eg, “Garbage cans in the rooms needed to be emptied more…”).

Comments containing non-Child HCAHPS content were actionable primarily at the organization level only (21%; 97 of 457) (Table 5). Actionable feedback primarily involved timeliness of care (eg, “Not enough echocardiogram techs, we waited 24 hours for an echo”); issues with care (eg, “Night nurse was unfamiliar with the seizures and left me to handle it!”); hospital sleeping and comfort (eg, “Had to sleep on the floor the first 2 nights”); and general aspects of communication (eg, “…Nurse never introduced the nurse that would cover the night shift”).

Comments about the inpatient experiences of healthier children (n = 469) were generally more positive (60% positive, 25% negative, 15% mixed) than those from parents of sicker children (38% positive, 48% negative, 14% mixed; n = 65). About half of the comments regarding sicker children (33 of 65) and half regarding healthier children (235 of 469) contained Child HCAHPS content, whereas >4 in 5 comments for each referred to non-Child HCAHPS content. Comments regarding care of sicker children were more actionable (55%; 36 of 65) than those regarding healthier children (36%; 169 of 469).

Comments about inpatient care also differed by the child’s race and ethnicity. Comments about the inpatient experiences of Hispanic children (n = 176) were more positive (71% positive, 22% negative, 7% mixed) than those for Asian and Pacific Islander children (57% positive, 25% negative, 18% mixed; n = 51), non-Hispanic White children (48% positive, 31% negative, 21% mixed; n = 252), and children of other and mixed races (53% positive, 30% negative, 17% mixed; n = 56). Comments about care for Asian and Pacific Islander children were less likely to contain Child HCAHPS content (37% of 51 comments) than those for non-Hispanic White children (53%; 134 of 252), Hispanic children (51%; 90 of 176), and children of other races (55%; 31 of 56). Comments about care for non-Hispanic White children (88% of 252) and those for Asian and Pacific Islander children (86% of 51) similarly contained non-Child HCAHPS content, whereas this was less likely for Hispanic children (77% of 176) and children of other and mixed races (80% of 56). Comments about care for non-Hispanic White children (47% of 252) were more actionable than those for children of other and mixed races (41% of 56), Asian and Pacific Islander children (37% of 51), and Hispanic children (26% of 176).

The value and use of narrative text information gathered in patient experience surveys are important and emerging research areas.^5,6  Some evidence points to open-ended narrative data as potentially rich sources of information about particular care experiences.²²  However, such data can be costly to analyze and use systematically for QI efforts.

We found many respondents of the Child HCAHPS survey provided comments that reinforced content asked about in the closed-ended items on the Child HCAHPS survey; however, more respondents added information beyond the specific content covered on the Child HCAHPS survey (ie, about non-Child HCAHPS content). Most frequently, parents or guardians added a comment that rated their child’s provider. This suggests that the open-ended narrative text primarily provides opportunity to add information beyond what is collected by the Child HCAHPS survey. For broad concepts, such as communication, capturing additional specific aspects of communication from open-ended narrative data that are not part of the communication behaviors captured in the survey (such as introducing/identifying a nurse during a shift change or being kept informed about delays) may assist leaders in targeting areas of communication for QI that are over and above what is captured in the closed-ended questions. More research is needed on the value of open-ended narrative data and how the content of open-ended and closed-ended survey answers are associated with each other.

Comments on the Child HCAHPS survey provided by parents about their children’s inpatient care were usually positive, tended to be about the entire hospital care experience, and indicated the staff involved. Less frequently, parents and guardians detailed the setting of events that they commented on. Comments most commonly included feedback about an overall rating of their care experience or were about communication with their child’s doctor or nurse. The comments that did not map to Child HCAHPS content were primarily about characteristics of the providers, timeliness of care, sleeping comfort, services, and communication topics not asked about by Child CAHPS questions. A small but nonnegligible number of comments highlighted patient safety issues or near misses, providing a potentially useful source of information for hospitals. Our findings add to the information that describes actionable content acquired from the open-ended Child HCAHPS question from parents and guardians from 1 study in Belgium, that is, in Bruyneel et al 2017.¹²  The actionable comments in the Belgium study were about suboptimal communication with parents and among staff, lack of information about in-hospital wait time to operation, inability to speak to the physician in charge of the child’s care, unsafe/untimely medication administration, vague information about isolation precautions, and too many vein punctures.¹²  Both sets of findings suggest that there is value in analyzing, classifying, and reviewing the valence, content, and actionability of the Child HCAHPS narrative data.

We found that negative comments were more often actionable than positive ones. This supports current evidence of the value of narrative information in making improvements.^15,20,21  Sicker children also constituted a small percentage of the sample but contributed a large share of actionable comments. This supports the current evidence that negative comments and those of sicker patients provide more actionable information. Furthermore, this suggests that hospitals could initially identify negative comments and comments from sicker patients to review for details that may add to QI efforts. This could be done on a regular basis by multiple reviewers to collectively note emerging topics and issues; this could be undertaken while the time and effort are spent to undergo a full qualitative coding exercise of all comments. We report the results for 548 comments over a 3.5-year (42 months) period. If done monthly, this would be ∼13 comments per month, which would be very manageable for most organizations to incorporate into existing QI programs. The richness of the detail and large portion of comments deemed actionable suggest that it may be valuable for hospitals to invest in coding narrative comments provided by parents and guardians about their child’s hospitalizations.

We analyzed comments from one hospital with two locations within an academic medical center, which may not be representative of larger national hospital samples. We also were not able to compare the demographics of respondents who provided comments to those who did not write comments or the general patient population. Nevertheless, our analysis provides insight into the content and patterns of the comments on the Child HCAHPS survey for which there is little published literature. Although we were able to achieve good interrater reliability across coders, including for the actionability of comments, this determination was made by researchers and not the QI team/staff of the hospital.

Little research has specifically examined the narrative data provided by parents and guardians on the Child HCAHPS survey about their children’s inpatient experiences. Our findings indicate that the open-ended question included on the Child HCAHPS survey provides opportunity for parents to both expound on the topics on the Child HCAHPS survey and provide additional insight into other non-Child HCAHPS aspects of their child’s care. Therefore, the content of the narrative Child HCAHPS comments provided insights into topics on the survey itself and on many issues not asked about on the survey. Moreover, more than a third (39%) of the narrative data contained content with sufficient specificity and actionability for both providers and organizations to make improvements in care. Given the richness of the detail and large portion of actionable information in the Child HCAHPS comments (especially the negative comments and comments from parents of sicker children), hospitals could benefit from regular review of these comments for details that may add to QI efforts, while simultaneously pursuing full qualitative coding of the Child HCAHPS comments. More research is needed on the value of open-ended narrative Child HCAHPS data, how the content of open-ended and closed-ended Child HCAHPS survey answers are associated with each other, and its use by quality leaders and frontline staff in QI efforts to improve inpatient pediatric care. Additionally, a multi-item, structured set of open-ended questions, such as the one developed for use with the Clinician and Group CAHPS survey,^1,22  might yield more content and actionable insights than the single item currently in place on the Child HCAHPS survey.

We acknowledge the time and support of the hospitals leaders who provided the survey data for this study. We also acknowledge the parents and guardians who completed the surveys and provided the narrative comments about their pediatric inpatient stays that were analyzed for this study. We thank Mary Ellen Slaughter, MS, for her contribution in providing the quantitative data in a form to link to the narrative data.