Development of an Approach to Assessing Pediatric Fellows’ Transport Medical Control Skills Free

We conducted a modified Delphi process to establish consensus and content validity for the items for assessment of TMC skills.^16–20  We selected the Delphi process because we sought to incorporate a broad range of expertise and account for the widespread geographic distribution of practice sites. The process was directed by a study team of faculty representing each of the 4 subspecialties. The study was considered exempt by the first author’s institutional review board.

To generate an initial list of observable behaviors and skills for the items for TMC assessment, the study team conducted a PubMed search using the search terms “transport,” “transport medicine,” “transport team,” “transport medical control,” and “interfacility transport” and reviewed 39 relevant articles. The study team also incorporated their own personal experience and expertise in TMC. List refinement occurred through virtual meetings of the study group and frequent e-mail exchanges, with representation and input from faculty from each of the 4 subspecialties.

After the development of an initial list of items, we recruited a separate group of transport experts to serve on the modified Delphi panel. We defined transport expertise as (1) transport medical director, (2) responsibility for fellow education in transport, (3) research in transport, (4) service on national transport committees, (5) additional training in transport medicine, and (6) the creation of transport policies. We invited 10 experts in each subspecialty to participate in the Delphi panel. Experts who participated in each round were invited to participate in the subsequent rounds, whereas nonrespondents were excluded.

The Delphi process consisted of a series of surveys sent via REDCap between March 2022 and May 2022 with up to 3 reminders per round.²¹  Experts used a 3-point Likert scale (marginal, important, essential) to indicate the importance of each potential item.^22,23  We used the following definitions: “marginal”: item is of minimal or no importance for the assessment of fellow performance of TMC; should not be included; “important”: item is part of the responsibility of TMC and would be helpful in the assessment of fellow performance; should be included; “essential”: item is a critical step for the performance of TMC; fellows who do not complete this step are not ready to serve as TMC without supervision. A priori, we defined consensus for inclusion as ≥80% agreement that an item was essential/important and consensus for exclusion as ≥80% agreement that an item was marginal.²⁴  The criteria to stop the Delphi process included consensus to include or exclude all potential items with a maximum of 3 rounds.

After evaluating the initial list of items, experts were asked to provide free-text suggestions for additional items and propose modifications to the initial items. After each round, the study team analyzed the responses, reviewed the free-text comments to combine duplicates, and removed non-observable behaviors and skills that would not be feasible for faculty to assess. For rounds 2 and 3, experts were asked to vote on the list of new/modified items.

The study team included 13 PCCM faculty from the Transport Education Working Group of the Education in Pediatric Intensive Care Investigators Network, 2 PEM faculty who serve on the AAP SOTM Executive Council, 2 NPM faculty who serve on the AAP SOTM Executive Council, and 3 PHM faculty with expertise in transport medicine and fellow education. All members of the study team are coauthors of the manuscript.

The study team generated a list of 28 initial items on the basis of a focused literature review and personal experience with TMC. On the basis of previous work, the initial list was divided into 3 domains: (1) information gathering, (2) medical knowledge and decision-making, and (3) communication and professionalism.²⁵ 

In round 1, 37 of 40 (93%) invited experts participated (Table 1). Experts reported a median of 12 years of post-training experience (interquartile range: 8 to 24 years) and represented 30 different institutions. Twenty-four of the initial items met the criteria for inclusion and none met the criteria for exclusion (Fig 1). Of the 4 items that did not meet the criteria for inclusion, 3 were related to gathering the caller’s information, and several experts commented that the caller’s information was collected by the call center at their institution. The study group reviewed all comments and created a list of 8 new/modified items for round 2.

In round 2, 37 of 37 (100%) experts participated. Seven items met the criteria for inclusion. The item “discussed bed availability at the receiving institution” did not meet the criteria for inclusion among all experts but met the criteria for inclusion among experts in PCCM and NPM. The study group reviewed all comments and created a list of 5 new/modified items for round 3. In round 3, 29 of 37 (78%) experts participated. All 5 of the new/modified items met the criteria for inclusion. The preliminary list of items for TMC assessment included 36 items (Supplemental Table 3). The study team consolidated related items into a final list of 26 total items for ease of use during future pilot testing and implementation (Table 2).

In this study, we used a consensus-based process among transport experts in 4 pediatric subspecialties to generate content validity for the items required for the assessment of pediatric subspecialty fellows’ TMC skills.

Of the 26 final items, 12 mapped to the domain of professionalism and communication. Our findings are consistent with the results of a previous Delphi process among members of the AAP SOTM to establish consensus on the core components of a TMC curriculum for PCCM fellows, in which 9 of the 17 included items focused on this domain.²⁵  Communication errors are common in IFT, and as a result, both referring and accepting physicians perceived communication as integral to the IFT process.^26,27  Previous efforts to improve IFT communication included standardization of the handoff process between the referring physician, transport team, and accepting physician.^28,29  Once complete, our TMC assessment tool will help faculty and training programs provide subspecialty fellows with formative feedback on their communication skills during IFT.

Despite the differences in the clinical setting of each subspecialty, experts supported the inclusion of nearly all the proposed items for the assessment of TMC skill. Consensus across subspecialties suggests that TMC is a common skill not dependent on specialty-specific knowledge. The subspecialty community, in partnership with the American Board of Pediatrics, defined 7 common pediatric subspecialty entrustable professional activities (EPAs) as activities that are essential for all subspecialists.³⁰  The first common subspecialty EPA, “provide consultation to other health care providers,” is an activity that encompasses serving as TMC. Transport medicine is specifically included in the American Board of Pediatrics subspecialty EPAs for PCCM, NPM, PEM, and PHM, and TMC is explicitly identified as a “sub-EPA” or “nested-EPA” for PCCM fellows.^31–36  The development of a TMC assessment tool will help fellowship training programs incorporate high-quality, reliable assessment data into these entrustment decisions.

Despite the consensus among pediatric subspecialty experts for most items, there were also important differences. Many experts pointed out the role of their institution’s call center for information-gathering, a role that is common and may increase transport efficiency.^37,38  Institutional practices regarding TMC documentation also vary, and although experts considered documentation important to include, we previously found that PCCM fellows rarely document TMC conversations.¹⁴  The only potential item that garnered disagreement between subspecialties was bed availability because both PCCM and NPM experts considered this important whereas PEM and PHM experts did not. This difference may reflect the more limited bed capacity within the intensive care units.

We acknowledge the limitations of our study. The recruitment of experts from only 4 subspecialties could have contributed to expert panel bias. Dropouts between the different rounds of our Delphi process could have affected the consensus level and contributed to attrition bias. We may have missed other important items by using a modified Delphi process rather than other methods for establishing consensus. Because we focused on shared TMC skills, we may have excluded important TMC skills that are subspecialty specific. In the domain of information gathering, we used the term “elicit,” but information may also be gathered during a transport call through listening. Future testing is necessary to determine how subspecialty faculty will approach these items if the fellow is able to gather the necessary information without asking additional questions of the caller. Finally, the testing and refinement of a TMC assessment tool will be necessary before implementation among training programs because we did not attempt to generate other types of validity evidence in this study.

We generated content validity for the items for the assessment of pediatric subspecialty fellow TMC skill with consensus among experts in PCCM, PEM, NPM, and PHM. Future studies should include the generation of additional validity evidence for a TMC assessment tool before implementation.