The American Board of Pediatrics (ABP) and the Pediatric Hospital Medicine (PHM) subboard developed a content outline to serve as a blueprint for the inaugural certification examination through practice analysis. The systematic approach of practice analyses process is described in the study.
A diverse, representative panel of 12 pediatric hospitalists developed the draft content outline using multiple resources (publications, textbooks, PHM Core Competencies, PHM fellow’s curriculum, etc). The panel categorized practice knowledge into 13 domains and 202 subdomains. By using the ABP database self-defined practicing pediatric hospitalists were identified. Participants rated the frequency and criticality of content domains and subdomains along with providing open-ended comments.
In total, 1449 (12.1%) generalists in the ABP database self-identified as pediatric hospitalists, and 800 full-time pediatric hospitalists responded. The content domains that were rated as highly critical and frequently required in practice were weighted more heavily (ie, the percentage of examination questions associated with a domain) than the less critical and less frequently rated. Both community and noncommunity pediatric hospitalists rated domains similarly (P = .943). Subdomain and preliminary weights were rated with similar means and SDs in the majority of topics.
There was concordance in the rating of domain and universal tasks among both community and noncommunity hospitalists. The areas of significant differences, although minor, could be explained by difference in practice settings. The practice analysis approach was structured, engaged the PHM community, reflected the breadth and depth of knowledge required for PHM practice, and used an iterative process to refine the final product.
Over the past 20 years, pediatric hospital medicine (PHM) has emerged as a distinct field and was recently approved by the American Board of Pediatrics (ABP) as the newest pediatric subspecialty.1–7 Dedicated PHM services are commonly seen in children’s hospitals, including 98% of hospitals associated with academic pediatric departments listed in the 2015–2016 US News & World Report Honor Roll.3–5 There is a growing body of literature that demonstrates the positive impact of pediatric hospitalists on the health of children through practice of high quality care, development and implementation of clinical practice guidelines, improvement in outcomes metrics, increase in patient satisfaction, increase in hospital capacity management, and system efficiency.8–24
The clinical domain of PHM is generalist in nature, being neither organ-specific nor disease-specific. Increasing inpatient acuity and complexity of hospitalized children, increasing demands on outpatient pediatricians, and the expanding complexity of hospital systems have impacted the specialized practice of PHM.8–11 The need for methodical scientific inquiry to ensure best practice stimulated the development of advanced training for pediatric hospitalists to acquire skills not included or developed during general pediatric residency training. More than a decade ago, fellowship programs with unique dedication to PHM were launched, and, through these programs, an early outline for the core elements of PHM practice was created.3,4,21 Subsequently, the National Council of Pediatric Hospital Medicine fellowship directors expanded and refined these expectations.21 PHM fellowship programs now have a standardized curricular framework with well-defined competencies and specialty-specific entrustable professional activities that define the breadth and depth of clinical and nonclinical practice.3,4,7,21 In parallel, the Pediatric Hospital Medicine Core Competencies were published in 2010, and the revision of these is actively underway.22 The PHM Core Competencies address the knowledge, skills, attitudes, and systems improvement competencies within 4 key domains that apply to all pediatric hospitalists.22
Once a subspecialty certification program has been approved, the ABP takes the responsibility for certifying the subspecialists in that field who meet and continue to meet the standards of excellence that promote high quality care of infants, children and adolescents.25 Board certification involves 2 primary components: (1) successful completion of an accredited training program and (2) passing a certification examination.
The ABP employs a process referred to as practice analysis to ensure that its examinations are measuring the testable knowledge and skills required for safe and effective practice. The practice analysis process differs from previous processes in its inclusion of input from physicians practicing in the field, organization of content into key domains, and planned link between content identification and examination questions. Industry standards in psychological measurement recognize practice analysis as the primary method for establishing the content validity and relevance of certification examinations.26–29 Practice analysis was first used for general pediatrics examination in 200726,27 and then again in 201630 to establish and validate the content to be assessed on the general pediatrics certifying examination. Beginning in 2017, the ABP began using practice analysis methodology to update subspecialty content outlines as well. As of 2019, in addition to PHM, 7 of the ABP’s 15 subspecialties have updated and validated their content outlines via practice analysis (similar to the process described here for PHM).
To accurately determine the scope of practice for any discipline, it is necessary to identify and collect information from those practicing in the field. Pediatric hospitalists practice in a wide variety of health care systems and settings, including large university-based centers, community and rural hospitals, emergency departments, and newborn nurseries.8,11,31,32 All pediatric hospitalists provide direct patient care and care coordination. Many also supervise trainees, provide consultation and comanagement for medical and surgical patients, and provide specialized services such as sedation, pain management, and palliative care.3,6,8,21 Therefore, the PHM practice analysis required input from a diverse group of practicing pediatric hospitalists to develop a content outline that represents the depth and breadth of PHM practice. Our goal with this article is to describe the systematic approach used by the ABP to analyze the practice of PHM and ultimately produce the PHM content outline, in which the ABP specifies the knowledge areas to be assessed by the certification examination.
Overview of the Practice Analysis Process
The overarching goal of the practice analysis process was to develop a content outline that truly represents the knowledge and skills required to practice as a competent pediatric hospitalist. All phases of this process relied primarily on feedback from practicing pediatric hospitalists that represented the field.
The PHM practice analysis process consisted of 3 steps: (1) developing a draft content outline, (2) surveying the PHM community to solicit feedback regarding the draft content outline, and (3) using the survey results to make final revisions to the outline and establish examination weights (the percentage of examination questions assigned to each knowledge area).
Phase 1: Development of a Draft Content Outline
Formation of a Panel of Pediatric Hospitalists
A diverse panel of 12 practicing pediatric hospitalists was convened by the ABP to serve as content experts for the practice analysis process. The PHM practice analysis panel size was selected on the basis of industry guidelines and was representative of the distribution of pediatricians in United States.26,27 In addition, panel members were selected with attention to diversity in many areas, including hospital practice setting, specialized practice areas in PHM (eg, surgical comanagement, newborn care and Med-Peds), patient population served, years of clinical experience, geographic location, race and/or ethnicity, and sex (Table 1). ABP staff, which included psychometricians and test development experts, provided guidance and expertise with respect to the practice analysis process.
Development of Content Domains and Subdomains
The panel’s initial charge was to develop a draft content outline, which involved the following steps: (1) identify and define the clinical and nonclinical roles and responsibilities of pediatric hospitalists in the care of hospitalized children, (2) delineate the critical tasks performed by pediatric hospitalists across common clinical settings, (3) identify the objective (ie, “testable”) knowledge required to perform these tasks, and (4) organize this body of knowledge into content domains and subdomains.
The panel identified a comprehensive list of tasks performed by pediatric hospitalists, grouped them into performance domains, and began to identify the full list of knowledge, skills, and abilities needed to perform the tasks within each performance domain. PHM textbooks, the Pediatric Hospital Medicine Core Competencies, fellowship curricula and conference proceedings from the American Academy of Pediatrics, Pediatric Academic Societies, Society of Hospital Medicine, and PHM annual meetings as well as PubMed indexed publications from the past 3 to 5 years were used during this process to ensure that the content adequately captured the critical and common roles and responsibilities of a practicing pediatric hospitalist.21,22,27,28
From this full list of knowledge, skills, and abilities, the body of knowledge that was deemed to be testable and appropriate for a written examination was then organized into content domains and subdomains. The final draft outline consisted of 13 domains or major knowledge areas required for practice. Each domain also contained several subdomains. The subdomains help to ensure that the test questions on the PHM certification examination are sufficiently diverse and comprehensive.
Development of Universal Tasks
Additionally, the panel simultaneously developed a second classification system for test questions to be unique to PHM (referred to as universal tasks), using the ABPs existing framework. The universal tasks reflect the type of applied knowledge that is being assessed on the examination that should be universal to all physicians, specifically categorized by what kind of mental task is being asked of the physician. The 4 universal tasks for PHM, including their operational definitions, are provided below:
1. Core science and pathophysiology: Apply foundational and evidence-based knowledge of diseases and conditions commonly seen in hospitalized children, including normal and abnormal functions of the body and mind in an age-specific developmental context;
2. Epidemiology and risk assessment: Recognize patterns of health and disease and understand the variables that influence those patterns and impact that those patterns may have on patient management principles;
3. Diagnosis and testing: Use available information (eg, patient history and physical examination) to formulate differential diagnoses, choose appropriate tests, and interpret test results to reach a likely diagnosis; and
4. Management and treatment: Formulate a comprehensive management and/or treatment plan, including appropriate consultation and comanagement, reevaluation, discharge planning, and long-term follow-up.
It is important to note that universal task classifications are only applicable to test questions that fall within direct patient care content domains such as medical conditions (domain 1) behavioral and mental health conditions (domain 2), and newborn care (domain 3); they do not apply to test questions that fall outside of direct patient care, such as quality improvement, patient safety, and systems-based improvement (domain 8).
Phase 2: Survey of the PHM Community to Solicit Feedback on the Draft Content Outline
The second phase involved collecting feedback from practicing pediatric hospitalists through an online survey.
The survey included 4 sections: (1) demographic and practice setting, (2) suggested examination weights for the content domains, (3) criticality and frequency ratings for the subdomains within each content domain and, (4) suggested universal tasks weights.
The questions within the demographic and practice setting section included current work hours (full-time and part-time), primary area of clinical practice (PHM, another specialty or subspecialty, etc), years of PHM experience, clinical practice setting (university-based hospital, community hospital, etc), residency and fellowship training experience, and percentage of time spent in direct patient care, administration, research, or medical education. These demographic and practice setting questions were used to evaluate the representativeness of the respondents and see if ratings for the domains and/or subdomains and universal tasks were different across subgroups.
In section 2 of the survey, respondents were asked to provide suggested examination weights for each of the 13 content domains to determine the percentage of examination questions from each domain. In section 3, however, respondents were asked to provide criticality and frequency ratings at the subdomain level to help identify any subdomains that should be removed or consolidated. The criticality rating was used to assess the level of harm that would result from a lack of knowledge in a subdomain, whereas the frequency rating was used to identify how often knowledge and/or skill in that subdomain was needed to practice as a pediatric hospitalist. Criticality and frequency are 2 of the most widely used rating scales in practice analysis surveys.28,29
To maximize the response rate and minimize survey fatigue, respondents were randomly divided into 2 groups. Group 1 was asked to rate the subdomains within domains 1 through 3, whereas group 2 rated the subdomains within domains 4 through 13.
In the fourth and final section of the survey, participants were asked to provide suggested examination weights for the universal tasks, similar to how suggested examination weights were provided for the 13 content domains in section 2 of the survey. Only the group 1 participants (ie, participants who were assigned to domains 1–3 in section 3 of the survey) completed section 4 because only the test questions within domains 1 to 3 receive universal task classifications.
The survey was first piloted with a convenience sample of 47 PHM fellowship program directors who also provided feedback on survey design and ease of use. After minor formatting revisions, the survey was disseminated to a large but targeted sample of diplomates. Specifically, survey invitations were sent to all board-certified general pediatricians in the ABP database who had indicated through other ABP surveys (eg, examination registration and maintenance of certification enrollment) as having practiced in a hospital setting within the past 5 years. Of the >70 000 certified general pediatricians in the ABP database, 12 010 were identified as potential pediatric hospitalists and, therefore, invited to participate. In addition to the 12 010 emailed invitations, an open call to participate was sent to the American Academy of Pediatrics Section of Hospital Medicine LISTSERV, with instructions to contact the ABP directly with expressed interest in survey participation. The survey was open for a period of 4 weeks, with a reminder e-mail sent to all nonresponders with 1 week remaining.
This survey approach reflected the desire to be as inclusive as possible and ensure that all pediatric hospitalists were given the opportunity to provide feedback. It was acknowledged at the time that some pediatricians who received the survey invitation may not be practicing pediatric hospitalists and would likely not respond because the e-mail invitation clearly stated that this survey was only seeking input from practicing pediatric hospitalists. Additionally, to ensure that respondents were currently practicing as pediatric hospitalists, responses to the practice setting questions in section 1 of the survey were used to remove nonpediatric hospitalists from the subsequent survey analyses (Table 2).
Phase 3: Finalizing the PHM Content Outline and Establishing Examination Weights
In the third and final phase, the panel reviewed and discussed the survey results to make final revisions to the content outline and establish the examination weights.
In total, 1449 individuals completed the survey; 800 self-identified as full-time pediatric hospitalists with primary practice in PHM versus 649 whose primary practice was a non-PHM specialty. Given the constrains of a new specialty with a lack of an available database of practicing PHM specialists, the panel felt that those with PHM as their primary practice were most representative of the broader field, given their distribution across clinical practice settings, years of practice, and weekly hours worked (Table 2). Additionally, secondary analyses were done for full-time versus part-time (at least 25% clinical) and community hospital versus noncommunity hospital (Table 3).
The survey response rate was 12.1% (1449 of 12 010) of the total invitations sent to the broad distribution group noted above. However, we anticipated the number of practicing pediatric hospitalists to range from 3000 to 5000.31,32 Using this estimate and the responses of those self-identifying as the full-time pediatric hospitalists results in a response rate of 15% (800 of 5000) to 25% (800 of 3000), which is considered within the typical range of response rates for a lengthy practice analysis survey like ours.33 Overall, the practice characteristics of the PHM respondents were deemed to be sufficiently representative of the field, and, therefore, the panel was confident that the information collected through this survey would be useful for making decisions about the PHM content outline (Table 1).
To finalize the content domains and subdomains, the panel reviewed and discussed the survey results pertaining to the domains and subdomains. Specifically, the frequency and criticality ratings of the content subdomains were reviewed in conjunction with respondents’ open-ended comments that identified knowledge areas that should be added or removed from the outline. A small number of subdomains received relatively low frequency ratings, but, in all cases, the knowledge was deemed to be critical enough to remain in the outline. For example, needle thoracentesis (subdomain 5.I) was the lowest rated subdomain, with an average frequency rating of 1.3, indicating that knowledge in that area was required relatively infrequently for the practice of PHM. However, the average criticality rating was 2.4, indicating the level of harm that would result from a lack of knowledge in this area was somewhere between “slight harm” (rating of 2) and “moderate harm” (rating of 3). The panel discussed these results and decided that there were some knowledge elements associated with that procedure (eg, indications and contraindication and risks and benefits) that would be reasonable to be assessed on the PHM examination. Ultimately, no content subdomains were removed on the basis of the criticality and/or frequency ratings (Table 3).
Only 3 modifications were made to the outline on the basis of the open-ended comments. First, phlebotomy was removed from the outline, whereas inflammatory bowel disease (subdomain 1.E.12) and circumcision (subdomain 5.O) were added. Finally, the panel used the survey results to establish the examination “weights” for both the content domains and the universal tasks. A weight is the percentage of examination questions that would be assigned to each domain. Additionally, for large domains like domain 2 (common medical conditions), weights were also specified for the subdomains to ensure an appropriate distribution of test questions. The panel’s final decision regarding the examination weights for the content domains and universal tasks are presented in Tables 3 and 4, respectively.
Overall, the domain weights were similarly rated by those practicing in community versus noncommunity hospitals, although some statistically significant differences were found (Table 3). For example, there was no significant difference between the suggested weights for the medical conditions and behavior and mental health conditions (P > .05). There were, however, statistically significant differences in the suggested domain weights for several other content areas. For example, community hospitalists suggested higher weights for newborn medicine, transitions of care, and medical procedures (P < .05). Noncommunity hospitalists, on the other hand, suggested higher weights for domains such as children with medical complexity, quality improvement, patient safety and systems-based improvement, teaching, and education (P < .05). Of note, quality improvement, patient safety, and systems-based improvement were rated highly by both groups, although significantly higher by noncommunity hospitalists (P < .05). It is worth noting that although some statistically significant differences were found between community and noncommunity pediatric hospitalists, the relative importance of each domain (based on weights) was highly consistent between these groups, which supports the use of a single outline for all practicing pediatric hospitalists.
Universal tasks, although not applicable to all domains, were rated similarly by both community and noncommunity pediatric hospitalists, with the exception that noncommunity hospitalists suggested slightly higher weights for epidemiology and risk assessment (P < .05) (Table 4). Both groups agreed that diagnosis and evaluation (universal task 3) and management and treatment (universal task 4) should represent a vast majority of examination questions within relevant domains at ∼70% combined. Additional analysis of full-time versus part-time practicing pediatric hospitalists shows no statistically significant difference in ratings for both examination weights and universal tasks (data not shown). The final content outline is available at https://www.abp.org/sites/abp/files/pdf/hospital_medicine_content_outline.pdf.
Certain limitations should be noted. Because of the lack of an available database of practicing pediatric hospitalists, we had to rely on survey respondents to self-identify as pediatric hospitalists versus general outpatient pediatricians or pediatric specialists with hospital-based work. Therefore, it is possible that the survey results do not truly represent the population of practicing pediatric hospitalists. Additionally, a nonresponse bias may factor into survey response. Finally, in this process, self-defined specialists in the field were surveyed, which may have introduced voluntary response bias.
In this article, we describe the practice analysis process used to develop the PHM content outline for the inaugural PHM board certification examination. Given PHM was a new field, the content outline was vetted by the community of self-identified practicing pediatric hospitalists who were broadly representative with respect to practice setting, years of practice, fellowship training and administrative, leadership, and research responsibilities. The process used to develop the content outline for this inaugural PHM certification examination was structured, engaged the PHM community at large, and used an iterative process to refine the final product. Overall, there was concordance in ratings of content domain and universal tasks between community and noncommunity hospitalists. The areas of significant differences, although limited, may reflect differences in practice setting between these groups. Moving forward, the ABP will use a similar process at regular intervals to update the PHM content outline to ensure its continued relevance to the practice of PHM.
Drs Mittal, Shah, O’Toole, Fisher, and Dwyer conceptualized and designed the study, drafted the initial manuscript, and reviewed and revised the manuscript; Drs Rauch, Carlson, Percelay, Black, Lee, Russo, and Woods designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Drs Chase and Quinonez conceptualized and designed the study, coordinated and supervised data collection, and critically reviewed the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
Dr Black was a pediatric hospitalist at Children’s National Hospital during practice analyses process.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.