The American Board of Pediatrics (ABP) has been taking steps to ensure its General Pediatrics Certifying Examination (GP Exam) questions do not have a gender or racial/ethnic bias.
“The ABP aims to ensure that all its assessment scores, including those from its GP Exam, are valid and accurate reflections of each test taker’s knowledge,” the group said in a new report, “Identifying and Addressing Item Bias in the General Pediatrics Certifying Examination,” (Dwyer AC, et al. Pediatrics. May 24, 2023).
The GP Exam is a seven-hour test with 335 multiple choice questions. In 2021, the ABP used a two-phase approach to address potential bias based on the way the questions were written. It performed a statistical analysis to determine if some subgroups outperformed others on certain items. It then asked a diverse panel of 12 subject matter experts to review the items.
In the statistical analysis, no items were flagged for bias based on gender. Just under 3% were flagged based on race/ethnicity. The expert panel then recommended about 14% of those flagged items be removed from scoring, a move that was approved by the General Pediatrics Oversight Committee. The items removed represented 0.4% of the total items.
“We were encouraged by this finding because we instruct our item writers (board-certified pediatricians) and staff editors to avoid using language that may be culturally biased or offensive, and this finding is an indication that those efforts are working,” study author Andrew C. Dwyer, Ph.D., director of psychometrics for the ABP, said via email. “… That being said, now that we have started this process, we are learning a lot about how complex and nuanced item bias can be, and more work is needed to determine whether that 0.4% represents an overestimate or underestimate of items that are truly problematic.”
For example, one question that was removed asked how to manage a college student requesting replacement of two lost prescriptions for attention-deficit/hyperactivity disorder along with an increased dosage. The question was removed because one of the possible answers (discuss the possibility he is selling amphetamines) could be viewed as accusatory, which can be triggering for some cultures.
The ABP said it plans to continue reviewing exam questions. It also is improving the process of creating them, using AAP resources such as Words Matter: AAP Guidance on Inclusive, Anti-biased Language and “AAP Perspective: Race-Based Medicine.”
The ABP said these steps “are designed to prevent offensive item content regardless of whether it leads to statistical performance differences.”
Resources
