Laying the Foundation: How to Develop Rigorous Health Professions Education Scholarship Free

Developing a strong and meaningful educational research project is an iterative process. Throughout this process, we recommend that you find content mentors and methodologic experts to support you. In addition, it can be helpful to identify other individuals and resources to support your endeavors (eg, clinical librarians, statisticians, educational grants).

After identifying an area of focus, it is imperative to conduct a thorough literature review to understand if, and how, others have examined the same problem. Of note, it is important to review literature outside your own specialty and even outside of medicine to gain a broader perspective, such as through the PsycInfo or EBSCO databases. As you survey the existing body of research, be alert for conceptual frameworks that may serve as the infrastructure around which to build your project. Conceptual frameworks are similar to blueprints in that they allow you to build on previous educational theory while providing context for your work. Several articles review conceptual frameworks, and although a thorough review of these frameworks is out of the scope of this article, it is critical to consider at least a handful of different frameworks when laying the foundation of your project. Conceptual frameworks can help curtail preconceived ideas or biases and instead anchor your research question, intervention, and outcome(s) in well-established concepts.^1,2 

Once ready to develop your research question, use the FINER criteria (Feasible, Interesting, Novel, Ethical, Relevant) to fine-tune your question.³  Pairing these criteria with results from your literature search can ensure that the question you ask is innovative, meaningful, and perhaps most importantly, that it contributes new knowledge.

The study of existing or newly implemented curricula affords the opportunity to establish evidence-based educational practices that have the power not only to increase learner knowledge and skills, but ultimately to shape patient outcomes. although not in itself a category under study design, the modification and/or introduction of new curricula is a commonly used intervention in health professions education research. As such, it is important to discuss the principles of rigorous curriculum development because they go hand-in-hand with quantitative and qualitative research methodologies. When creating curricular interventions, you can choose from several widely accepted frameworks to guide your program’s development and evaluation. Perhaps the most well-known of these is Kern’s 6-step approach to curriculum development, which we describe here.⁴  Though these steps are represented linearly, they steps can be iterative, with 1 step potentially influencing another.

The first step in curriculum development is that of problem identification: identifying the educational gap. A general needs assessment helps explore, in the literature and through national organizations’ websites, whether others have noted the problem, what current approaches exist for addressing it, and what ideal approaches are/could be.

The second step is an individualized needs assessment to better understand the learners’, faculty’s, and educational leaders’ perspectives. This is a key time to pull in both learners (to cocreate solutions that will best meet their needs) and educational leaders and collaborators (to gain their wisdom and gauge feasibility of implementing the curriculum in their program). Individualized needs assessments can rely on recent data (eg, Liaison Committee on Medical Education/Accreditation Council for Graduate Medical Education surveys, in-training examination scores, patient feedback) and collection of new data as needed, such as from informal or formal focus groups, interviews, and surveys of proposed learners and faculty.

Though it is tempting to jump directly into different educational strategies, it is imperative to develop goals and objectives around which to scaffold the curriculum. Consider how you would like your learners to be different on completion of the curriculum. Goals represent the big picture idea of how the curriculum will improve the learner and educational system. Objectives, on the other hand, reflect specific, measurable aims. These can be used not only to refine your content, but also to inform the educational approaches you will use and the methods by which they will be assessed.⁴  Objectives come in 3 varieties or levels: learner, process, and outcome objectives. Learner objectives identify the distinct knowledge, skill, and/or attitudinal change you hope learners will gain by completing the curriculum. Ideally, each learner objective will link to an educational strategy and then specific, measurable outcome. Bloom’s Taxonomy can be particularly helpful in developing learner objectives rooted in various cognitive domains.⁵  Process objectives describe the manner by which the curriculum is implemented. Educators can use these to measure the steps taken to carry out the educational activities (eg, numbers impacted and educational activities completed). Last, outcome objectives measure the downstream effect of learner engagement with the curriculum, which can range from effects on learners’ professional development to patient health and quality of care outcomes.

The next step is to create educational strategies that are linked to the learner objectives and the objectives’ cognitive levels. It is important to choose educational activities that are appropriate for the learners and setting, keeping in mind learners’ and faculty’s time. As an example, problem-based learning may be an appropriate strategy for achieving cognitive objectives, whereas role-play is a better strategy for realizing affective or skills-based objectives.

The learners and educators that you engaged in step 2 will be valuable partners as you implement your curriculum. Ensure that you have approval to implement the curriculum from the educational director overseeing the learners. Engaging them early and potentially adding them to your study team can aid in obtaining this approval. In addition, ensure you have the resources (eg, faculty, materials) to implement the curriculum successfully. In our experience, it can be helpful to designate the initial release of the curriculum as a “pilot”; this will remind everyone involved that feedback is welcome and makes learners and faculty more open to change.

You can use Miller’s Pyramid of Assessment to guide learner assessment.⁶  This model arranges the attainment of clinical competence in 4 domains within a hierarchical triangle, with lower-level cognitive processes (knowledge and application) at the bottom and higher-level behavioral changes (demonstration of clinical skills and competency) at the upper tiers. Knowledge can be assessed through tests (eg, multiple-choice, true/false), whereas application can be assessed through case presentations, essays, or tests. Demonstration of clinical skills and competency require review of proxies (eg, chart review) or direct observation (eg, with standardized patients, real patients).

The New World Kirkpatrick’s Model, which describes levels of learner outcomes in relation to program evaluation, is a useful guide for developing rigorous curriculum intervention and other research projects.⁷  Starting from least rigorous to most rigorous, the levels are learners’ satisfaction, engagement, and relevance (level 1), change (or commitment to change) in learners’ knowledge, skills, and attitudes (level 2), application of learning (level 3), and impact on mission (level 4). Most educational research projects will incorporate a few of Kirkpatrick’s levels. The more rigorous the level, the higher impact the study has, though it should be noted that higher impact studies often require more resources.

Common approaches for evaluating curricula include a pretest and posttest assessing knowledge, skills, and attitudes, with inclusion of a control group to avoid confounding variables such as maturation bias. Randomization of learners into intervention and control groups further avoids the potential for selection bias and strengthens the design of the study, particularly if the aim is publication. The manner by which you collect data should also be considered carefully depending on what outcomes are being assessed. Commonly used formats include surveys or questionnaires, which may consist of Likert-scale–based and/or free-response questions, individual or group interviews, direct observation, and audits of performance in either a simulated or real clinical setting. When possible, we recommend that an instrument with strong validity evidence that has already undergone peer review be incorporated into the measurement of outcomes. The next sections provide more detail regarding different study designs, focusing on both quantitative and qualitative research, which can be used to apply a scholarly lens to your curricular product.

Quantitative designs allow researchers to characterize associations among variables, illuminate generalizable findings, and make use of large data sets.

A primary design consideration is determining the study goal: is the study descriptive/observational, without manipulation of variables, or is the study hypothesis-testing/explanatory, in which an independent variable is manipulated and the effect on the dependent variable is studied?^8,9  If the study goal is hypothesis-testing/explanatory, study design options range from less rigorous (ie, preexperimental) to more rigorous (ie, quasiexperimental, true experimental). The key feature of more rigorous explanatory designs is the inclusion of 2 or more groups for comparison. The Medical Education Research Study Quality Index (MERSQI) helps provide a guide to assessing the rigor of your medical education quantitative research project.¹⁰ 

Table 1 summarizes the advantages and disadvantages of the study designs related to study goals. Descriptive/observational studies are often more feasible and less resource-intensive but limited to a point in time, whereas more rigorous explanatory designs may be more resource-intensive yet allow for comparison between groups.

Several strategies can be employed to optimize study designs. First, be realistic. If few resources and/or learners are available but the idea is novel, you could adopt a pilot or feasibility approach.¹¹  This may still be publishable even with lower-level outcomes. If the study is not particularly innovative, however, higher-level outcomes are needed or a mixed methods approach. Second, aim to minimize threats to internal and external validity. Internal validity is the extent to which observed effects can be attributed to the independent variable (ie, how much the study results can be trusted). Techniques to enhance internal validity include calculating the sample size needed for statistical power, describing the educational context and intervention in detail, avoiding loss of participants and reporting on nonresponders, standardizing intervention conditions, employing blinded assessors, and using control groups. External validity is the extent to which results are generalizable from the research sample to the larger population. Techniques to enhance external validity include using random or stratified sampling and replicating the study in another context. We recommend trying to avoid the one-group pre-post design because of the associated threats to internal and external validity. Third, understand and describe the psychometric evidence for the surveys and tools employed in the study.¹²  Fourth, select a study design with methods, analysis plan, and desired outcomes in mind to answer the question, “so what?” with regard to the results. Remember that differences measured in attitudes, knowledge, and behaviors in research settings are only that; they may not translate directly to other environments and contexts.

Although a quantitative methodology is used to ask the question “how many” or “how much,” and is largely hypothesis-driven, a qualitative methodology explores the question of “what,” “how,” or the “why” of a phenomenon, and thus may be hypothesis-generating.¹³  Qualitative methodology is beneficial because it encourages learners to share their story and explore complex social phenomena.¹⁴ 

The specific qualitative approach should be chosen based on the purpose of the educational research (Table 2). It represents the overarching philosophy guiding the collection and analysis of data and is distinct from the research methods.¹⁵  Grounded theory, for example, should be used if the educational research is attempting to explain the process, relationships, and theoretical understanding of educational experiences. Alternatively, to describe the lived experiences of participants, a phenomenological approach can be used. Ethnography aims to understand the meanings and behaviors associated with culture-sharing groups.

Qualitative methods include identifying sources of data, designing instruments for data collection, and analyzing the data. Sources of data for qualitative studies include semistructured interviews, focus groups, and observations, among other methods. Open-ended survey questions may also be employed in qualitative research if there is sufficient descriptive data to generate quality findings. It is important to choose the data collection method that is aligned with your methodology and study purpose. Focus groups are beneficial when group interaction may stimulate responses and the topic is amenable to less input from individual respondents. Interviews are best when a topic is too sensitive for group discussion or warrants greater depth of individual responses. Use evidence-based methods to design instruments for conducting qualitative research and conduct cognitive interviewing to refine your instruments before beginning your study. We encourage you to look at examples of qualitative research that use a similar approach and are published in a high-quality health professions education journals as a guide, and to refer to standards for publishing qualitative methods.¹⁴ 

Qualitative analysis allows you to generate interpretations of the phenomenon in question. The overall goal of analysis is to take raw data (transcripts or memos), abstract that data through coding, generate tentative groups or themes, and form an interpretation. The type of qualitative approach will directly influence how the qualitative data are analyzed and the output. For example, thematic analysis focuses on identifying themes and categorization of themes to describe a phenomenon.¹⁶  In grounded theory, the research “moves from the raw data, to themes, to categorization of themes, to identifying relationships between themes, and ultimately to the development of theoretical explanations of the phenomenon.”¹⁷  In phenomenology, classifying codes into themes occurs by describing individual experiences and the essence of a phenomenon to describe “what happened” and “how was the phenomenon experienced.” In ethnography, classifying codes into themes describes the social setting and paints a picture of the setting. This analysis, in turn, generates an interpretation of “how a shared culture works.”¹⁶  As described earlier in this article, conceptual frameworks can, and should, be applied at the study design, analysis, or interpretation stage of qualitative research. Data should continue to be collected until thematic sufficiency is reached, meaning that new concepts are no longer emerging from the data.

Curricula can be published in Association of American Medical College’s peer-reviewed, PubMed-indexed MedEdPortal. In addition, curricula and health professions education research studies can be published in a number of different journals, including both health professions education journals and specialty journals, as original research reports, innovation articles, commentaries, and perspectives. The MERSQI,¹⁰  as noted earlier, and O’Brien et al’s Standards for Reporting Qualitative Research¹⁷  can help you develop your project in the most rigorous way possible, both to improve your level of impact and likelihood of successful publishing.

We are thrilled to see the world of health professions educational scholarship growing and increasing in rigor. Your own research will help others in further developing impactful and meaningful educational interventions that will benefit our learners, faculty, and staff, and ultimately the patients and communities we serve. Table 3 provides key take-home points as you engage in educational research.

The authors thank our many learners and mentors for the ways in which they have helped us expand and refine our thinking about educational scholarship.