Putting the Mix in Mixed Methods Research

MMR combines and integrates qualitative and quantitative research approaches to produce findings that represent more than a sum of the individual approaches. As Creswell and Plano Clark highlight, MMR allows a study, “To provide a more complete picture by noting trends and generalizations as well as in-depth knowledge of participants’ perspectives”.¹  MMR has become a third major research approach, alongside quantitative and qualitative research.²  Given the strengths and applications of MMR, pediatric hospitalists engaged in reviewing, interpreting, and generating research could benefit from understanding the key features of MMR and their application. We recommend starting with the National Institutes of Health’s definition of MMR as a research approach or methodology that³ :

1. Utilizes more than 1 method

2. Focuses on questions that call for real-life understanding, contextual knowledge, and multilevel perspectives

3. Employs rigorous quantitative and qualitative research

4. Intentionally integrates or combines these methods to draw on the strengths of each

5. Frames the investigation within philosophical and theoretical positions

Potential benefits of using MMR approaches vary depending on the research question. For example, to investigate the reasons why an intervention works in one hospital, but fails in another, researchers can use an MMR study to identify and examine features of the contextual determinants (eg, organizational culture, physical environment) of success.⁴  Examining contextual elements both qualitatively (providing nuance and depth) and quantitatively (providing a broad view of measurable characteristics) and exploring the interplay between these data can comprehensively identify potential barriers and facilitators relevant to successful implementation.

Investigators should be aware of several unique features of MMR studies. The intent to employ a MMR design should be present from study inception, and the design of an MMR study should reflect the research purpose. The design is defined as a procedure for conducting both qualitative and quantitative methods to collect, analyze, and combine data to accomplish a study’s objective. Plano-Clark and Creswell anchor the MMR approach with 3 core designs: explanatory sequential, exploratory sequential, and convergent.¹  An explanatory sequential study design starts with quantitative methods and follows with qualitative methods. Frequently, this sequence occurs when a study team seeks to further explain qualitatively what was found quantitatively. In an exploratory sequential study design, a qualitative approach precedes quantitative data collection and analysis. The qualitative findings inform the design of a quantitative instrument, intervention, or trial. Finally, a convergent MMR design involves simultaneous qualitative and quantitative data collection and often analysis.

The reasoning for utilizing MMR methodology, termed rationale, should be clear and well-described. A critical component of the design in MMR is the timing of the qualitative and quantitative data collection and analysis strategy, termed sequence, as well as the overarching emphasis in weighing both methods to produce findings, termed priority. MMR studies may have the qualitative and quantitative components performed consecutively (one after the other, termed sequential) or at the same time (in parallel, termed concurrent). For example, if a study aims to implement an intervention, they may first qualitatively assess facilitators and barriers to implementation and then quantitatively test the intervention’s uptake and efficacy. In contrast, a study aiming to understand an intervention’s effect may first quantitatively test intervention efficacy and then qualitatively assess contextual factors that determined why the intervention succeeded or not. Alternatively, a study may simultaneously collect and analyze qualitative and quantitative data to combine and compare the findings obtained through the 2 approaches. In any of these study designs, the qualitative or quantitative components may be more heavily weighted. However, in all instances the 2 components are deliberately integrated, or combined, which can occur at any phase of the study including data collection, analysis, and interpretation.

Integration is an intentional comparing and bringing together of the qualitative and quantitative data collection and analysis and is the heart of MMR.⁵  Integration enables an MMR study to produce findings of more value than those produced from isolated quantitative and qualitative studies.⁶  The key to integration is considering the study’s points of interface (or places where qualitive and quantitative are connected).

There are 4 basic types of integration in a MMR study, as described by Fetters, Curry, and Creswell.⁶  One approach is merging, an approach to integration where 2 data sets are combined for analysis. A second approach is connecting, which links methods by linking the analysis of 1 data set to the collecting of a second data set through sampling (eg, qualitatively interviewing a subset of quantitative survey respondents). Through merging, the qualitative and quantitative components lose their individual identity to an extent. Whereas with connecting, data retain their identity and features while still being combined. A third integration approach is building, where 1 database informs subsequent data collection rather than having a direct connection. A fourth integration approach is embedding, where either connecting, merging, or building occurs throughout study phases as qualitative and quantitative data are collected at various points within multiple procedures. All approaches to integration ultimately allow these qualitative and quantitative data to become something that is more than the sum of their parts.

Below, we present published studies in pediatric hospital medicine as examples for how the 3 core MMR study designs are put into practice. For each, we describe how MMR and their sequence of methods aligned with their study design purpose and how these study teams performed integration, or the mixing of qualitative and quantitative data. Although not an exhaustive list, we also use each study design to highlight some available MMR tools: joint displays for representing MMR findings; implementation matrices for mapping an MMR study’s research questions with its approach, data collection methods, and expected outcomes; and procedural diagrams that use flowsheets or diagrams to show the steps of the MMR research process and expected products.

Shanley et al aimed to first quantify the proportion of dehydration-related admissions that were preventable and then explain how and why those admissions could be prevented.⁷  They first identified the prevalence of avoidable dehydration-related admissions (quantitative). Next, they interviewed caregivers and providers to explore how and why these admissions may have been avoidable (qualitative).

Shanley et al’s data integration occurred at multiple levels in their explanatory sequential design. After preliminary completion of quantitative analysis, a sample of patients admitted with a preventable condition and their care team were recruited for interviews. The purpose of this strategy was to qualitatively explain why a preventable admission occurred. This is an example of data integration through connection. Once they had finished data collection, there was also a final stage of integration through merging of both quantitative and qualitative data. In Fig 1A, we modified a study figure that functioned as a joint display. A joint display helps illustrate how merging was handled in publication. Study participant quotes (qualitative) were stratified by agreement or disagreement of the interview participant (quantitative) about the preventability of dehydration-related hospitalizations.

Lockwood et al sought to describe rapid response system characteristics in pediatric hospitals using an exploratory sequential MMR design.⁸  Their study used cognitive interviews (qualitative) to create and validate a survey disseminated to pediatric hospitals examining hospital variability in rapid response systems (quantitative). Here, qualitative methods inform the design of a quantitative instrument.

In Fig 1B, we created an implementation matrix that summarizes Lockwood’s published methods. The elements of this matrix include the sequence of qualitative and quantitative methods and descriptions of each element and relevant study procedure. To optimize their final survey, the authors merged 3 rounds of qualitative data collection and analysis into the final version. They also connected their quantitative and qualitative data by interpreting their quantitative survey results in light of the earlier qualitative data collection and analysis.

Curran et al designed and executed a complex multistage mixed methods study that at its core is a convergent MMR study.⁹  They used a case-study approach to develop regional recommendations for the transition to home for hospitalized children with medical complexity. In phase 1, they collected qualitative (semistructured interview) and quantitative (scores from surveys) data concurrently, independently analyzed both data types, and then merged these data from each case study to describe the experience of patients and families during transition. Figure 1C shows a procedural diagram of how the authors used a convergent MMR study design in their first study phase to simultaneously embed qualitative and quantitative data collection and then merge them during analysis.

MMR explanatory study designs may also use surveys with open-ended questions. When utilizing MMR in this manner, we would encourage readers to consider there may be challenges in optimizing qualitative, quantitative, and MMR data collection and analysis. Response rates to open-ended questions in quantitative surveys tend to be low, biased, and terse, limiting purposive sampling and valid inferences regarding themes.¹⁰  Responses to open-ended questions cannot be further probed, substantiated, and clarified, as it is done in a semistructured interview. Researchers should examine response rates, bias (based on responses to quantitative questions), and data quality when aiming to integrate quantitative and qualitative responses particularly with nonface-to-face surveys.¹¹ 

In pediatric hospital medicine, many research questions could benefit from further depth and breadth of understanding. Though MMR is ideally suited to clarify the contexts and explain the complexities, not all studies require mixed methods approaches, and resources are an important consideration. MMR may take longer or use more resources than a single method study. Lockwood et al took multiple phases to appropriately ensure content- and response-process validity of the designed survey before administration of the survey, data collection, and analysis.⁸  Convergent study designs may take less time longitudinally than other MMR designs since data collection occurs concurrently; however, they are not appropriate for all research questions and often require more financial resources and parallel expert teams. Curran’s study had multiple and simultaneous phases that included a consensus meeting.⁹ 

Studies employing a MMR approach could play a crucial role in the field of pediatric hospital medicine to accelerate improvements in clinical outcomes and bring research interventions to our patients and families (Table 1). However, the quality of the MMR study is dependent on the rigor of all 3 components: qualitative, quantitative, and mixed methods. There are ample opportunities for self-guided learning, including primary literature, textbooks, and workshops offered by conferences and universities. To ensure the rigor of a MMR study, you and/or your team may consider consulting or partnering with experts in qualitative, quantitative, and MMR methodologies.