Getting Started With Multi-site Research: Lessons From the Eliminating Monitor Overuse (EMO) Study Free

When planning a multisite project – particularly in PHM, which focuses on evaluating as well as improving hospital care systems – one of the first questions to address is which regulatory category the project falls into: quality improvement (QI) or research. This is crucial because QI and research projects require different regulatory approaches (Table 1).⁷  We recommend this determination come before any additional project planning because of downstream implications. According to the US Department of Health and Human Services, research is defined as “a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge.”⁸  The primary focus of research is developing new knowledge instead of integrating new knowledge into routine practice. By contrast, QI focuses on implementing existing knowledge into day-to-day clinical operations and iteratively measuring the results.⁹  Many IRBs have questionnaires or checklists to differentiate between QI and research; we have included 1 in the Supplemental Information (Supplemental Fig 1).¹⁰  Intent to publish alone does not designate a project as research.¹¹ 

For those pursuing a multisite study, a fundamental step is contacting the primary site’s IRB for guidance. Multisite research funded by the National Institutes of Health (NIH) must use a single, central IRB as the primary IRB (sIRB) to review the research. Participating sites in this research must “rely” on the sIRB rather than conduct separate, independent IRB reviews to streamline review and reduce unnecessary administrative burden.¹²  Even for research that is not funded by the NIH, working via a sIRB mechanism is the best practice.

If the PI of a proposed project believes that the project is QI, it is essential to first ask the sIRB if there is an established QI determination process. If the sIRB agrees that the project is QI, they will provide that acknowledgment, and ongoing IRB oversight will not be needed as it does not meet the criteria for research. This process would then need to be repeated at each participating site before study teams can commence their QI efforts.

In some circumstances, a project involves a combination of QI and research. For example, the primary purpose of a multisite dissemination and implementation project might be to help hospitals adopt a proven process (QI). Still, data might be collected more broadly across sites to produce generalizable data on predictors of successful implementation. Some hospital sites might view such work as primarily QI, whereas others might view it as research. For example, in the PRIS-sponsored I-PASS handoff mentored implementation project, the sIRB determined that the project was QI at the level of the participating sites but that the central aggregation and analysis of data were research.¹¹  Each participating site requested a QI determination from their IRB. If the sites’ IRBs concurred, local work was not overseen by the IRB; if the participating sites’ IRBs did not concur and instead viewed the work as research, then their IRBs had to oversee the work. Regardless of the QI versus research determination, all sites needed to establish DUAs to share data with the central site. To navigate these complex challenges, it helps to collaborate with an IRB Director with expertise in how your institution applies these regulations. We will mainly focus on research in this paper, as it is the primary emphasis of our study team.

If an IRB has determined a project is research, it goes through IRB review: exempt review, expedited review, or full review (Table 1).^13–18  Each type of IRB review has specific eligibility requirements defining the type of review most appropriate for the research.

Multisite studies cannot run without the support of a skillful study team of investigators and professional research staff. Each study has distinct staffing needs, so taking a realistic inventory of the operational needs long before it begins is a key step. Although the specific titles may vary by institution, typically, the main research support staff positions include Clinical Research Assistants, Clinical Research Coordinators, Clinical Research Nurses, and Clinical Research Project or Program Managers. Clinical Research Assistants are entry-level research professionals who may not have research experience but can be trained on data entry, recruiting, consenting, and the administrative needs of your study. Clinical Research Coordinators have more extensive research experience and can complete advanced study tasks such as IRB applications, study manuals, and protocols. Clinical Research Nurses are helpful for study procedures that require nursing support (eg, medication administration). Project or Program Managers manage the research team and oversee the budgeting and operational needs of the study. Nevertheless, the site and overall investigator being active and present with their study team is the foundation of a successful study.

Hiring the right people with the right skills can be one of the most consequential decisions you make for your studies. It is important to build a team that aligns with your research group’s core values; identifying the values that define the research team and hiring accordingly is a fundamental step in creating a diverse, unified team. Giving team members opportunities to operate at the top of their skillset and exploring ways to incorporate their scientific interests can motivate them to feel invested in the work and excel.

In addition to study type and regulatory issues, discussion about funding and other support for the work should commence early in the process. Because of the time and professional coordination needed between sites (eg, coordination of regulatory approvals, DUAs, data collection, timing of interventions), most multisite QI and research projects have internal and/or external funding. These funding models vary widely, from federal grants that compensate sites for their participation in research to foundation or industry sponsored projects to multisite QI collaboratives that charge sites to participate, such as those organized through the American Academy of Pediatrics Pediatric Acute and Critical Care Quality Network. Regardless of the model, funding supports the personnel directing and managing the project. An outline of potential basic elements of a multisite project budget is shown in Table 2. Ideally, the amount and type of funding and resources needed should be clear 6 to 12 months before launching the project.

Site engagement and recruitment are foundational when conducting a multisite study. Without engaged, enthusiastic, and responsive study sites, a project may fail to achieve participant recruitment or data collection goals. It is beneficial to develop a thoughtful engagement and recruitment strategy and implement it from the outset, as communicating with external institutions requires going the extra-mile to keep participants engaged and interested. Partnering with existing networks provides access to many potential sites that otherwise might be difficult to access. Initiating contact with sites before protocol submission to understand capabilities and challenges may inform other aspects of the work.

Once a site expresses interest, regular and consistent points of engagement ensure that critical details about the timeline, protocol, roles, and responsibilities are followed. Virtual meetings and webinars can communicate these details and personalize site interactions to strengthen engagement. Although scheduling across time zones can be difficult, offering several opportunities can ensure that sites are present and engaged. It is also helpful to encourage questions during the meeting for clarity. In addition to meetings, accessibility and responsiveness over e-mail are crucial to establishing rapport and addressing potential problems early.

Participating in research or in a project that may lead to direct, tangible benefits for an institution may be sufficient motivation to be involved in research; however, sites may also appreciate additional incentives. For instance, working and networking with senior researchers and/or supporting professional promotion for site investigators at critical points in their careers. Furthermore, study stipends, authorship, substudy leadership opportunities, and continuing education credits are other ways to further engage sites and incentivize participation.

To gauge and track interest in our EMO Trial,^4,5  a brief questionnaire was distributed to all hospitals in the PRIS Network. This served as the study’s primary recruitment source, resulting in more than 60 interested sites within days. Once recruited, all participating physicians could obtain MOC credit, express interest in authorship of future publications, and submit substudy project ideas. This effort led to several substudies being initiated by clinicians at other sites.

Building partnerships with stakeholders both within and outside the central site is instrumental to the success of a multisite study. It takes strong collaborative effort across institutions and departments to make these studies possible. Key stakeholders involved in the study startup and maintenance process include the IRB, DUA contract leads, and business managers. Developing these relationships may accelerate the establishment of regulatory, legal, and financial agreements between sites. When engaging sites, collect contact information for department leaders to establish a relationship as early as possible. The regulatory, legal, and financial procedures associated with starting a new study are unique at each institution, and as a result, these vital approval processes can be long and rigorous. Clearly communicating deadlines and timelines will ensure your study is prioritized appropriately.

Another group of important stakeholders to identify are patient families and the underlying communities that you intend to serve. It is imperative to codesign the study with these stakeholders. During the EMO Trial, we engaged with our institution’s Family Advisory Council to request their insight on language to best explain the trial and national guidelines to patients and families. It was paramount to not only ensure diversity across and within sites, but we worked to ensure equity was integrated into outcome measurement and analysis on a routine basis to identify disparities in baseline data and ensure families were not disproportionately impacted by the implementation.

As site recruitment continues through the study's early phases, regulatory issues remain a primary focus. Tracking each site’s progress toward key study start-up tasks (eg, IRB reliance, DUA execution) and ongoing research efforts (eg, data collected) is essential to managing a multisite project.

Creating an effective and functional tracking system for sites was crucial to the success of the start-up phase of the EMO Trial. Our team used Research Electronic Data Capture (REDCap), a HIPAA-compliant web-based survey platform, and a series of spreadsheets to collect information and monitor site status. To initiate contact, a personalized “welcome” e-mail was sent to each interested study team, along with a link to a REDCap survey with logistical questions, including hospital name, principal investigator (PI) name and contact information, and optional additional contacts. Once this first survey was complete, a second survey was sent for in-depth questions related to IRB and DUA contact information (ie, institution name, Federal Wide Assurance Number, local context details, and signatories). By splitting the survey into 2, we gauged site engagement and interest without requiring immediate steps for the site PI. On average, sites responded to the second survey within 2 weeks.

In addition to the first 2 surveys in REDCap, our study team created a third form to track site progress. This form was connected to each site’s record and tracked milestones throughout the study-startup phase (ie, site IRB approval, signed documents). REDCap allowed us to track everything in 1 database, create reports, and organize the data as needed. REDCap also offered automatic weekly reminders to complete the second survey. Utilizing a dashboard, we monitored and communicated site progress and activation (see next section).

As study activities commence, communication is key for successful data collection and site retention. Various tools can help to maintain necessary communication with sites, including informal question-answer sessions, project listservs, newsletters, and webinars. Other studies have used methods such as teams of sites and site-assigned coaches to help with motivation, engagement, and bandwidth.¹⁹ 

During the start-up phase of the EMO Trial, our study team held office hours 3 days a week, allowing site PIs and study-team members to bring their questions at scheduled times. We also shared newsletters, an effective and productive way to disseminate information (Supplemental Fig 2). For instance, we included flowcharts for clear steps to follow, study and administrative updates, spotlights on different study staff, frequently asked questions, and the site status dashboard to share site progress. This dashboard helped monitor progress across sites with milestones for survey completion, reliance agreement progress (ie, personnel shared with, signatures, reliance portal completion), DUA execution, and more while also serving as a daily, internal tracking method (Supplemental Figs 3–5).

Additionally, the overall PI hosted webinars to communicate more complex information, such as training requirements and IRB education. This was also an opportunity for sites to bring questions and concerns to the central study team and foster connections. The meeting invitation was shared with all sites and study team members, and the webinar was recorded for later distribution and archiving (via unlisted YouTube links).

Proactive and responsive communication can facilitate regulatory approval of relying sites. For a multisite study, it is important to remember that every hospital, state, and country has its own policies and laws that make up the “local context.” It is also important to assist site PIs with IRB applications, as some may not have participated in research before. International sites have different processes regarding reliance agreements and funding mechanisms. These sites should be informed that their application processes will be different and lengthier, often requiring full review by their local IRB. To expedite this process and provide adequate time for institutional review, all materials should be sent to international sites as soon as possible.

Each EMO Trial site’s completed second survey provided the information needed to add them as a relying site to our study protocol via the central IRB system. Subsequently, a “cede letter”—a reliance agreement template from SMART IRB—was generated for each site.²⁰  This document contained information about the study, the site PI, the overall PI, and the signatory for the relying site. These letters were sent to the IRB Director for her review and signature. Once signed, cede letters were distributed to site PIs with instructions to pass it to their IRB and signatory. Many sites were able to sign this cede letter quickly (within days), whereas others took weeks to months because of differing levels of review and approval requirements.

After a site returned the cede letter, the next step was to submit the document and any local context details or changes to the sIRB. Many institutions’ IRBs have their own electronic reliance portal or system; aiding site PIs in navigating these portals can greatly increase compliance. To track the cede review process, we used the site status dashboard.

Many site IRBs had specific questions about the protocol, making it crucial to be accessible and responsive to relying site leads and their IRB staff. This flexibility allowed us to accommodate several sites that could only participate if they opted out of specific protocol components (eg, survey recruitment by sharing staff e-mail addresses with the data coordinating center). The central study team made it clear to site PIs that they were available for 1-on-1 video calls to assist throughout the process, facilitating timely IRB reliance approvals.

As teams complete necessary regulatory steps, various instruments must be developed and maintained. In addition to the study protocol, consent forms (if applicable), and IRB-facing documents, other documents that should be developed and updated include data collection forms, a manual of procedures, and site-specific delegation logs. Easy-to-use, clear, and concise data collection forms are essential to a study’s success as the accuracy and completeness of the data depend on a well-designed form to guide correct data entry within a specialized software and an individual with these skills. The manual of procedures serves as a study reference guide for study personnel that includes an even more granular level of detail beyond the study protocol by specifying key logistical aspects of the study to help data collectors. Finally, the delegation log tracks each study team member’s responsibilities, training, and IRB approval for participation.

IRBs mandate that individuals meet specific training requirements before engaging in human subjects research.¹⁸  Although these requirements may vary slightly at each institution, Human Subjects Research Protections modules in the CITI Program are usually required.²¹  In addition to regulatory training requirements, there are usually study-specific training prerequisites that the overall PI and central study team will dictate for consistent training across sites.

Depending on the grant source, additional training may be required. For instance, NIH-funded clinical trials require study teams complete Good Clinical Practice training, whereas the FDA has specific regulations for investigational drugs and devices.²²  It is necessary to determine training requirements early to avoid delays to the start of the project.

In the EMO Trial, REDCap supported accurate data collection with a wide selection of tools, including sequential completion of specific fields before the form was finalized, branching logic (making specific questions appear only under certain circumstances), and data validation (alerting the person entering data if they have entered a value that is out of range, such as an incorrect date).²³  The delegation log listed every person at each site who completed the study-specific training. Live virtual trainings were held to review the study’s aims and data collection tools extensively.

Contracts and agreements must be in place to allow participating sites to share data with the central site. The DUA covers data exchange across sites. Terms of the DUA are established by the central site and negotiated with participating sites, typically through an institution’s contracts office or legal team. Even when sites agree on the terms, executing DUAs can be a long, tedious process. Therefore, it is important to set strict timelines and follow up frequently. If your study uses a Data Coordinating Center, this task can be outsourced but will still require support from the core study team.

Depending on a study’s degree of risk and/or the preferences of the funder or sIRB, a Data and Safety Monitoring Board (DSMB) may be convened. A DSMB is a committee of independent experts without relevant conflicts of interest who continuously monitor the safety and productivity of a clinical trial. They are experts in the specific research and clinical area of the trial; additional experts may include biostatisticians, ethicists, and others. To ensure an unbiased review of the study and any events of concern, DSMB members cannot be otherwise involved in the study. If at any time the DSMB has safety concerns or believes the research question has been answered, they can recommend early termination of a trial.

Despite the high visibility of large freestanding children’s hospitals, most pediatric patients are hospitalized in nonfreestanding community settings.²⁴  To maximize generalizability and optimize the development of new standards of care and practice guidelines, multisite projects must include community hospitals as research results may vary based on the hospital setting (eg, trials involving implementation of scientific methods and outcome measures). Community hospitals often also serve a more diverse and representative population, which is crucial to ensure a more equitable population in the research. In a perspective piece, McDaniel and colleagues outline key barriers to community site participation in research.²⁵  Two of these barriers are especially relevant to multisite research: (1) much smaller sample sizes than freestanding children’s hospitals, with 1 study showing an average daily census of just 6 hospitalized patients in community pediatric departments; and (2) minimal local research expertise, resources, and IRB experience reviewing multisite clinical research proposals.

In the EMO Trial, we faced challenges in study start-up and site retention when working with community hospitals. Despite the substantial time and effort we devoted to partnering with community sites, some were unable to launch successfully because of challenges and delays in local IRB reliance agreements and/or DUA submission or review. Those that successfully launched still struggled to collect enough observations to qualify to proceed into the next phase of the trial because of very low bronchiolitis patient volumes and limited staff bandwidth to perform data collection. New approaches for community hospital engagement, such as the HEROIC Trials, are needed to reframe the conditions for participation of community sites to meaningfully engage community sites, advance this important research priority for our field, and reduce the risk of health inequities.²⁶ 

Once data collection commences, it is vital to ensure that incoming data are complete and accurate. Within the first few days of data collection, we recommend downloading the data, visually examining it for completeness, and importing it into your statistical analysis software to ensure it is formatted appropriately and appears as expected. Ongoing quality monitoring (ie, remote or in-person monitoring visits, data queries, audits) throughout the study period is also essential since there may be turnover in data collection teams, resulting in new data entry errors, glitches in the data collection forms because of software updates, or other unanticipated changes.

The Data Coordinating Center was involved in data quality monitoring in the EMO Trial. It was responsible for generating reports, developing systems to track and check for duplicate entries, and sending queries to sites to inquire about complex data.

As data collection is ongoing and nears completion, study teams will need to develop ways to share their data. Throughout the study life cycle, study teams may need to update their funders on progress. For example, federally funded projects usually require annual progress reports outlining expenditures, study subject accrual, milestone completion, a narrative review of study progress, and a plan and budget for the following year. There may be other stakeholders who also need to be kept updated, including an external advisory board, supporting research network leadership, and sites-specific stakeholders (eg, clinicians, investigators, administrators).

When study findings are available, it is important to disseminate findings to the scientific and PHM clinical communities and local community stakeholders identified earlier. This may take the form of abstracts at annual meetings, presentations, webinars, or publications. Although 1 or more publications may be expected at the end of a multisite study, there may be interim publications that help to boost authorship opportunities and share important early findings.

In the EMO Trial and the preliminary work that preceded it, we disseminated updates to the supporting research network via research webinar presentations. To communicate with the scientific and clinical communities, we presented at Pediatric Academic Societies and Pediatric Hospital Medicine annual meetings and published a study protocol manuscript and manuscripts reporting key results and individual subanalyses.^5,6 

This paper outlines vital logistical aspects of managing multisite research projects. With careful planning and organization, regulatory and operational support, and site engagement, a skilled central study team can successfully manage a large multisite project. Research networks like PRIS provide unique opportunities to reduce barriers to multisite projects by offering immediate access to over 100 hospitals, each with a site lead interested in conducting multisite research. In the future, additional effort is needed in developing approaches to better include and support community hospitals in multisite research where the context is likely to impact results (eg, implementation-focused trials). Since most pediatric hospitalizations in the United States occur in community settings, even extensive clinical trials will have limited generalizability without community site representation. We recognize that this paper contains a breadth of information; use this as a guide and be sure to lean on your colleagues and mentors for support. Our aim is that this paper facilitates a continuation of these efforts as the field of academic PHM continues its rapid growth.

We thank the Pediatric Research in Inpatient Settings (PRIS) Network Executive Council and Site Leads for their extraordinary support of the Eliminating Monitor Overuse (EMO) study portfolio; and Dr Claire Daniele for her incredible assistance throughout this publication process.