BACKGROUND

Adolescents with chronic disease engage in sexual activity similar to their healthy peers, with generally low utilization of contraception. Adolescents with rheumatic diseases prescribed teratogenic medications may be at risk for unplanned pregnancy.

METHODS

Using structured quality improvement (QI) methods with behavior economic (BE) principles, a multidisciplinary team aimed to implement pregnancy prevention processes for females on high-risk medications. We leveraged BE-inspired interventions including improved accessibility of consents, utilizing distinctly colored consent forms, real-time reminders, peer comparison, and audit and feedback. Our primary aim was to increase the number of days between pregnancies for postmenarcheal females followed in rheumatology clinics who were taking teratogenic medications. Phase 1 focused on annual consenting of female adolescents prescribed teratogenic drugs. Phase 2 emphasized sexual history screening and pregnancy prevention planning at every clinic visit for females ≥12 years on teratogenic medications.

RESULTS

We increased the days between pregnancies for female adolescents prescribed teratogenic medications from 52 days to >900 days by using QI methodology with BE strategies. In phase 1, annual consents for postmenarcheal patients on teratogenic medications improved from 0% in 2017 to 95% in 2021. In phase 2, sexual history screening and pregnancy prevention planning at every clinic visit improved from 2% in 2019 to over 78% in 2021.

CONCLUSIONS

A multiphase, multidisciplinary QI project with integration of behavior economic strategies can improve patient and caregiver counseling to prevent unplanned pregnancies for adolescents on teratogenic medications.

What’s Known on This Subject:

Teratogenic medications are used to treat pediatric rheumatic diseases. Adolescents are often sexually active, yet use of contraception is low, which increases the risk of unintended pregnancies. Quality improvement methodology and behavioral economic strategies can address this gap in care.

What this Study Adds:

Our study highlights a multidisciplinary approach in sexual history screening and pregnancy prevention counseling. We overcame challenges and changed behaviors utilizing both quality improvement methods and various behavioral economic strategies.

Teratogenic medications may result in fetal malformations or pregnancy loss if taken during pregnancy. Teratogenic exposure occurs in about 6% of US pregnancies.1  The US Food and Drug Administration classifies several medications used to treat rheumatic diseases as teratogenic, placing women at risk for negative pregnancy outcomes.2  Contraceptive use for adolescent females on teratogenic drugs is low.3,4 

Adolescents with chronic illnesses, including rheumatic diseases, tend to be as sexually active as healthy peers.5  The US Food and Drug Administration developed Risk Evaluation and Mitigation Strategies (REMS) for many high-risk medications. Some teratogenic drugs prescribed by rheumatologists do not have REMS. Counseling, referral, or contraceptive prescribing occurs in less than a third of females aged 14 to 25 prescribed teratogenic medications.4  A retrospective chart review of 200 female adolescents prescribed methotrexate in our pediatric rheumatology clinics from June 2016 to May 2017 showed only a third of visits had documentation of methotrexate-related reproductive health risks.6  One-third of these patients received a prescription for hormonal contraception (10% on long-acting, reversible contraception) and 3 (1.5%) unplanned pregnancies occurred on methotrexate.

Using the Model for Improvement,7  a multidisciplinary team proposed to improve pregnancy prevention efforts for females on high-risk medications. Studies support using quality improvement (QI) strategies to address gaps in care by increasing the frequency of teratogenic medication pregnancy risk-counseling and screening.810  Various behavioral economic (BE) strategies have also been effective in changing behaviors.1113  BE incorporates economics and psychology to promote interventions leading to social change.11,12  BE-derived practices include implementing low intensity “nudges” to enhance clinician and patient behavior by providing peer performance data and information at critical time points.

Our QI efforts focused on prevention of pregnancy in teens taking teratogenic medications prescribed by rheumatology providers in our clinics. The primary aim was to increase days between pregnancies in postmenarcheal females prescribed teratogenic drugs from an average of 52 days to an arbitrary goal of >200 days. We created a 2-phased approach to attain this goal by implementing new processes for teratogenic education as well as sexual health screening and pregnancy prevention planning. The first phase focused on the use of an annual teratogenic medication consent form, with a specific aim of increasing consent completion by postmenarcheal females prescribed teratogenic medications from 0% to 90%. The second phase focused on documentation of sexual history screening and pregnancy prevention planning for females ≥12 years prescribed teratogenic medications, with a specific aim of increasing documentation from 2% to 80%.

Two pediatric rheumatology clinical sites at an urban, quaternary care, pediatric center with combined >6000 annual patient visits participated in the project. Seven pediatric rheumatology attending physicians, 2 nurse practitioners, and 3 rheumatology fellows provided care to patients up until the age of 18 to 21 (timing of transition to adult care per patient or family preference). The QI team consisted of pediatric rheumatology attending physicians and fellows, pharmacist, adolescent medicine attending, QI coordinator, administrative assistant, and nursing representatives.

The QI team developed key driver diagrams specific to the goals of each phase. Key drivers included workflow development, staff education, and patient awareness (Figs 1 and 2). We leveraged BE-inspired interventions, including improved accessibility of consents, utilizing distinctly colored consent forms, real-time reminders, peer comparison, and audit and feedback.12,1416 

FIGURE 1

Key driver diagram for phase 1: annual teratogenic medication consents.

FIGURE 1

Key driver diagram for phase 1: annual teratogenic medication consents.

Close modal
FIGURE 2

Key driver diagram for phase 2: sexual history screening and pregnancy prevention planning.

FIGURE 2

Key driver diagram for phase 2: sexual history screening and pregnancy prevention planning.

Close modal

Phase 1, use of annual teratogenic medication consents, was initiated in January 2018 with the implementation of teratogenic counseling for mycophenolate mofetil (MMF) by utilizing the REMS Patient-Prescriber Acknowledgment form for females ≥14 years.17  Patients, legal guardians (if the patient is less than 18), and providers signed the form acknowledging risks to the fetus if the patient became pregnant while taking MMF and the need for pregnancy testing, using reliable contraception, and immediate provider notification of pregnancy.

The form was made easily available in clinics and was printed on colored paper to improve visibility. Prescribers or a pharmacist reviewed and signed the form before starting treatment, annually, or with any medication changes. Although the REMS program did not suggest a frequency, we determined annual consent was appropriate because of the variability of sexual activity. In September 2018, we created a teratogenic consent form for leflunomide and methotrexate that was modeled from the MMF form. The Patient-Prescriber Acknowledgment form and leflunomide and methotrexate teratogenic consent forms will be hereafter referred to as consents. Signed consents were dated and scanned into the electronic health record (EHR).

Simultaneously, we created a manual previsit planning (PVP) process. At each visit, nurses reviewed the date of last consent, flagged patients with expiring consents, and supplied the form to the provider or pharmacist during the visit. With subsequent Plan- Do-Study-Act cycles, we expanded the period of flagging expiring consents from 30 to 60 days (June 2020) and then to 90 days (March 2021). In September 2020, we implemented an automated EHR PVP report that used data entered into the sexual history screening and pregnancy prevention planning smartform implemented in phase 2.

In early 2019, during the planning stage of expansion to include postmenarcheal patients, we discovered inconsistent documentation of menstrual histories. We implemented nursing education to improve collection and documentation of menses.

Phase 2, initiated in September 2019, focused on sexual history screening and pregnancy prevention planning at each visit for females ≥12 years. At this time, the rheumatology division joined a hospital QI collaborative focused on contraceptive access for adolescents. Through this partnership, we developed an EHR smartform for providers to document sexual history, pregnancy intention, and a pregnancy prevention plan at each visit (Supplemental Table 1). The EHR smartform included orders for screening tests and referrals to adolescent medicine. Providers documented discussion of teratogenic medication use, recommendations for 2 reliable forms of contraception when sexually active, and whether the consent was updated during the visit. The smarform went live in the EHR in April 2019. Official training occurred in June 2019, and we tested the EHR smartform in a pilot population in lupus clinic in September 2019 before expansion to all rheumatology clinics in November 2019. We partnered with adolescent medicine and pediatric gynecology colleagues to provide education to the rheumatology division on obtaining sexual and contraception history to improve comfort level with these conversations. To maintain privacy, parents were asked to leave the room for sexual history screening and pregnancy prevention planning.

In April 2020, accommodating telehealth visits during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, we adjusted to electronic nursing intake and PVP, incorporated verbal consents, and maintained privacy during sexual history and pregnancy prevention conversations by asking parents to leave the room and recommending that patients use headphones during telehealth visits.

We developed funnel charts to deliver provider-level performance feedback in June 2020 (Supplemental Fig 6). In November 2020, we mailed a letter to parents or guardians of females ≥12 years describing our efforts to improve pregnancy prevention counseling within the rheumatology clinics. In December 2020, an EHR alert banner identified postmenarcheal patients on teratogenic medication and provided a link to the EHR smartform. In spring 2021, provider education sessions with adolescent medicine addressed telehealth and privacy concerns.

The primary aim was to increase days between pregnancies for postmenarcheal patients prescribed teratogenic medications within rheumatology clinics. Pregnancies were identified through patient and parent report or from serum or urine testing completed within the health system. The date assigned was the discovery or report date to the department. Eligible patients included females seen in the rheumatology outpatient clinics from January 2017 through December 2021 prescribed methotrexate, leflunomide, or MMF. The initial population included females aged ≥14 years but was expanded to include all postmenarcheal females in October 2019.

The process measure for phase 1, annual teratogenic medication consents, was the percent of patient visits for female patients on teratogenic medications within a given month having a signed consent within the previous 365 days. Eligible visits initially included females aged ≥14 years prescribed methotrexate, leflunomide, and MMF. The denominator was inclusive of all 3 medications even during the MMF pilot phase. We omitted cyclophosphamide because of inconsistencies in extracting reliable data regarding inpatient and outpatient prescription of this medication, and teratogenic counseling was performed by a separate fertility team. Once a consent was signed, a patient was deemed compliant for the following 365 days for any appointments that occurred over that given year. If a patient had multiple visits within a given month, the first visit with a prescriber was included in the data. Data from January 2018 through December 2021 were analyzed, with the baseline period defined as unique females aged ≥14 years prescribed a teratogenic medication from January 2017 to December 2017. In October 2019, eligibility expanded to include all postmenarcheal teratogenic patient visits. When the target population expanded to include all postmenarcheal females, visits with missing menstrual history were assumed to be in the postmenarcheal population during analysis.

The process measure for phase 2, sexual history screening and pregnancy prevention planning, was the percent of patient visits with a completed sexual history form within the past month. Eligible visits included females aged ≥12 years prescribed methotrexate, leflunomide, and MMF. We considered a form used and counted in the numerator if the questions: (1) Any past sexual involvement, (2) would you like to become pregnant within the next year, or (3) birth control plan were completed because this indicated that a conversation on sexual history was initiated, pregnancy intention was assessed, and a plan was developed. If a patient had multiple visits within a given month, the first visit with a prescriber was included in the data. Data from September 2019 to December 2021 were evaluated, with a baseline period from January to August 2019.

To track progress in measures and evaluate the impact of interventions over time, Shewhart control charts were updated and reviewed monthly. A T-chart for pregnancies was updated monthly or with pregnancy discovery. P-charts were used for annual consents and documentation of sexual history screening and pregnancy prevention planning. These control charts identified special cause variation and followed American Society for Quality rules, including the 8-point rule and aggregate point rule.18  The impacts of the interventions were reviewed monthly and summary data were presented to the rheumatology division at least quarterly, supplemented with e-mail updates. Rolling funnel charts for the previous 3 months with provider-level data were distributed on a monthly basis to share peer performance.

Per institutional policy, this QI project was not considered human subjects research and did not require institutional review board review.

From the baseline mean of 52 days between pregnancies from August 2, 2017 to February 23, 2018 (pregnancy dates: August 2, 2017, September 21, 2017, October 4, 2017, February 23, 2018), >900 days have elapsed without a known pregnancy on a teratogenic medication as of December 31, 2021, surpassing our goal of 200 days (Fig 3). Aside from pregnancies in the baseline period, pregnancies occurred on the following dates: May 16, 2018, July 23, 2018, January 11, 2019, February 22, 2019, and June 4, 2019, indicating 3 pregnancies per year before 2020. There was a total of 261 unique females who were prescribed teratogenic medications during a rheumatology office visit from January 1, 2017 to February 23, 2018. We used a period from first incident in 2017 to the start of phase 1 to calculate baseline mean. Of 914 unique females prescribed teratogenic medications during a rheumatology office visit during the entire analysis, the median age was 15.8 years (Q1 = 13.6, Q3 = 18.1; IQR 4.4) at first visit. Before inclusion of all postmenarcheal females in October 2019, the median age was 16.9 years (Q1 = 15.1, Q3 = 18.9; IQR 3.9) and was adjusted to 15.6 years (Q1 = 12.6, Q3 = 18.1; IQR 5.5) after expansion.

FIGURE 3

Days between pregnancies for females on teratogenic medications. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

FIGURE 3

Days between pregnancies for females on teratogenic medications. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

Close modal

During the 12-month baseline period, 629 visits occurred in the rheumatology clinics for female adolescents on teratogenic medications and there were no signed consents. With the MMF consent implementation, the subsequent 15 months showed improvement in consents signed among 25% of eligible visits (Fig 4). From April to November 2019, after PVP implementation, the proportion of consents increased to 83%, and a centerline shift was observed. An additional centerline shift occurred from April to November 2020, with 88% of eligible visits having annual consents. This shift aligned with efforts to strengthen documenting menstrual history, modifications to PVP, and implementation of the sexual history screening and pregnancy prevention planning EHR smartform. A fourth centerline shift occurred from April 2021 to October 2021 after modifications to PVP with consents signed among 95% of eligible visits. The expansion of the population from ≥14 years old to all postmenarcheal females in August 2019 increased the denominator by an average of 10 females per month.

FIGURE 4

Percentage of postmenarcheal females with up-to-date annual teratogenic medication consents. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

FIGURE 4

Percentage of postmenarcheal females with up-to-date annual teratogenic medication consents. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

Close modal

During the 8 month baseline period, 546 visits occurred in the rheumatology clinics for females ≥12 years on teratogenic medications, and 2% of eligible visits had documented sexual history screening and pregnancy prevention planning. Following initial interventions, the centerline shifted between September 2019 and April 2020, with 25% eligible visits with documented counseling (Fig 5). We observed continued improvement after implementing monthly provider feedback with funnel charts in June 2020, resulting in a second centerline shift between May to December 2020, with 46% of eligible visits with documented counseling. Improvement continued after an EHR alert banner was initiated in December 2020, resulting in a third centerline shift from January to July 2021 with 78% of eligible visits with documented counseling.

FIGURE 5

Percentage of female patients >12 years on teratogenic medications with documented sexual history screening and pregnancy prevention planning. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

FIGURE 5

Percentage of female patients >12 years on teratogenic medications with documented sexual history screening and pregnancy prevention planning. Dotted lines indicate an established process stage mean for process stage shifts, whereas solid lines show the continuing tracking of that previous process stage mean compared with updated data.

Close modal

Poor pregnancy outcomes from teratogenic medication exposure in adolescents drove our focus to prevent conception while on these medications.1921  The use of the Model for Improvement and incorporating BE strategies improved patient and guardian counseling regarding teratogenic medication risks and reduced the number of pregnancies exposed to teratogenic medications. We achieved our aim of increasing the days between pregnancies for patients on teratogenic medication by attaining >900 days, surpassing our goal of >200 days. We successfully accommodated telehealth visits during the SARS-CoV-2 pandemic. We maintained an annual completion rate of >90% for 9 months and, of those, 7 consecutive months were >95%. Documentation of sexual history and pregnancy prevention counseling occurred in >80% of visits for 8 consecutive months.

Key contributing strategies to our improvements included addressing provider knowledge gaps, consent form accessibility, forgetfulness, and time constraints during appointments, similar to published findings.8  Provider education addressed gaps in clinical practice, highlighting the need for counseling about teratogenic medication risks and the necessity for screening sexual history and pregnancy prevention planning. We partnered with a hospital-wide QI collaborative addressing adolescent contraceptive use and provided education about best practices to approach sexual history discussions. We used colored paper for annual consents to improve visibility among other paperwork in clinic. We leveraged real time reminders, both manual PVP and automatic EHR banners, and a smartform to efficiently document sexual history screening and pregnancy prevention planning to overcome forgetfulness and time constraints.

Similar to previously published literature, the PVP process was an integral component of clinic team preparation and was responsible for the consistent improvement in phase 1.8,22,23  The nurses identified individuals eligible for annual consents and provided the colored form to the prescriber to review with the patient. Incorporating nurses into the PVP process helped to remove the burden of identifying eligible patients from the prescriber. A buffer window for flagging eligible patients for consents was found to be beneficial as this allowed multiple attempts to capture consents before expiration. Our PVP process incorporated 3 BE-influenced interventions including (1) improved accessibility of consents, (2) utilizing distinctly colored consent forms to improve visibility, and (3) providing clinicians with real-time reminders.14,15 

In phase 2, we noted rapid improvement in sexual history screening utilizing 2 primary BE strategies: (1) peer comparison leading to social norms and (2) audit and feedback. We shared provider-level performance in funnel charts leveraging a BE strategy based on social norms, ie, the idea of individuals changing behaviors to align with a group.2426  Another effective BE strategy, audit and feedback, was used as noted in a Cochran review by Ivers et al.16  This strategy was effective in our project in the setting of low baseline performance, and frequent feedback from a source hierarchically equivalent or higher.16  We combined both strategies by providing monthly feedback with support from our divisional leadership. Through peer comparison utilizing provider funnel charts, high performers shared best practices to drive better performance across the division. These BE strategies coupled with rapid, iterative Plan-Do-Study-Act cycles potentially explain the effectiveness of our interventions.

Phase 2 of the project, initiated in 2020, had the greatest impact in increasing days between pregnancies by focusing on preventing pregnancy through frequent conversations with adolescents and caregivers. Initially, our efforts to improve safe use of teratogenic medications in females focused primarily on annual counseling, which is similar to established literature.8  Cooper et al was able to implement a successful process that included universal pregnancy screening at routine visits.8  We decided against this approach because pregnancy may occur between clinic appointments, added costs of testing, and concerns over reduced reliability of point-of-care pregnancy screenings early in the pregnancy.2729  Our EHR smartform facilitated standardized conversations about sexual history and pregnancy prevention planning. Integrated orders within the smartform streamlined processes for sexually transmitted infection screening, pregnancy testing, and referrals for contraception care. An EHR alert banner relieved reliance on providers to identify patients and facilitated utilization of the smartform utilizing BE strategy of real-time reminders. We identified functionality within the EHR to incorporate clinical decision support to identify high-risk patients for pregnancy based on objective data, such as last menstrual period and recent intercourse. As a next step, we are considering adopting targeted testing to capture potential unintended pregnancies while balancing the concerns previously described for universal pregnancy screening.24 

Our aim of days between pregnancies was reliant on patient and guardian report of pregnancy. It is possible that some pregnancies, spontaneous abortions, or elective abortions were not disclosed to the rheumatology team. We acknowledge that adolescent pregnancies may be a rare event and the pandemic may have changed teen sexual behavior, so a longer follow up period may be needed to confirm the effectiveness of our interventions. Limitations to external generalizability include that this is a single-center, single-department initiative. Not all centers have access to a pharmacist, EHR customization, adolescent medicine specialists, or QI coordinators. Implementing QI methodology with BE strategies can apply and be customized to any center. We did not include a balancing measure, such as clinic appointment times. During the same period, a process improvement effort was undertaken to improve clinic flow, which could have skewed results.

We faced challenges to data availability because of informatics upgrades and SARS-CoV-2 virus-pandemic related changes adding delays in data analysis and outcome progress. Change fatigue3032  and switching to a predominantly virtual environment for meetings and patient visits added further complexity. The virtual format brought barriers to provider buy-in and comfort level with discussing sensitive topics and patient privacy.33  Education by our adolescent medicine colleagues on topics of telehealth, privacy, and preventing pregnancy was integral to success. Despite telehealth challenges, sexual history screening and documentation was sustained during the pandemic time, indicating that this intervention can be successfully adapted into a virtual environment.

A multiphase, multidisciplinary QI initiative with integration of BE strategies improved patient and guardian education and prevented pregnancies for postmenarcheal females on teratogenic medications. This has been sustainable for over 3 years and has become integrated into clinical practice. Given these successes, our team is working with the hospital to implement similar approaches to teratogenic medication counseling across multiple specialties.

Drs Mruk, Barbar-Smiley, Ardoin and Ms Lemle contributed to conception and design, data review, and analysis of results; Dr Wise performed annual consent entry, conception and design, data review, and analysis of results; Mr McGinnis, Mr Gallup, Mr Mitchell, provided QI data support for analysis of results and revised the manuscript for important intellectual content; Drs Stevens, Driest, Yildirim-Toruner, Oberle, Berlan, and Sivaraman, Ms Jones, and Ms Maher contributed to conception and design of the project, interpretation of data, and revised the manuscript for important intellectual; and all authors approved the final version to be published and approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: No external funding.

CONFLICT OF INTEREST DISCLOSURES: Dr Berlan is a consultant for Merck and Bayer, a Nexplanon Clinical Trainer for Merck and Inc, and receives research support from Merck and Organon. The remaining authors have no financial relationships relevant to this article to disclose.

BE

behavioral economics

EHR

electronic health record

PVP

previsit planning

QI

quality improvement

REMS

Risk Evaluation and Mitigation Strategies

1
Schwarz
EB
,
Maselli
J
,
Norton
M
,
Gonzales
R
.
Prescription of teratogenic medications in United States ambulatory practices
.
Am J Med
.
2005
;
118
(
11
):
1240
1249
2
Pernia
S
,
DeMaagd
G
.
The new pregnancy and lactation labeling rule
.
P T
.
2016
;
41
(
11
):
713
715
3
Schwarz
EB
,
Postlethwaite
DA
,
Hung
YY
,
Armstrong
MA
.
Documentation of contraception and pregnancy when prescribing potentially teratogenic medications for reproductive-age women
.
Ann Intern Med
.
2007
;
147
(
6
):
370
376
4
Stancil
SL
,
Miller
M
,
Briggs
H
,
Lynch
D
,
Goggin
K
,
Kearns
G
.
Contraceptive provision to adolescent females prescribed teratogenic medications
.
Pediatrics
.
2016
;
137
(
1
)
5
Surís
JC
,
Resnick
MD
,
Cassuto
N
,
Blum
RW
.
Sexual behavior of adolescents with chronic disease and disability
.
J Adolesc Health
.
1996
;
19
(
2
):
124
131
6
Berlan
E
,
Hilbert
M
,
Wise
K
, et al
.
Reproductive health risk counseling in adolescent and young adult women on methotrexate therapy for rheumatic diseases
.
J Pediatr Adolesc Gynecol
.
2018
;
31
(
2
):
168
169
7
Institute for Healthcare Improvement
.
Quality improvement essentials toolkit
.
8
Cooper
AM
,
Horwitz
M
,
Becker
ML
.
Improving the safety of teratogen prescribing practices in a pediatric rheumatology clinic
.
Pediatrics
.
2019
;
143
(
4
):
e20180803
9
Shroff
S
,
McNeil
M
,
Borrero
S
.
An innovative framework to improve teratogenic medication risk counseling
.
J Midwifery Womens Health
.
2017
;
62
(
3
):
353
357
10
Hayward
K
,
Haaland
WL
,
Hrachovec
J
, et al
.
Reliable pregnancy testing before intravenous cyclophosphamide: a quality improvement study
.
Pediatrics
.
2016
;
138
(
6
):
e20160378
11
Chang
LL
,
DeVore
AD
,
Granger
BB
,
Eapen
ZJ
,
Ariely
D
,
Hernandez
AF
.
Leveraging behavioral economics to improve heart failure care and outcomes
.
Circulation
.
2017
;
136
(
8
):
765
772
12
Jenssen
BP
,
Buttenheim
AM
,
Fiks
AG
.
Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness
.
Acad Pediatr
.
2019
;
19
(
1
):
4
10
13
Emanuel
EJ
,
Ubel
PA
,
Kessler
JB
, et al
.
Using behavioral economics to design physician incentives that deliver high-value care
.
Ann Intern Med
.
2016
;
164
(
2
):
114
119
14
Vlaev
I
,
King
D
,
Dolan
P
, et al
.
The theory and practice of “nudging”: changing health behaviors
.
Public Adm Rev
.
2016
;
76
(
4
):
550
561
15
Cho
I
,
Bates
DW
.
Behavioral economics interventions in clinical decision support systems
.
Yearb Med Inform
.
2018
;
27
(
1
):
114
121
16
Ivers
N
,
Jamtvedt
G
,
Flottorp
S
, et al
.
Audit and feedback: effects on professional practice and healthcare outcomes
.
Cochrane Database Syst Rev
.
2012
(
6
):
CD000259
18
ASQ
.
About control chart
.
Available at: https://asq.org/quality-resources/control-chart. Accessed August 23, 2021
19
Weber-Schoendorfer
C
,
Chambers
C
,
Wacker
E
, et al;
Network of French Pharmacovigilance Centers
.
Pregnancy outcome after methotrexate treatment for rheumatic disease prior to or during early pregnancy: a prospective multicenter cohort study
.
Arthritis Rheumatol
.
2014
;
66
(
5
):
1101
1110
20
Gromnica-Ihle
E
,
Krüger
K
.
Use of methotrexate in young patients with respect to the reproductive system
.
Clin Exp Rheumatol
.
2010
;
28
(
5 Suppl 61
):
S80
S84
21
Thai
TN
,
Sarayani
A
,
Wang
X
,
Albogami
Y
,
Rasmussen
SA
,
Winterstein
AG
.
Risk of pregnancy loss in patients exposed to mycophenolate compared to azathioprine: a retrospective cohort study
.
Pharmacoepidemiol Drug Saf
.
2020
;
29
(
6
):
716
724
22
Sivaraman
V
,
Wise
KA
,
Cotton
W
, et al
.
Previsit planning improves pneumococcal vaccination rates in childhood-onset SLE
.
Pediatrics
.
2020
;
145
(
1
):
e20183141
23
Savarino
JR
,
Kaplan
JL
,
Winter
HS
,
Moran
CJ
,
Israel
EJ
.
Improving clinical remission rates in pediatric inflammatory bowel disease with previsit planning
.
BMJ Qual Improv Rep
.
2016
;
5
(
1
):
u211063.w214361
24
Dallaire
J
,
Hazell
M
,
Kottke
MJ
.
Implementation of pregnancy checklist into clinic workflow: a quality improvement initiative
.
J Pediatr Adolesc Gynecol
.
2020
;
33
(
5
):
536
542
25
Tang
MY
,
Rhodes
S
,
Powell
R
, et al
.
How effective are social norms interventions in changing the clinical behaviours of healthcare workers? A systematic review and meta-analysis
.
Implement Sci
.
2021
;
16
(
1
):
8
26
Brown
B
,
Gude
WT
,
Blakeman
T
, et al
.
Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research
.
Implement Sci
.
2019
;
14
(
1
):
40
27
Greene
DN
,
Schmidt
RL
,
Kamer
SM
, %
Grenache
DG
,
Hoke
C
,
Lorey
TS
.
Limitations in qualitative point of care hCG tests for detecting early pregnancy
.
Clin Chim Acta
.
2013
;
415
:
317
321
28
Kamer
SM
,
Foley
KF
,
Schmidt
RL
, %
Greene
DN
.
Analytical sensitivity of four commonly used hCG point of care devices
.
Clin Biochem
.
2015
;
48
(
6
):
448
452
29
Cervinski
MA
,
Lockwood
CM
,
Ferguson
AM
, et al
.
Qualitative point-of-care and over-the-counter urine hCG devices differentially detect the hCG variants of early pregnancy
.
Clin Chim Acta
.
2009
;
406
(
1-2
):
81
85
30
Giusti
EM
,
Pedroli
E
,
D’Aniello
GE
, et al
.
The psychological impact of the COVID-19 outbreak on health professionals: a cross-sectional study
.
Front Psychol
.
2020
;
11
:
1684
31
Pappa
S
,
Athanasiou
N
,
Sakkas
N
, et al
.
From recession to depression? Prevalence and correlates of depression, anxiety, traumatic stress and burnout in healthcare workers during the COVID-19 pandemic in Greece: a multi-center, cross-sectional study
.
Int J Environ Res Public Health
.
2021
;
18
(
5
):
2390
32
Forrest
CB
,
Xu
H
,
Thomas
LE
, et al;
HERO Registry Research Group
.
Impact of the early phase of the COVID-19 pandemic on US healthcare workers: results from the HERO registry
.
J Gen Intern Med
.
2021
;
36
(
5
):
1319
1326
33
Shachar
C
,
Engel
J
,
Elwyn
G
.
Implications for telehealth in a postpandemic future: regulatory and privacy issues
.
JAMA
.
2020
;
323
(
23
):
2375
2376

Supplementary data