OBJECTIVES:

Incomplete subspecialty referrals, whether unscheduled or unattended, represent unmet patient needs and an opportunity to improve patient safety and experiences. Our objectives were to describe the rates of appointment scheduling and visit attendance after pediatric subspecialty referral and to examine patient and systems factors associated with scheduled referrals and attended appointments.

METHODS:

We conducted a retrospective review of referrals within a network of 52 primary and urgent care sites from November 2016 to October 2017. We included referrals for children ≤17 years old referred to medical or surgical subspecialists. We examined patient and health systems factors associated with (1) appointment scheduling and (2) visit attendance.

RESULTS:

Of 20 466 referrals, 13 261 (65%) resulted in an appointment scheduled within 90 days and 10 514 (51%) resulted in a visit attended within 90 days. In adjusted analyses, referral to surgical subspecialists was associated with an increased likelihood of appointment scheduling but a decreased likelihood of visit attendance. Compared with appointments scheduled within 7 days, appointments with intervals from referral to scheduled appointment exceeding 7 days were associated with decreasing likelihood of visit attendance (adjusted odds ratio 8–14 days 0.48; 95% confidence interval 0.37–0.61). Patient factors associated with decreased likelihood of both appointment scheduling and visit attendance included African American race, public insurance, and lower zip code median income.

CONCLUSIONS:

Patient and system factors were associated with variation in appointment scheduling and visit attendance. Decreased interval to appointment was significantly associated with visit attendance. These factors represent targets for interventions to improve referral completion.

What’s Known on This Subject:

Incomplete subspecialty referrals represent an opportunity to improve patient safety and experiences. Data on factors associated with variation in appointment scheduling and visit attendance after pediatric subspecialty referral are limited.

What This Study Adds:

Two-thirds of pediatric subspecialty referrals resulted in scheduled appointments, and half resulted in attended visits. Both patient and system factors were associated with variation in appointment scheduling and visit attendance, and time to appointment was strongly associated with attendance.

Referral to subspecialty care is an important task within pediatric primary care and occurs on average at least once annually for every 3 pediatric patients.1  Despite the ubiquity and importance of subspecialty referral, multiple studies report that patients,2  primary care providers (PCPs),35  and subspecialists4,6  find the process of connecting patients with subspecialists to be ineffective and inefficient. Complexities in referral processes and appointment scheduling,2,3  coupled with substantial limitations and variation in the availability of subspecialists,7,8  reduce patients’ access to subspecialty care. As a result, in the United States, 24% of families in need of subspecialty care report difficulty accessing this care.8  Families without ready access to pediatric subspecialists are less likely to receive subspecialty care,7  more likely to receive care from specialists who are not trained in pediatrics,9  and more likely to report emergency department visits.7 

In 2017, the Institute for Healthcare Improvement and the National Patient Safety Foundation released a report recognizing communication breakdown during subspecialist referral as a threat to patient safety.10  The report, along with others, highlighted the sequential steps required to complete referrals and provided guidance on interventions to improve specific steps in the process.2,10,11  Some emerging interventions aim to aid in primary care referral decisions, including referral guidelines12  and electronic consultations.1315  Other interventions, such as electronic referrals, aim to bypass barriers in the scheduling process, thereby improving appointment scheduling rates.16  Others aim to address scheduling issues by increasing subspecialist capacity, including increased use of advanced practice providers in specialty care.17,18  Finally, to improve visit attendance, systems and payer strategies range from reminders to telemedicine to transportation assistance.

The degree to which these interventions will address barriers to subspecialty care depends in part on an understanding of the overall unmet need for care, the steps in which referral processes break down, and whether these roadblocks vary by patient or system factors. The largest pediatric study to date in this area focused on 577 referrals made from 2008 to 2009 from 2 community health centers.19,20  Other examinations of pediatric subspecialty care focus either on rates of referral21  or appointments attended,1,22  which contribute to an understanding of referral or use patterns but cannot fully capture referrals that go uncompleted. Thus, to guide further interventions improving subspecialty care, there is a need for updated analysis from a larger sample of primary care settings focusing on how patients move through the stages of the referral process.

We aim to fill this knowledge gap by analyzing both appointment scheduling and subsequent visit attendance for referrals from 52 pediatric primary and urgent care practices across a large geographic region. These practices transitioned to an electronic referral system in the year before the study,16  allowing for tracking of referrals, including subsequent appointment scheduling and visit attendance. We aimed first to quantify unscheduled referrals and unattended appointments and second to identify patient and system characteristics associated with each of these types of incomplete referrals.

We conducted a retrospective review of referrals from November 1, 2016, through October 31, 2017, within a large primary care network affiliated with an academic pediatric hospital. This analysis is part of an ongoing quality-improvement project to improve patient care within our institution and was approved by the University of Pittsburgh Medical Center Quality Improvement Review Committee. Projects approved by this committee do not meet the formal definition of human subjects research, so approval by an institutional review board is not required.

We analyzed referrals from 43 pediatric primary care sites and 9 pediatric urgent care locations across 13 counties in western Pennsylvania. Pennsylvania is the sixth most populous state in the United States, with state-wide child demographics generally reflecting those of the United States. For example, 12% of the Pennsylvania population is reported to be African American and 12.5% live in poverty compared with 13% and 12%, respectively, nationally. Two exceptions are that a greater percentage of the Pennsylvania population lives in rural areas (27% in Pennsylvania versus 19% nationally), and Pennsylvania has a smaller Hispanic population (7% in Pennsylvania versus 18% nationally). Compared with Pennsylvania and national demographics, the children receiving referrals in our sample consisted of a higher percentage who identified as African American (21%) and had a slightly lower zip code median income for households with children ($65 833 vs $71 700 for Pennsylvania and $68 011 for the United States).23,24 

The 43 practices contributing referral data in this study are affiliated with an academic children’s hospital (University of Pittsburgh Medical Center Children’s Hospital of Pittsburgh [CHP]), which is the primary pediatric referral center in the region, providing >240 000 outpatient specialty visits annually. As of May 2016, all 52 sites used electronically transmitted referrals16  such that referrals entered by PCPs into the electronic health record are electronically transmitted to the subspecialty schedulers. Subspecialty schedulers call families within 48 hours (and up to 3 times) to schedule appointments, and schedulers route the results of their contact (or attempted contact) back to PCPs. To assist with scheduling, electronically transmitted orders include family primary language and reason for referral. For urgent subspecialty concerns, PCPs page on-call subspecialists.

During the study period, 24 527 electronically transmitted referrals to CHP subspecialists were entered. We included referrals to medical and surgical subspecialists placed for children 0 to 17 years of age. We excluded duplicate referrals to the same subspecialty on the same day (n = 30), referrals from non-PCPs (n = 189), and referrals with missing patient zip codes (n = 417). We excluded 2778 referrals to allied health professionals (eg, physical therapy) and services not participating in the electronically transmitted referral process (eg, dentistry and behavioral health). Within each subspecialty, the 3 most common referral reasons accounted for 31% to 65% of referrals and are available in Supplemental Table 4.

The referral outcome was categorized as scheduled if an appointment for the referred subspecialty was scheduled to occur within 90 days of the referral date. Scheduled appointments were categorized as attended if the patient attended an appointment with the referred subspecialty within 90 days of the referral date. We used 90 days as a marker of access within a reasonably timely manner, which is similar to previous studies.19 

We assessed the relationship between appointment scheduling and visit attendance and patient and/or family and system factors. We extracted age, sex, race, primary language, and primary insurance from electronic health record data. We determined zip code median income for families with children using US Census Bureau data.24  For all referrals, we determined travel time from the patient’s zip code centroid to the main academic children’s hospital, where the majority of subspecialty appointments occur. For scheduled appointments, we also determined travel time from the patient’s zip code centroid to the scheduled appointment site (eg, main hospital or satellite clinic). Travel times were calculated by using on-road travel time within ArcGIS (Esri, Redlands, CA). Systems factors assessed included referrer practice site (primary care versus urgent care), referred subspecialty type (medical versus surgical subspecialist), referred subspecialty capacity (high capacity [>10 000 visits annually] versus low capacity [≤10 000 annually]), and the season during which the referral occurred. Among referrals with scheduled appointments, the interval between the referral date and appointment date was calculated and categorized into ≤7, 8 to 14, 15 to 30, 31 to 60, or 61 to 90 days from referral.

We examined sample characteristics and visit outcomes using descriptive statistics, including the volume of referrals and outcome of referrals to each specific subspecialty. We used χ2 tests to test the bivariate association between each referral and child characteristic and (1) scheduled appointments among all referrals and (2) attended visits among scheduled appointments, with referral being the unit of analysis in this unadjusted analysis.

We used multilevel mixed-effects logistic regression models of referral outcomes with child-level random effects to assess the association of each factor with scheduled appointments among referrals and, in a separate model, with attended visits among scheduled appointments. The model for scheduled appointments included referral characteristics (referrer type, subspecialty type, subspecialty capacity, and referral season), child characteristics (age, sex, race and/or ethnicity, preferred language, insurance type, zip code median income, and on-road travel time from the child’s zip code centroid to the main children’s hospital site), and child-level random intercepts. The model for attended visits included referral characteristics (referrer type, subspecialty type, subspecialty capacity, season, on-road travel time from the child’s zip code to the site of their scheduled appointment, and interval between referral and appointment), child characteristics (age, sex, race and/or ethnicity, preferred language, insurance type, and zip code median income), and child-level random intercepts.

Additionally, we performed a sensitivity analysis by repeating our primary analysis using a categorical variable identifying each specific subspecialty in place of the subspecialty categories used in the main analysis (subspecialty type and subspecialty capacity). In an additional analysis (because the interval between referral and scheduled appointment was significantly associated with visit attendance), we performed post hoc analyses to understand the relationship between this variable and other patient and system characteristics, using χ2 tests for categorical variables and the Kruskal-Wallis test by ranks for continuous variables.

All analyses were performed by using Stata version 14.2 (Stata Corp, College Station, TX). We adjusted for multiple comparisons using the Benjamini-Hochberg false discovery rate method, resulting in a critical P of .032.25 

After applying exclusion criteria, 20 466 referrals for 17 232 children were included in our analysis (Fig 1). The children had a median age of 7.5 years and a median travel time to the academic children’s hospital of 23 minutes (Table 1). Referral volume differed across subspecialties, with slightly more than half (52%) of referrals being placed to surgical subspecialties and with 6 subspecialties (otorhinolaryngology, orthopedics, cardiology, ophthalmology, neurology, and gastroenterology) accounting for >60% of referrals (Fig 2).

FIGURE 1

Study flow diagram. Children may be represented in >1 box because of multiple referrals with differing outcomes.

FIGURE 1

Study flow diagram. Children may be represented in >1 box because of multiple referrals with differing outcomes.

Close modal
TABLE 1

Characteristics of Children With 1 or More Referrals (Total 17 232)

N (%)
Age, y  
 <1 2608 (15) 
 ≥1–6 5090 (30) 
 ≥6–12 4962 (29) 
 ≥12–17 4572 (27) 
Female sex 8387 (49) 
Race and/or ethnicity  
 White, non-Hispanic 11 712 (68) 
 African American, non-Hispanic 3640 (21) 
 Hispanic 462 (3) 
 Asian American and/or Pacific Islander 292 (2) 
 Other or declined 1126 (7) 
Primary language is not English or declined 452 (3) 
Public insurance 8242 (48) 
Zip code median income, quartile  
 First ($17 727–$50 883) 4080 (24) 
 Second ($50 884–$65 000) 4210 (24) 
 Third ($65 139–$88 313) 4584 (27) 
 Fourth ($90 313–$157 819) 4358 (25) 
Travel time to pediatric hospital, min  
 ≤30 11 268 (65) 
 30–60 3943 (23) 
 60–90 745 (4) 
 90–120 272 (2) 
 >120 1004 (6) 
N (%)
Age, y  
 <1 2608 (15) 
 ≥1–6 5090 (30) 
 ≥6–12 4962 (29) 
 ≥12–17 4572 (27) 
Female sex 8387 (49) 
Race and/or ethnicity  
 White, non-Hispanic 11 712 (68) 
 African American, non-Hispanic 3640 (21) 
 Hispanic 462 (3) 
 Asian American and/or Pacific Islander 292 (2) 
 Other or declined 1126 (7) 
Primary language is not English or declined 452 (3) 
Public insurance 8242 (48) 
Zip code median income, quartile  
 First ($17 727–$50 883) 4080 (24) 
 Second ($50 884–$65 000) 4210 (24) 
 Third ($65 139–$88 313) 4584 (27) 
 Fourth ($90 313–$157 819) 4358 (25) 
Travel time to pediatric hospital, min  
 ≤30 11 268 (65) 
 30–60 3943 (23) 
 60–90 745 (4) 
 90–120 272 (2) 
 >120 1004 (6) 
FIGURE 2

Appointments scheduled and visits attended by subspecialty for 20 466 referrals. Subspecialty capacity was determined by the volume of actual visits (not referrals) over 12 months. Other specialties include infectious disease, the child development unit, physical medicine and rehabilitation, and neonatology. a Surgical subspecialties.

FIGURE 2

Appointments scheduled and visits attended by subspecialty for 20 466 referrals. Subspecialty capacity was determined by the volume of actual visits (not referrals) over 12 months. Other specialties include infectious disease, the child development unit, physical medicine and rehabilitation, and neonatology. a Surgical subspecialties.

Close modal

Among all referrals, 13 261 (65% of all referrals) resulted in an appointment scheduled to occur within 90 days (Fig 1). Of these, the median interval to the scheduled appointment was 27 days. Among the 35% of appointments that remained unscheduled, 35% (2526 referrals) had a notation from schedulers indicating attempted contact with no answer. Eleven percent (826 referrals) were documented as having declined an appointment, and the remaining 53% (3853 referrals) had no documented explanation for being unscheduled.

Among the 13 261 referrals with a scheduled appointment, 10 514 (79% of scheduled and 51% of total referrals) had an attended appointment within 90 days. Among the 2747 referrals with scheduled but unattended appointments, 1374 (50%) had at least 1 canceled appointment, and the remaining unattended appointments were no-shows. Referral scheduling among individual subspecialties ranged from 7% (weight management) to 84% (urology), and rates of visit attendance among scheduled appointments varied by individual subspecialties from 63% (ophthalmology) to 98% (infectious disease; Fig 2).

In an unadjusted analysis, the likelihood of scheduling an appointment after referral increased for referrals for the youngest children (P < .001), children with private insurance (P < .001), and children in zip codes with the highest median income levels (P < .001; Table 2). The likelihood of a scheduled appointment was also higher for referrals from urgent care sites (P < .001), to surgical subspecialties (P < .001), and to higher-capacity subspecialties (P < .001). Referral scheduling varied significantly across seasons, with the lowest rates of scheduling for referrals being made from March to May (58%; P < .001).

TABLE 2

Referral and Child Characteristics Associated With Appointment Scheduling Among 17 232 Children With 1 or More Referrals

Characteristics of Each ReferralReferralsAppointment ScheduledReferrals With Appointment Scheduled, %PaOR (95% CI)
Total referrals 20 466 13 261 65 — — 
Referral level      
Referrer practice site    <.001*   
 Primary care 19 247 12 343 64  Reference  
 Urgent care 1219 908 74  1.19 (1.01–1.39)  
Subspecialty type    <.001*   
 Medical 9858 5307 54  Reference  
 Surgical 10 608 7954 75  2.16 (1.99–2.34)*  
Subspecialty appointment capacity    <.001*   
 Large 15 549 10 770 69  Reference  
 Small 4917 2491 51  0.46 (0.42–0.50)*  
Season of referral    <.001*   
 December–February 4770 3396 71*  Reference  
 March–May 5475 3192 58*  0.49 (0.44–0.55)*  
 June–August 5027 3194 64*  0.68 (0.61–0.75)*  
 September–November 5194 3479 67*  0.81 (0.73–0.90)*  
Child level       
Age, y    <.001*   
 <1 3212 2594 81  Reference  
 ≥1–6 5993 3984 66  0.44 (0.39–0.50)*  
 ≥6–12 5796 3414 59  0.31 (0.27–0.35)*  
 ≥12–17 5465 3269 60  0.33 (0.29–0.38)*  
Sex    .006*   
 Female 9959 6359 64  1.01 (0.93–1.08)  
 Male 10 507 6902 66  Reference  
Race and/or ethnicity    <.001*   
 Hispanic 575 388 67*  1.04 (0.82–1.33)  
 White, non-Hispanic 9056 9056 68  Reference  
 African American, non-Hispanic 4841 2704 56*  0.71 (0.64–0.79)*  
 Asian American and/or Pacific Islander 347 239 69  0.89 (0.66–1.19)  
 Other or declined 1303 874 67  0.79 (0.68–0.92)*  
Primary language    .071   
 English 19 909 12 880 65  Reference  
 Non-English 557 381 68  1.01 (0.79–1.29)  
Insurance    <.001*   
 Private 10 096 6943 69  Reference  
 Public 10 370 6318 61  0.83 (0.77–0.90)*  
Zip code median income, quartile    <.001*   
 First (lowest) 5205 3071 59*  0.81 (0.72–0.91)*  
 Second 50 042 3224 64  0.89 (0.80–1.00)  
 Third 5313 3529 66*  0.87 (0.79–0.97)*  
 Fourth (highest) 4906 3437 70  Reference  
Travel time to pediatric hospital, min    <.001*   
 ≤30 13 743 8814 64  Reference  
 30–60 446 3100 69  1.08 (0.98–1.19)  
 60–90 841 573 68  1.13 (0.94–1.37)  
 90–120 304 192 63  0.99 (0.74–1.33)  
 >120 1114 582 52  0.57 (0.49–0.67)*  
Characteristics of Each ReferralReferralsAppointment ScheduledReferrals With Appointment Scheduled, %PaOR (95% CI)
Total referrals 20 466 13 261 65 — — 
Referral level      
Referrer practice site    <.001*   
 Primary care 19 247 12 343 64  Reference  
 Urgent care 1219 908 74  1.19 (1.01–1.39)  
Subspecialty type    <.001*   
 Medical 9858 5307 54  Reference  
 Surgical 10 608 7954 75  2.16 (1.99–2.34)*  
Subspecialty appointment capacity    <.001*   
 Large 15 549 10 770 69  Reference  
 Small 4917 2491 51  0.46 (0.42–0.50)*  
Season of referral    <.001*   
 December–February 4770 3396 71*  Reference  
 March–May 5475 3192 58*  0.49 (0.44–0.55)*  
 June–August 5027 3194 64*  0.68 (0.61–0.75)*  
 September–November 5194 3479 67*  0.81 (0.73–0.90)*  
Child level       
Age, y    <.001*   
 <1 3212 2594 81  Reference  
 ≥1–6 5993 3984 66  0.44 (0.39–0.50)*  
 ≥6–12 5796 3414 59  0.31 (0.27–0.35)*  
 ≥12–17 5465 3269 60  0.33 (0.29–0.38)*  
Sex    .006*   
 Female 9959 6359 64  1.01 (0.93–1.08)  
 Male 10 507 6902 66  Reference  
Race and/or ethnicity    <.001*   
 Hispanic 575 388 67*  1.04 (0.82–1.33)  
 White, non-Hispanic 9056 9056 68  Reference  
 African American, non-Hispanic 4841 2704 56*  0.71 (0.64–0.79)*  
 Asian American and/or Pacific Islander 347 239 69  0.89 (0.66–1.19)  
 Other or declined 1303 874 67  0.79 (0.68–0.92)*  
Primary language    .071   
 English 19 909 12 880 65  Reference  
 Non-English 557 381 68  1.01 (0.79–1.29)  
Insurance    <.001*   
 Private 10 096 6943 69  Reference  
 Public 10 370 6318 61  0.83 (0.77–0.90)*  
Zip code median income, quartile    <.001*   
 First (lowest) 5205 3071 59*  0.81 (0.72–0.91)*  
 Second 50 042 3224 64  0.89 (0.80–1.00)  
 Third 5313 3529 66*  0.87 (0.79–0.97)*  
 Fourth (highest) 4906 3437 70  Reference  
Travel time to pediatric hospital, min    <.001*   
 ≤30 13 743 8814 64  Reference  
 30–60 446 3100 69  1.08 (0.98–1.19)  
 60–90 841 573 68  1.13 (0.94–1.37)  
 90–120 304 192 63  0.99 (0.74–1.33)  
 >120 1114 582 52  0.57 (0.49–0.67)*  

—, not applicable.

*

P < .032.

In adjusted analysis, referrals were significantly less likely to result in a scheduled appointment for children who were older (12–17 years old: adjusted odds ratio [aOR] 0.33; 95% confidence interval [CI] 0.29–0.38), identified as non-Hispanic African American (aOR 0.71; 95% CI 0.64–0.79), received public insurance (aOR 0.83; 95% CI 0.77–0.90), resided in a zip code with a lower median income (lowest quartile: aOR 0.81; 95% CI 0.72–0.91), and lived >120 minutes from the main academic hospital (aOR 0.57; 95% CI 0.49–0.67) compared with those in reference groups. Referrals were more likely to result in a scheduled appointment for children who were referred from an urgent care site (aOR 1.19; 95% CI 1.01–1.39) or to a surgical subspecialty (aOR 2.16; 95% CI 1.99–2.34). A sensitivity analysis with subspecialty-specific categories yielded similar findings (Supplemental Table 5).

In an unadjusted analysis, the likelihood of a scheduled appointment resulting in an attended visit was significantly higher for referrals for the youngest children (P < .001), Asian American and/or Pacific Islander children (P < .001), children with private insurance (P < .001), and children from zip codes with a higher median income (P < .001; Table 3). The likelihood of a scheduled appointment resulting in an attended visit also increased for referrals from urgent care (P < .001), to specialties with a lower appointment capacity (P < .001), and when appointment dates were within 7 days of referral (P < .001).

TABLE 3

Referral and Child Characteristics Associated With Attendance of Scheduled Appointments Among 11 808 Children With 1 or More Scheduled Appointments

CharacteristicsScheduled AppointmentsVisit AttendedReferrals With Appointment Scheduled, %PaOR (95% CI)
Total referrals 13 261 10 514 79 — — 
Referral      
Referrer practice site    <.001*  
 Primary care 12 353 9742 79  Reference 
 Urgent care 908 772 85  0.64 (0.50–0.84)* 
Subspecialty type    .046  
 Medical 5307 4162 78  Reference 
 Surgical 7954 6352 80  0.88 (0.78–1.00) 
Subspecialty capacity    <.001*  
 Large 10 770 8475 79  Reference 
 Small 2491 2039 82  1.11 (0.96–1.29) 
Season    .350  
 December–February 3396 2712 80  Reference 
 March–May 3192 2512 79  0.83 (0.70–0.97)* 
 June–August 3194 2555 80  0.97 (0.83–1.14) 
 September–November 3479 2735 79  0.85 (0.72–0.99) 
Travel time to appointment site, min    <.001*  
 ≤30 9422 7364 78  Reference 
 30–60 2625 2194 84  1.09 (0.93–1.28) 
 60–90 643 510 79  0.91 (0.69–1.20) 
 90–120 179 146 82  1.16 (0.71–1.91) 
 >120 min 392 300 77  0.77 (0.56–1.08) 
Interval from referral to scheduled appointment, d    <.001*  
 0–7 2703 2486 92  Reference 
 8–14 1743 1504 86  0.48 (0.37–0.61)* 
 15–30 2902 2423 83  0.36 (0.29–0.45)* 
 31–60 3425 2561 75  0.18 (0.14–0.23)* 
 61–90 2488 1540 62  0.09 (0.07–0.12)* 
Child      
Age, y    <.001*  
 <1 2594 2248 87  Reference 
 ≥1–6 3894 3137 79  0.70 (0.58–0.84)* 
 ≥6–12 3414 2612 77  0.55 (0.45–0.66)* 
 ≥12–18 3269 2517 77  0.45 (0.37–0.56)* 
Sex    .021*  
 Female 6359 4988 78  0.94 (0.84–1.06) 
 Male 6902 5526 80  Reference 
Race and/or ethnicity    <.001*  
 Hispanic 388 302 78  0.79 (0.55–1.15) 
 White, non-Hispanic 9056 7536 83  Reference 
 African American, non-Hispanic 2704 1754 65  0.45 (0.37–0.55)* 
 Asian American and/or Pacific Islander 239 207 87  1.04 (0.63–1.70) 
 Other or declined 874 715 82  0.77 (0.61–0.99)* 
Primary language    .161  
 English 12 880 10 201 79  Reference 
 Non-English 381 313 82  1.47 (0.98–2.19) 
Insurance    <.001*  
 Private 6943 5903 85  Reference 
 Public 6318 4611 73  0.62 (0.54–0.71)* 
Zip code median income, quartile    <.001*  
 First (lowest) 3071 2149 70  0.68 (0.56–0.82)* 
 Second 3224 2512 78  0.83 (0.69–1.00) 
 Third 3529 2937 83  0.99 (0.83–1.18) 
 Fourth (highest) 3437 2916 85  Reference 
CharacteristicsScheduled AppointmentsVisit AttendedReferrals With Appointment Scheduled, %PaOR (95% CI)
Total referrals 13 261 10 514 79 — — 
Referral      
Referrer practice site    <.001*  
 Primary care 12 353 9742 79  Reference 
 Urgent care 908 772 85  0.64 (0.50–0.84)* 
Subspecialty type    .046  
 Medical 5307 4162 78  Reference 
 Surgical 7954 6352 80  0.88 (0.78–1.00) 
Subspecialty capacity    <.001*  
 Large 10 770 8475 79  Reference 
 Small 2491 2039 82  1.11 (0.96–1.29) 
Season    .350  
 December–February 3396 2712 80  Reference 
 March–May 3192 2512 79  0.83 (0.70–0.97)* 
 June–August 3194 2555 80  0.97 (0.83–1.14) 
 September–November 3479 2735 79  0.85 (0.72–0.99) 
Travel time to appointment site, min    <.001*  
 ≤30 9422 7364 78  Reference 
 30–60 2625 2194 84  1.09 (0.93–1.28) 
 60–90 643 510 79  0.91 (0.69–1.20) 
 90–120 179 146 82  1.16 (0.71–1.91) 
 >120 min 392 300 77  0.77 (0.56–1.08) 
Interval from referral to scheduled appointment, d    <.001*  
 0–7 2703 2486 92  Reference 
 8–14 1743 1504 86  0.48 (0.37–0.61)* 
 15–30 2902 2423 83  0.36 (0.29–0.45)* 
 31–60 3425 2561 75  0.18 (0.14–0.23)* 
 61–90 2488 1540 62  0.09 (0.07–0.12)* 
Child      
Age, y    <.001*  
 <1 2594 2248 87  Reference 
 ≥1–6 3894 3137 79  0.70 (0.58–0.84)* 
 ≥6–12 3414 2612 77  0.55 (0.45–0.66)* 
 ≥12–18 3269 2517 77  0.45 (0.37–0.56)* 
Sex    .021*  
 Female 6359 4988 78  0.94 (0.84–1.06) 
 Male 6902 5526 80  Reference 
Race and/or ethnicity    <.001*  
 Hispanic 388 302 78  0.79 (0.55–1.15) 
 White, non-Hispanic 9056 7536 83  Reference 
 African American, non-Hispanic 2704 1754 65  0.45 (0.37–0.55)* 
 Asian American and/or Pacific Islander 239 207 87  1.04 (0.63–1.70) 
 Other or declined 874 715 82  0.77 (0.61–0.99)* 
Primary language    .161  
 English 12 880 10 201 79  Reference 
 Non-English 381 313 82  1.47 (0.98–2.19) 
Insurance    <.001*  
 Private 6943 5903 85  Reference 
 Public 6318 4611 73  0.62 (0.54–0.71)* 
Zip code median income, quartile    <.001*  
 First (lowest) 3071 2149 70  0.68 (0.56–0.82)* 
 Second 3224 2512 78  0.83 (0.69–1.00) 
 Third 3529 2937 83  0.99 (0.83–1.18) 
 Fourth (highest) 3437 2916 85  Reference 

—, not applicable.

*

P < .032.

In an adjusted analysis, appointment attendance significantly decreased for referrals for older children (aOR 0.45; 95% CI 0.37–0.56), children identifying as non-Hispanic African American (aOR 0.45; 95% CI 0.37–0.55), children with public insurance (aOR 0.62; 95% CI 0.54–0.71), and children from zip codes with the lowest median income (aOR 0.68; 95% CI 0.56–0.82). In an adjusted analysis, attended appointments decreased for referrals from urgent care (aOR 0.64; 95% CI 0.50–0.84) and with longer intervals from the referral to the scheduled appointment (61–90 days: aOR 0.09; 95% CI 0.07–0.12). A sensitivity analysis with a subspecialty-specific variable again yielded similar findings (Supplemental Table 5).

Because of the significant association between the time to the scheduled appointment and appointment attendance, we examined the median time to the scheduled appointment across referral and child characteristics (Supplemental Table 6). Shorter intervals from the referral to the scheduled appointment were associated with referral from urgent care (5 days versus 28 days for referral from primary care; P < .001), to surgical subspecialists (23 days versus 30 days for referral to medical subspecialties; P < .001), to small-capacity subspecialties (24 days versus 27 days for referrals to high-capacity subspecialties; P < .001), and from March to May (23 vs 32 days for referrals from December to February; P ≤ .001). Shorter intervals were also associated with referrals for children with specific patient characteristics, including younger age, (P < .001), Asian American and/or Pacific Islander race (P < .001), private insurance (P < .001), and 30- to 60-minute travel times (P < .001).

In this large primary care network, only 51% of referrals resulted in a visit completed within 90 days. Approximately one-sixth of all referrals resulted in a scheduled but unattended appointment, and one-third of all referrals had no appointment scheduled within 90 days. This overall referral completion rate was lower than the 70% completion rate found in 1 previous pediatric study of referrals from 2 health centers19  but higher than a recent analysis in an adult health system, where 35% of referrals were completed.26  We also found that the rates of 2 separate steps, referral scheduling and visit attendance, varied significantly with patient and system factors.

We found significant variation in scheduling and attendance rates across multiple system factors. Among scheduled appointments, longer intervals from the referral to the scheduled appointment date decreased the likelihood of visit attendance. Previous work reported that attendance was improved in appointments scheduled within 60 days of the subspecialty referral, and a recent study also demonstrated lower no-show rates for ophthalmology appointments scheduled with a shorter lead time.19,27  In our adjusted analysis, even appointments 8 to 14 days after referral were substantially less likely to be attended than those scheduled for 0 to 7 days after. Our cross-sectional data cannot determine if shorter intervals from the referral to the scheduled appointment have a causal impact on visit attendance; an alternative explanation is that unmeasured variables increased both the likelihood of an earlier appointment and visit attendance (eg, increased clinical acuity, increased parental anxiety, and/or increased PCP advocacy for an earlier appointment). Future research could test whether interventions to decrease the interval to appointments improve appointment attendance. Although providing appointments within a week or even a month remains a substantial challenge for many pediatric subspecialties,5  specific interventions to decrease scheduling intervals (eg, open access), increase appointment capacity (eg, advanced practice providers or primary care physicians in subspecialty care clinics), or reduce referral burden (eg referral guidelines or electronic consultations) are worth examining for their impact on the visit interval and subsequently visit attendance, no-show rates, and subspecialty clinic efficiency.18,28,29 

Among related systems factors, we observed that referrals from urgent care and to surgical specialties were more likely to result in a scheduled appointment in unadjusted analysis and were associated with more timely appointments. Although referral from urgent care was also associated with an increased likelihood of visit attendance in an unadjusted analysis, referral from urgent care was associated with a decreased likelihood of visit attendance after accounting for the timeliness of the appointment. These results further support the association between appointment timeliness and visit attendance while suggesting opportunities to learn from strategies employed by specific specialties and referrers to improve the timeliness of appointment scheduling.

Additional variables highlight the value of separately assessing appointment scheduling and visit attendance. As one might expect, subspecialties with lower appointment capacity had lower appointment scheduling, but appointment capacity was not significantly associated with visit attendance among scheduled appointments. Season of referral was also associated with scheduling but not attendance. The lowest seasonal rates of scheduling occurred in the springtime months. One possible explanation is that appointments generated from these referrals would fall in the summer months, when family and physician schedules may be disrupted by summer recesses, travel, and new trainees requiring higher levels of supervision. Understanding whether seasonal variation is driven more by shifts in family availability versus appointment availability could drive additional improvement strategies.

In addition to the above system factors, several patient factors were significantly associated with referral scheduling and visit attendance. Specifically, African American race, public insurance, and lower zip code median income were each associated with both unscheduled and unattended appointments. Because demographic factors were associated with both decreased scheduling of appointments and decreased attendance at scheduled appointments, there is potential value for interventions at both of these phases with a focus on specific populations’ needs to address disparities in referral completion. Of note, these factors (African American race, public insurance, and lower zip code median income) were also associated with longer time intervals to appointment scheduling. Notably, these disparities in scheduling were observed within a system in which subspecialist schedulers were tasked with actively calling each of these families, suggesting a need to explore alternative means of contacting families (eg, texting), address alternative barriers to scheduling (eg, unpredictable work schedules, limited health literacy, or previous experience of perceived prejudice or racism in health care settings), and consider possible implicit or explicit biases in appointment scheduling processes. Also, although extreme travel time was associated with decreased visit scheduling, we found that visit attendance among scheduled appointments did not differ across travel time. This suggests that families may account for the realities of travel time in their decisions to schedule, but those who are scheduling are equally likely to attend despite prolonged travel times. Interventions designed to address geographic barriers, such as telemedicine, might then have the largest impact on appointment scheduling (rather than attendance rates among scheduled appointments) for patients in these communities.

This study was limited by the data available. Most notably, these data did not include why patients declined to schedule or why they canceled or missed appointments. These data also did not include information on subspecialty visits that occurred outside the studied health system. However, we limited our analysis only to referrals routed specifically to CHP subspecialists for completion within the studied health system. Additionally, we note that institutional variation in subspecialty clinic structure and subspecialist supply may limit the generalizability of subspecialty-specific data. For this reason, we focused on more generalizable characteristics of subspecialties (medical versus surgical and high versus low capacity) in our main analysis while acknowledging subspecialty-specific variation and including more details on subspecialty-specific referral reasons and outcomes for specific subspecialties in our supplemental analyses. Next, we could not address the appropriateness or urgency of referral. Some unscheduled referrals and unattended appointments may represent unnecessary referrals that did not truly require a subspecialty visit or cases in which symptoms resolved so that the need for referral resolved. For this reason, we would not expect all referrals to be completed, but we do seek to address unwarranted variation in scheduling and completion. Finally, we note that our analysis occurred within the setting of a relatively unfragmented referral catchment area with an electronically transmitted referral system in place. These features likely make our results best case scenarios by simplifying scheduling choices and steps for families, which is consistent with previous work showing that the implementation of electronically transmitted referral improved referral completion.16  It is also possible that electronically transmitted referrals help reduce disparities by reaching out to families with communication barriers or time constraints, in which case other settings may also observe larger disparities across child characteristics. However, the presence of the electronically transmitted referrals within a large, relatively unfragmented catchment area also enhances the quality of our data for this analysis (by comprehensively capturing referrals and subsequent visits), thereby enhancing the internal validity of our findings.

We identified system-level factors associated with referral scheduling and appointment attendance among children referred for pediatric subspecialty care, including referral site, subspecialty type, subspecialty capacity, and interval from referral to scheduled appointment. We also identified inequities in appointment scheduling and visit attendance associated with patient sociodemographic characteristics. Targeting barriers to appointment scheduling and visit attendance may result in more efficient and equitable access to pediatric subspecialty care.

Dr Bohnhoff conceptualized and designed the study, analyzed the data, drafted the initial manuscript, and reviewed and revised the manuscript; Ms Taormina collected the data, assisted with interpretation, and critically revised the manuscript; Ms Ferrante and Dr Wolfson assisted with interpretation and critically revised the manuscript; Dr Ray supervised study conceptualization and design as well as analysis, interpreted results, and critically revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: No external funding.

COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2019-2975.

     
  • aOR

    adjusted odds ratio

  •  
  • CHP

    Children’s Hospital of Pittsburgh

  •  
  • CI

    confidence interval

  •  
  • PCP

    primary care provider

1
Vernacchio
L
,
Muto
JM
,
Young
G
,
Risko
W
.
Ambulatory subspecialty visits in a large pediatric primary care network
.
Health Serv Res
.
2012
;
47
(
4
):
1755
1769
2
Ray
KN
,
Ashcraft
LE
,
Kahn
JM
,
Mehrotra
A
,
Miller
E
.
Family perspectives on high-quality pediatric subspecialty referrals
.
Acad Pediatr
.
2016
;
16
(
6
):
594
600
3
Mehrotra
A
,
Forrest
CB
,
Lin
CY
.
Dropping the baton: specialty referrals in the United States
.
Milbank Q
.
2011
;
89
(
1
):
39
68
4
Gandhi
TK
,
Sittig
DF
,
Franklin
M
, et al
.
Communication breakdown in the outpatient referral process
.
J Gen Intern Med
.
2000
;
15
(
9
):
626
631
5
Pletcher
BA
,
Rimsza
ME
,
Cull
WL
, et al
.
Primary care pediatricians’ satisfaction with subspecialty care, perceived supply, and barriers to care
.
J Pediatr
.
2010
;
156
(
6
):
1011
1015.e1
6
Stille
CJ
,
Primack
WA
,
Savageau
JA
.
Generalist-subspecialist communication for children with chronic conditions: a regional physician survey
.
Pediatrics
.
2003
;
112
(
6, pt 1
):
1314
1320
7
Ray
KN
,
Bogen
DL
,
Bertolet
M
,
Forrest
CB
,
Mehrotra
A
.
Supply and utilization of pediatric subspecialists in the United States
.
Pediatrics
.
2014
;
133
(
6
):
1061
1069
8
Bethell
CD
,
Kogan
MD
,
Strickland
BB
, et al
.
A national and state profile of leading health problems and health care quality for US children: key insurance disparities and across-state variations
.
Acad Pediatr
.
2011
;
11
(
suppl 3
):
S22
S33
9
Ray
KN
,
Kahn
JM
,
Miller
E
,
Mehrotra
A
.
Use of adult-trained medical subspecialists by children seeking medical subspecialty care
.
J Pediatr
.
2016
;
176
:
173
181.e1
10
Institute for Healthcare Improvement
:
Closing the Loop: A Guide to Safer Ambulatory Referrals in the EHR Era
.
Cambridge, MA
:
Institute for Healthcare Improvement
;
2017
11
Guevara
JP
,
Hsu
D
,
Forrest
CB
.
Performance measures of the specialty referral process: a systematic review of the literature
.
BMC Health Serv Res
.
2011
;
11
:
168
12
Cornell
E
,
Chandhok
L
,
Rubin
K
.
Implementation of referral guidelines at the interface between pediatric primary and subspecialty care
.
Healthc (Amst)
.
2015
;
3
(
2
):
74
79
13
Kim
Y
,
Chen
AH
,
Keith
E
,
Yee
HF
 Jr
,
Kushel
MB
.
Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health system
.
J Gen Intern Med
.
2009
;
24
(
5
):
614
619
14
Barnett
ML
,
Yee
HF
 Jr
,
Mehrotra
A
,
Giboney
P
.
Los Angeles safety-net program econsult system was rapidly adopted and decreased wait times to see specialists
.
Health Aff (Millwood)
.
2017
;
36
(
3
):
492
499
15
Lee
MS
,
Ray
KN
,
Mehrotra
A
, et al
.
Primary care practitioners’ perceptions of electronic consult systems: a qualitative analysis
.
JAMA Intern Med
.
2018
;
178
(
6
):
782
789
16
Ray
KN
,
Drnach
M
,
Mehrotra
A
,
Suresh
S
,
Docimo
SG
.
Impact of implementation of electronically transmitted referrals on pediatric subspecialty visit attendance
.
Acad Pediatr
.
2018
;
18
(
4
):
409
417
17
Martsolf
GR
,
Barnes
H
,
Richards
MR
, et al
.
Employment of advanced practice clinicians in physician practices
.
JAMA Intern Med
.
2018
;
178
(
7
):
988
990
18
Ray
KN
,
Martsolf
GR
,
Mehrotra
A
,
Barnett
ML
.
Trends in visits to specialist physicians involving nurse practitioners and physician assistants, 2001 to 2013
.
JAMA Intern Med
.
2017
;
177
(
8
):
1213
1216
19
Zuckerman
KE
,
Cai
X
,
Perrin
JM
,
Donelan
K
.
Incomplete specialty referral among children in community health centers
.
J Pediatr
.
2011
;
158
(
1
):
24
30
20
Zuckerman
KE
,
Perrin
JM
,
Hobrecker
K
,
Donelan
K
.
Barriers to specialty care and specialty referral completion in the community health center setting
.
J Pediatr
.
2013
;
162
(
2
):
409
14.e1
21
Forrest
CB
,
Glade
GB
,
Baker
AE
, et al
.
The pediatric primary-specialty care interface: how pediatricians refer children and adolescents to specialty care
.
Arch Pediatr Adolesc Med
.
1999
;
153
(
7
):
705
714
22
Forrest
CB
,
Majeed
A
,
Weiner
JP
,
Carroll
K
,
Bindman
AB
.
Referral of children to specialists in the United States and the United Kingdom
.
Arch Pediatr Adolesc Med
.
2003
;
157
(
3
):
279
285
23
US Census Bureau
. QuickFacts.
2018
. Available at: https://www.census.gov/quickfacts/fact/table/PA,US/PST045218. Accessed Aug 26, 2018
24
Annie E. Casey Foundation
.
Median family income among households with children in the United States.
Available at https://datacenter.kidscount.org/data#USA/1/0/char/0. Accessed Aug 26, 2018
25
Benjamin
Y
,
Hochberg
Y
.
Controlling the false discovery rate: a practical and powerful approach to multiple testing
.
J R Stat Soc B
.
1995
;
57
(
1
):
289
300
26
Patel
MP
,
Schettini
P
,
O’Leary
CP
, et al
.
Closing the referral loop: an analysis of primary care referrals to specialists in a large health system
.
J Gen Intern Med
.
2018
;
33
(
5
):
715
721
27
McMullen
MJ
,
Netland
PA
.
Lead time for appointment and the no-show rate in an ophthalmology clinic
.
Clin Ophthalmol
.
2015
;
9
:
513
516
28
Newman
ED
,
Harrington
TM
,
Olenginski
TP
,
Perruquet
JL
,
McKinley
K
.
“The rheumatologist can see you now”: successful implementation of an advanced access model in a rheumatology practice
.
Arthritis Rheum
.
2004
;
51
(
2
):
253
257
29
Di Guglielmo
MD
,
Plesnick
J
,
Greenspan
JS
,
Sharif
I
.
A new model to decrease time-to-appointment wait for gastroenterology evaluation
.
Pediatrics
.
2013
;
131
(
5
).

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data