Incomplete subspecialty referrals, whether unscheduled or unattended, represent unmet patient needs and an opportunity to improve patient safety and experiences. Our objectives were to describe the rates of appointment scheduling and visit attendance after pediatric subspecialty referral and to examine patient and systems factors associated with scheduled referrals and attended appointments.
We conducted a retrospective review of referrals within a network of 52 primary and urgent care sites from November 2016 to October 2017. We included referrals for children ≤17 years old referred to medical or surgical subspecialists. We examined patient and health systems factors associated with (1) appointment scheduling and (2) visit attendance.
Of 20 466 referrals, 13 261 (65%) resulted in an appointment scheduled within 90 days and 10 514 (51%) resulted in a visit attended within 90 days. In adjusted analyses, referral to surgical subspecialists was associated with an increased likelihood of appointment scheduling but a decreased likelihood of visit attendance. Compared with appointments scheduled within 7 days, appointments with intervals from referral to scheduled appointment exceeding 7 days were associated with decreasing likelihood of visit attendance (adjusted odds ratio 8–14 days 0.48; 95% confidence interval 0.37–0.61). Patient factors associated with decreased likelihood of both appointment scheduling and visit attendance included African American race, public insurance, and lower zip code median income.
Patient and system factors were associated with variation in appointment scheduling and visit attendance. Decreased interval to appointment was significantly associated with visit attendance. These factors represent targets for interventions to improve referral completion.
Incomplete subspecialty referrals represent an opportunity to improve patient safety and experiences. Data on factors associated with variation in appointment scheduling and visit attendance after pediatric subspecialty referral are limited.
Two-thirds of pediatric subspecialty referrals resulted in scheduled appointments, and half resulted in attended visits. Both patient and system factors were associated with variation in appointment scheduling and visit attendance, and time to appointment was strongly associated with attendance.
Referral to subspecialty care is an important task within pediatric primary care and occurs on average at least once annually for every 3 pediatric patients.1 Despite the ubiquity and importance of subspecialty referral, multiple studies report that patients,2 primary care providers (PCPs),3–5 and subspecialists4,6 find the process of connecting patients with subspecialists to be ineffective and inefficient. Complexities in referral processes and appointment scheduling,2,3 coupled with substantial limitations and variation in the availability of subspecialists,7,8 reduce patients’ access to subspecialty care. As a result, in the United States, 24% of families in need of subspecialty care report difficulty accessing this care.8 Families without ready access to pediatric subspecialists are less likely to receive subspecialty care,7 more likely to receive care from specialists who are not trained in pediatrics,9 and more likely to report emergency department visits.7
In 2017, the Institute for Healthcare Improvement and the National Patient Safety Foundation released a report recognizing communication breakdown during subspecialist referral as a threat to patient safety.10 The report, along with others, highlighted the sequential steps required to complete referrals and provided guidance on interventions to improve specific steps in the process.2,10,11 Some emerging interventions aim to aid in primary care referral decisions, including referral guidelines12 and electronic consultations.13–15 Other interventions, such as electronic referrals, aim to bypass barriers in the scheduling process, thereby improving appointment scheduling rates.16 Others aim to address scheduling issues by increasing subspecialist capacity, including increased use of advanced practice providers in specialty care.17,18 Finally, to improve visit attendance, systems and payer strategies range from reminders to telemedicine to transportation assistance.
The degree to which these interventions will address barriers to subspecialty care depends in part on an understanding of the overall unmet need for care, the steps in which referral processes break down, and whether these roadblocks vary by patient or system factors. The largest pediatric study to date in this area focused on 577 referrals made from 2008 to 2009 from 2 community health centers.19,20 Other examinations of pediatric subspecialty care focus either on rates of referral21 or appointments attended,1,22 which contribute to an understanding of referral or use patterns but cannot fully capture referrals that go uncompleted. Thus, to guide further interventions improving subspecialty care, there is a need for updated analysis from a larger sample of primary care settings focusing on how patients move through the stages of the referral process.
We aim to fill this knowledge gap by analyzing both appointment scheduling and subsequent visit attendance for referrals from 52 pediatric primary and urgent care practices across a large geographic region. These practices transitioned to an electronic referral system in the year before the study,16 allowing for tracking of referrals, including subsequent appointment scheduling and visit attendance. We aimed first to quantify unscheduled referrals and unattended appointments and second to identify patient and system characteristics associated with each of these types of incomplete referrals.
Methods
Study Design
We conducted a retrospective review of referrals from November 1, 2016, through October 31, 2017, within a large primary care network affiliated with an academic pediatric hospital. This analysis is part of an ongoing quality-improvement project to improve patient care within our institution and was approved by the University of Pittsburgh Medical Center Quality Improvement Review Committee. Projects approved by this committee do not meet the formal definition of human subjects research, so approval by an institutional review board is not required.
Study Context
We analyzed referrals from 43 pediatric primary care sites and 9 pediatric urgent care locations across 13 counties in western Pennsylvania. Pennsylvania is the sixth most populous state in the United States, with state-wide child demographics generally reflecting those of the United States. For example, 12% of the Pennsylvania population is reported to be African American and 12.5% live in poverty compared with 13% and 12%, respectively, nationally. Two exceptions are that a greater percentage of the Pennsylvania population lives in rural areas (27% in Pennsylvania versus 19% nationally), and Pennsylvania has a smaller Hispanic population (7% in Pennsylvania versus 18% nationally). Compared with Pennsylvania and national demographics, the children receiving referrals in our sample consisted of a higher percentage who identified as African American (21%) and had a slightly lower zip code median income for households with children ($65 833 vs $71 700 for Pennsylvania and $68 011 for the United States).23,24
The 43 practices contributing referral data in this study are affiliated with an academic children’s hospital (University of Pittsburgh Medical Center Children’s Hospital of Pittsburgh [CHP]), which is the primary pediatric referral center in the region, providing >240 000 outpatient specialty visits annually. As of May 2016, all 52 sites used electronically transmitted referrals16 such that referrals entered by PCPs into the electronic health record are electronically transmitted to the subspecialty schedulers. Subspecialty schedulers call families within 48 hours (and up to 3 times) to schedule appointments, and schedulers route the results of their contact (or attempted contact) back to PCPs. To assist with scheduling, electronically transmitted orders include family primary language and reason for referral. For urgent subspecialty concerns, PCPs page on-call subspecialists.
Study Population
During the study period, 24 527 electronically transmitted referrals to CHP subspecialists were entered. We included referrals to medical and surgical subspecialists placed for children 0 to 17 years of age. We excluded duplicate referrals to the same subspecialty on the same day (n = 30), referrals from non-PCPs (n = 189), and referrals with missing patient zip codes (n = 417). We excluded 2778 referrals to allied health professionals (eg, physical therapy) and services not participating in the electronically transmitted referral process (eg, dentistry and behavioral health). Within each subspecialty, the 3 most common referral reasons accounted for 31% to 65% of referrals and are available in Supplemental Table 4.
Referral Outcomes
The referral outcome was categorized as scheduled if an appointment for the referred subspecialty was scheduled to occur within 90 days of the referral date. Scheduled appointments were categorized as attended if the patient attended an appointment with the referred subspecialty within 90 days of the referral date. We used 90 days as a marker of access within a reasonably timely manner, which is similar to previous studies.19
Patient-, Referral-, and System-Level Characteristics
We assessed the relationship between appointment scheduling and visit attendance and patient and/or family and system factors. We extracted age, sex, race, primary language, and primary insurance from electronic health record data. We determined zip code median income for families with children using US Census Bureau data.24 For all referrals, we determined travel time from the patient’s zip code centroid to the main academic children’s hospital, where the majority of subspecialty appointments occur. For scheduled appointments, we also determined travel time from the patient’s zip code centroid to the scheduled appointment site (eg, main hospital or satellite clinic). Travel times were calculated by using on-road travel time within ArcGIS (Esri, Redlands, CA). Systems factors assessed included referrer practice site (primary care versus urgent care), referred subspecialty type (medical versus surgical subspecialist), referred subspecialty capacity (high capacity [>10 000 visits annually] versus low capacity [≤10 000 annually]), and the season during which the referral occurred. Among referrals with scheduled appointments, the interval between the referral date and appointment date was calculated and categorized into ≤7, 8 to 14, 15 to 30, 31 to 60, or 61 to 90 days from referral.
Statistical Analysis
We examined sample characteristics and visit outcomes using descriptive statistics, including the volume of referrals and outcome of referrals to each specific subspecialty. We used χ2 tests to test the bivariate association between each referral and child characteristic and (1) scheduled appointments among all referrals and (2) attended visits among scheduled appointments, with referral being the unit of analysis in this unadjusted analysis.
We used multilevel mixed-effects logistic regression models of referral outcomes with child-level random effects to assess the association of each factor with scheduled appointments among referrals and, in a separate model, with attended visits among scheduled appointments. The model for scheduled appointments included referral characteristics (referrer type, subspecialty type, subspecialty capacity, and referral season), child characteristics (age, sex, race and/or ethnicity, preferred language, insurance type, zip code median income, and on-road travel time from the child’s zip code centroid to the main children’s hospital site), and child-level random intercepts. The model for attended visits included referral characteristics (referrer type, subspecialty type, subspecialty capacity, season, on-road travel time from the child’s zip code to the site of their scheduled appointment, and interval between referral and appointment), child characteristics (age, sex, race and/or ethnicity, preferred language, insurance type, and zip code median income), and child-level random intercepts.
Additionally, we performed a sensitivity analysis by repeating our primary analysis using a categorical variable identifying each specific subspecialty in place of the subspecialty categories used in the main analysis (subspecialty type and subspecialty capacity). In an additional analysis (because the interval between referral and scheduled appointment was significantly associated with visit attendance), we performed post hoc analyses to understand the relationship between this variable and other patient and system characteristics, using χ2 tests for categorical variables and the Kruskal-Wallis test by ranks for continuous variables.
All analyses were performed by using Stata version 14.2 (Stata Corp, College Station, TX). We adjusted for multiple comparisons using the Benjamini-Hochberg false discovery rate method, resulting in a critical P of .032.25
Results
After applying exclusion criteria, 20 466 referrals for 17 232 children were included in our analysis (Fig 1). The children had a median age of 7.5 years and a median travel time to the academic children’s hospital of 23 minutes (Table 1). Referral volume differed across subspecialties, with slightly more than half (52%) of referrals being placed to surgical subspecialties and with 6 subspecialties (otorhinolaryngology, orthopedics, cardiology, ophthalmology, neurology, and gastroenterology) accounting for >60% of referrals (Fig 2).
Study flow diagram. Children may be represented in >1 box because of multiple referrals with differing outcomes.
Study flow diagram. Children may be represented in >1 box because of multiple referrals with differing outcomes.
Characteristics of Children With 1 or More Referrals (Total 17 232)
| N (%) | |
|---|---|
| Age, y | |
| <1 | 2608 (15) |
| ≥1–6 | 5090 (30) |
| ≥6–12 | 4962 (29) |
| ≥12–17 | 4572 (27) |
| Female sex | 8387 (49) |
| Race and/or ethnicity | |
| White, non-Hispanic | 11 712 (68) |
| African American, non-Hispanic | 3640 (21) |
| Hispanic | 462 (3) |
| Asian American and/or Pacific Islander | 292 (2) |
| Other or declined | 1126 (7) |
| Primary language is not English or declined | 452 (3) |
| Public insurance | 8242 (48) |
| Zip code median income, quartile | |
| First ($17 727–$50 883) | 4080 (24) |
| Second ($50 884–$65 000) | 4210 (24) |
| Third ($65 139–$88 313) | 4584 (27) |
| Fourth ($90 313–$157 819) | 4358 (25) |
| Travel time to pediatric hospital, min | |
| ≤30 | 11 268 (65) |
| 30–60 | 3943 (23) |
| 60–90 | 745 (4) |
| 90–120 | 272 (2) |
| >120 | 1004 (6) |
| N (%) | |
|---|---|
| Age, y | |
| <1 | 2608 (15) |
| ≥1–6 | 5090 (30) |
| ≥6–12 | 4962 (29) |
| ≥12–17 | 4572 (27) |
| Female sex | 8387 (49) |
| Race and/or ethnicity | |
| White, non-Hispanic | 11 712 (68) |
| African American, non-Hispanic | 3640 (21) |
| Hispanic | 462 (3) |
| Asian American and/or Pacific Islander | 292 (2) |
| Other or declined | 1126 (7) |
| Primary language is not English or declined | 452 (3) |
| Public insurance | 8242 (48) |
| Zip code median income, quartile | |
| First ($17 727–$50 883) | 4080 (24) |
| Second ($50 884–$65 000) | 4210 (24) |
| Third ($65 139–$88 313) | 4584 (27) |
| Fourth ($90 313–$157 819) | 4358 (25) |
| Travel time to pediatric hospital, min | |
| ≤30 | 11 268 (65) |
| 30–60 | 3943 (23) |
| 60–90 | 745 (4) |
| 90–120 | 272 (2) |
| >120 | 1004 (6) |
Appointments scheduled and visits attended by subspecialty for 20 466 referrals. Subspecialty capacity was determined by the volume of actual visits (not referrals) over 12 months. Other specialties include infectious disease, the child development unit, physical medicine and rehabilitation, and neonatology. a Surgical subspecialties.
Appointments scheduled and visits attended by subspecialty for 20 466 referrals. Subspecialty capacity was determined by the volume of actual visits (not referrals) over 12 months. Other specialties include infectious disease, the child development unit, physical medicine and rehabilitation, and neonatology. a Surgical subspecialties.
Among all referrals, 13 261 (65% of all referrals) resulted in an appointment scheduled to occur within 90 days (Fig 1). Of these, the median interval to the scheduled appointment was 27 days. Among the 35% of appointments that remained unscheduled, 35% (2526 referrals) had a notation from schedulers indicating attempted contact with no answer. Eleven percent (826 referrals) were documented as having declined an appointment, and the remaining 53% (3853 referrals) had no documented explanation for being unscheduled.
Among the 13 261 referrals with a scheduled appointment, 10 514 (79% of scheduled and 51% of total referrals) had an attended appointment within 90 days. Among the 2747 referrals with scheduled but unattended appointments, 1374 (50%) had at least 1 canceled appointment, and the remaining unattended appointments were no-shows. Referral scheduling among individual subspecialties ranged from 7% (weight management) to 84% (urology), and rates of visit attendance among scheduled appointments varied by individual subspecialties from 63% (ophthalmology) to 98% (infectious disease; Fig 2).
Factors Associated With Scheduled Referrals
In an unadjusted analysis, the likelihood of scheduling an appointment after referral increased for referrals for the youngest children (P < .001), children with private insurance (P < .001), and children in zip codes with the highest median income levels (P < .001; Table 2). The likelihood of a scheduled appointment was also higher for referrals from urgent care sites (P < .001), to surgical subspecialties (P < .001), and to higher-capacity subspecialties (P < .001). Referral scheduling varied significantly across seasons, with the lowest rates of scheduling for referrals being made from March to May (58%; P < .001).
Referral and Child Characteristics Associated With Appointment Scheduling Among 17 232 Children With 1 or More Referrals
| Characteristics of Each Referral | Referrals | Appointment Scheduled | Referrals With Appointment Scheduled, % | P | aOR (95% CI) | |
|---|---|---|---|---|---|---|
| Total referrals | 20 466 | 13 261 | 65 | — | — | |
| Referral level | ||||||
| Referrer practice site | <.001* | |||||
| Primary care | 19 247 | 12 343 | 64 | Reference | ||
| Urgent care | 1219 | 908 | 74 | 1.19 (1.01–1.39) | ||
| Subspecialty type | <.001* | |||||
| Medical | 9858 | 5307 | 54 | Reference | ||
| Surgical | 10 608 | 7954 | 75 | 2.16 (1.99–2.34)* | ||
| Subspecialty appointment capacity | <.001* | |||||
| Large | 15 549 | 10 770 | 69 | Reference | ||
| Small | 4917 | 2491 | 51 | 0.46 (0.42–0.50)* | ||
| Season of referral | <.001* | |||||
| December–February | 4770 | 3396 | 71* | Reference | ||
| March–May | 5475 | 3192 | 58* | 0.49 (0.44–0.55)* | ||
| June–August | 5027 | 3194 | 64* | 0.68 (0.61–0.75)* | ||
| September–November | 5194 | 3479 | 67* | 0.81 (0.73–0.90)* | ||
| Child level | ||||||
| Age, y | <.001* | |||||
| <1 | 3212 | 2594 | 81 | Reference | ||
| ≥1–6 | 5993 | 3984 | 66 | 0.44 (0.39–0.50)* | ||
| ≥6–12 | 5796 | 3414 | 59 | 0.31 (0.27–0.35)* | ||
| ≥12–17 | 5465 | 3269 | 60 | 0.33 (0.29–0.38)* | ||
| Sex | .006* | |||||
| Female | 9959 | 6359 | 64 | 1.01 (0.93–1.08) | ||
| Male | 10 507 | 6902 | 66 | Reference | ||
| Race and/or ethnicity | <.001* | |||||
| Hispanic | 575 | 388 | 67* | 1.04 (0.82–1.33) | ||
| White, non-Hispanic | 9056 | 9056 | 68 | Reference | ||
| African American, non-Hispanic | 4841 | 2704 | 56* | 0.71 (0.64–0.79)* | ||
| Asian American and/or Pacific Islander | 347 | 239 | 69 | 0.89 (0.66–1.19) | ||
| Other or declined | 1303 | 874 | 67 | 0.79 (0.68–0.92)* | ||
| Primary language | .071 | |||||
| English | 19 909 | 12 880 | 65 | Reference | ||
| Non-English | 557 | 381 | 68 | 1.01 (0.79–1.29) | ||
| Insurance | <.001* | |||||
| Private | 10 096 | 6943 | 69 | Reference | ||
| Public | 10 370 | 6318 | 61 | 0.83 (0.77–0.90)* | ||
| Zip code median income, quartile | <.001* | |||||
| First (lowest) | 5205 | 3071 | 59* | 0.81 (0.72–0.91)* | ||
| Second | 50 042 | 3224 | 64 | 0.89 (0.80–1.00) | ||
| Third | 5313 | 3529 | 66* | 0.87 (0.79–0.97)* | ||
| Fourth (highest) | 4906 | 3437 | 70 | Reference | ||
| Travel time to pediatric hospital, min | <.001* | |||||
| ≤30 | 13 743 | 8814 | 64 | Reference | ||
| 30–60 | 446 | 3100 | 69 | 1.08 (0.98–1.19) | ||
| 60–90 | 841 | 573 | 68 | 1.13 (0.94–1.37) | ||
| 90–120 | 304 | 192 | 63 | 0.99 (0.74–1.33) | ||
| >120 | 1114 | 582 | 52 | 0.57 (0.49–0.67)* | ||
| Characteristics of Each Referral | Referrals | Appointment Scheduled | Referrals With Appointment Scheduled, % | P | aOR (95% CI) | |
|---|---|---|---|---|---|---|
| Total referrals | 20 466 | 13 261 | 65 | — | — | |
| Referral level | ||||||
| Referrer practice site | <.001* | |||||
| Primary care | 19 247 | 12 343 | 64 | Reference | ||
| Urgent care | 1219 | 908 | 74 | 1.19 (1.01–1.39) | ||
| Subspecialty type | <.001* | |||||
| Medical | 9858 | 5307 | 54 | Reference | ||
| Surgical | 10 608 | 7954 | 75 | 2.16 (1.99–2.34)* | ||
| Subspecialty appointment capacity | <.001* | |||||
| Large | 15 549 | 10 770 | 69 | Reference | ||
| Small | 4917 | 2491 | 51 | 0.46 (0.42–0.50)* | ||
| Season of referral | <.001* | |||||
| December–February | 4770 | 3396 | 71* | Reference | ||
| March–May | 5475 | 3192 | 58* | 0.49 (0.44–0.55)* | ||
| June–August | 5027 | 3194 | 64* | 0.68 (0.61–0.75)* | ||
| September–November | 5194 | 3479 | 67* | 0.81 (0.73–0.90)* | ||
| Child level | ||||||
| Age, y | <.001* | |||||
| <1 | 3212 | 2594 | 81 | Reference | ||
| ≥1–6 | 5993 | 3984 | 66 | 0.44 (0.39–0.50)* | ||
| ≥6–12 | 5796 | 3414 | 59 | 0.31 (0.27–0.35)* | ||
| ≥12–17 | 5465 | 3269 | 60 | 0.33 (0.29–0.38)* | ||
| Sex | .006* | |||||
| Female | 9959 | 6359 | 64 | 1.01 (0.93–1.08) | ||
| Male | 10 507 | 6902 | 66 | Reference | ||
| Race and/or ethnicity | <.001* | |||||
| Hispanic | 575 | 388 | 67* | 1.04 (0.82–1.33) | ||
| White, non-Hispanic | 9056 | 9056 | 68 | Reference | ||
| African American, non-Hispanic | 4841 | 2704 | 56* | 0.71 (0.64–0.79)* | ||
| Asian American and/or Pacific Islander | 347 | 239 | 69 | 0.89 (0.66–1.19) | ||
| Other or declined | 1303 | 874 | 67 | 0.79 (0.68–0.92)* | ||
| Primary language | .071 | |||||
| English | 19 909 | 12 880 | 65 | Reference | ||
| Non-English | 557 | 381 | 68 | 1.01 (0.79–1.29) | ||
| Insurance | <.001* | |||||
| Private | 10 096 | 6943 | 69 | Reference | ||
| Public | 10 370 | 6318 | 61 | 0.83 (0.77–0.90)* | ||
| Zip code median income, quartile | <.001* | |||||
| First (lowest) | 5205 | 3071 | 59* | 0.81 (0.72–0.91)* | ||
| Second | 50 042 | 3224 | 64 | 0.89 (0.80–1.00) | ||
| Third | 5313 | 3529 | 66* | 0.87 (0.79–0.97)* | ||
| Fourth (highest) | 4906 | 3437 | 70 | Reference | ||
| Travel time to pediatric hospital, min | <.001* | |||||
| ≤30 | 13 743 | 8814 | 64 | Reference | ||
| 30–60 | 446 | 3100 | 69 | 1.08 (0.98–1.19) | ||
| 60–90 | 841 | 573 | 68 | 1.13 (0.94–1.37) | ||
| 90–120 | 304 | 192 | 63 | 0.99 (0.74–1.33) | ||
| >120 | 1114 | 582 | 52 | 0.57 (0.49–0.67)* | ||
—, not applicable.
P < .032.
In adjusted analysis, referrals were significantly less likely to result in a scheduled appointment for children who were older (12–17 years old: adjusted odds ratio [aOR] 0.33; 95% confidence interval [CI] 0.29–0.38), identified as non-Hispanic African American (aOR 0.71; 95% CI 0.64–0.79), received public insurance (aOR 0.83; 95% CI 0.77–0.90), resided in a zip code with a lower median income (lowest quartile: aOR 0.81; 95% CI 0.72–0.91), and lived >120 minutes from the main academic hospital (aOR 0.57; 95% CI 0.49–0.67) compared with those in reference groups. Referrals were more likely to result in a scheduled appointment for children who were referred from an urgent care site (aOR 1.19; 95% CI 1.01–1.39) or to a surgical subspecialty (aOR 2.16; 95% CI 1.99–2.34). A sensitivity analysis with subspecialty-specific categories yielded similar findings (Supplemental Table 5).
Factors Associated With Attended Appointments
In an unadjusted analysis, the likelihood of a scheduled appointment resulting in an attended visit was significantly higher for referrals for the youngest children (P < .001), Asian American and/or Pacific Islander children (P < .001), children with private insurance (P < .001), and children from zip codes with a higher median income (P < .001; Table 3). The likelihood of a scheduled appointment resulting in an attended visit also increased for referrals from urgent care (P < .001), to specialties with a lower appointment capacity (P < .001), and when appointment dates were within 7 days of referral (P < .001).
Referral and Child Characteristics Associated With Attendance of Scheduled Appointments Among 11 808 Children With 1 or More Scheduled Appointments
| Characteristics | Scheduled Appointments | Visit Attended | Referrals With Appointment Scheduled, % | P | aOR (95% CI) |
|---|---|---|---|---|---|
| Total referrals | 13 261 | 10 514 | 79 | — | — |
| Referral | |||||
| Referrer practice site | <.001* | ||||
| Primary care | 12 353 | 9742 | 79 | Reference | |
| Urgent care | 908 | 772 | 85 | 0.64 (0.50–0.84)* | |
| Subspecialty type | .046 | ||||
| Medical | 5307 | 4162 | 78 | Reference | |
| Surgical | 7954 | 6352 | 80 | 0.88 (0.78–1.00) | |
| Subspecialty capacity | <.001* | ||||
| Large | 10 770 | 8475 | 79 | Reference | |
| Small | 2491 | 2039 | 82 | 1.11 (0.96–1.29) | |
| Season | .350 | ||||
| December–February | 3396 | 2712 | 80 | Reference | |
| March–May | 3192 | 2512 | 79 | 0.83 (0.70–0.97)* | |
| June–August | 3194 | 2555 | 80 | 0.97 (0.83–1.14) | |
| September–November | 3479 | 2735 | 79 | 0.85 (0.72–0.99) | |
| Travel time to appointment site, min | <.001* | ||||
| ≤30 | 9422 | 7364 | 78 | Reference | |
| 30–60 | 2625 | 2194 | 84 | 1.09 (0.93–1.28) | |
| 60–90 | 643 | 510 | 79 | 0.91 (0.69–1.20) | |
| 90–120 | 179 | 146 | 82 | 1.16 (0.71–1.91) | |
| >120 min | 392 | 300 | 77 | 0.77 (0.56–1.08) | |
| Interval from referral to scheduled appointment, d | <.001* | ||||
| 0–7 | 2703 | 2486 | 92 | Reference | |
| 8–14 | 1743 | 1504 | 86 | 0.48 (0.37–0.61)* | |
| 15–30 | 2902 | 2423 | 83 | 0.36 (0.29–0.45)* | |
| 31–60 | 3425 | 2561 | 75 | 0.18 (0.14–0.23)* | |
| 61–90 | 2488 | 1540 | 62 | 0.09 (0.07–0.12)* | |
| Child | |||||
| Age, y | <.001* | ||||
| <1 | 2594 | 2248 | 87 | Reference | |
| ≥1–6 | 3894 | 3137 | 79 | 0.70 (0.58–0.84)* | |
| ≥6–12 | 3414 | 2612 | 77 | 0.55 (0.45–0.66)* | |
| ≥12–18 | 3269 | 2517 | 77 | 0.45 (0.37–0.56)* | |
| Sex | .021* | ||||
| Female | 6359 | 4988 | 78 | 0.94 (0.84–1.06) | |
| Male | 6902 | 5526 | 80 | Reference | |
| Race and/or ethnicity | <.001* | ||||
| Hispanic | 388 | 302 | 78 | 0.79 (0.55–1.15) | |
| White, non-Hispanic | 9056 | 7536 | 83 | Reference | |
| African American, non-Hispanic | 2704 | 1754 | 65 | 0.45 (0.37–0.55)* | |
| Asian American and/or Pacific Islander | 239 | 207 | 87 | 1.04 (0.63–1.70) | |
| Other or declined | 874 | 715 | 82 | 0.77 (0.61–0.99)* | |
| Primary language | .161 | ||||
| English | 12 880 | 10 201 | 79 | Reference | |
| Non-English | 381 | 313 | 82 | 1.47 (0.98–2.19) | |
| Insurance | <.001* | ||||
| Private | 6943 | 5903 | 85 | Reference | |
| Public | 6318 | 4611 | 73 | 0.62 (0.54–0.71)* | |
| Zip code median income, quartile | <.001* | ||||
| First (lowest) | 3071 | 2149 | 70 | 0.68 (0.56–0.82)* | |
| Second | 3224 | 2512 | 78 | 0.83 (0.69–1.00) | |
| Third | 3529 | 2937 | 83 | 0.99 (0.83–1.18) | |
| Fourth (highest) | 3437 | 2916 | 85 | Reference |
| Characteristics | Scheduled Appointments | Visit Attended | Referrals With Appointment Scheduled, % | P | aOR (95% CI) |
|---|---|---|---|---|---|
| Total referrals | 13 261 | 10 514 | 79 | — | — |
| Referral | |||||
| Referrer practice site | <.001* | ||||
| Primary care | 12 353 | 9742 | 79 | Reference | |
| Urgent care | 908 | 772 | 85 | 0.64 (0.50–0.84)* | |
| Subspecialty type | .046 | ||||
| Medical | 5307 | 4162 | 78 | Reference | |
| Surgical | 7954 | 6352 | 80 | 0.88 (0.78–1.00) | |
| Subspecialty capacity | <.001* | ||||
| Large | 10 770 | 8475 | 79 | Reference | |
| Small | 2491 | 2039 | 82 | 1.11 (0.96–1.29) | |
| Season | .350 | ||||
| December–February | 3396 | 2712 | 80 | Reference | |
| March–May | 3192 | 2512 | 79 | 0.83 (0.70–0.97)* | |
| June–August | 3194 | 2555 | 80 | 0.97 (0.83–1.14) | |
| September–November | 3479 | 2735 | 79 | 0.85 (0.72–0.99) | |
| Travel time to appointment site, min | <.001* | ||||
| ≤30 | 9422 | 7364 | 78 | Reference | |
| 30–60 | 2625 | 2194 | 84 | 1.09 (0.93–1.28) | |
| 60–90 | 643 | 510 | 79 | 0.91 (0.69–1.20) | |
| 90–120 | 179 | 146 | 82 | 1.16 (0.71–1.91) | |
| >120 min | 392 | 300 | 77 | 0.77 (0.56–1.08) | |
| Interval from referral to scheduled appointment, d | <.001* | ||||
| 0–7 | 2703 | 2486 | 92 | Reference | |
| 8–14 | 1743 | 1504 | 86 | 0.48 (0.37–0.61)* | |
| 15–30 | 2902 | 2423 | 83 | 0.36 (0.29–0.45)* | |
| 31–60 | 3425 | 2561 | 75 | 0.18 (0.14–0.23)* | |
| 61–90 | 2488 | 1540 | 62 | 0.09 (0.07–0.12)* | |
| Child | |||||
| Age, y | <.001* | ||||
| <1 | 2594 | 2248 | 87 | Reference | |
| ≥1–6 | 3894 | 3137 | 79 | 0.70 (0.58–0.84)* | |
| ≥6–12 | 3414 | 2612 | 77 | 0.55 (0.45–0.66)* | |
| ≥12–18 | 3269 | 2517 | 77 | 0.45 (0.37–0.56)* | |
| Sex | .021* | ||||
| Female | 6359 | 4988 | 78 | 0.94 (0.84–1.06) | |
| Male | 6902 | 5526 | 80 | Reference | |
| Race and/or ethnicity | <.001* | ||||
| Hispanic | 388 | 302 | 78 | 0.79 (0.55–1.15) | |
| White, non-Hispanic | 9056 | 7536 | 83 | Reference | |
| African American, non-Hispanic | 2704 | 1754 | 65 | 0.45 (0.37–0.55)* | |
| Asian American and/or Pacific Islander | 239 | 207 | 87 | 1.04 (0.63–1.70) | |
| Other or declined | 874 | 715 | 82 | 0.77 (0.61–0.99)* | |
| Primary language | .161 | ||||
| English | 12 880 | 10 201 | 79 | Reference | |
| Non-English | 381 | 313 | 82 | 1.47 (0.98–2.19) | |
| Insurance | <.001* | ||||
| Private | 6943 | 5903 | 85 | Reference | |
| Public | 6318 | 4611 | 73 | 0.62 (0.54–0.71)* | |
| Zip code median income, quartile | <.001* | ||||
| First (lowest) | 3071 | 2149 | 70 | 0.68 (0.56–0.82)* | |
| Second | 3224 | 2512 | 78 | 0.83 (0.69–1.00) | |
| Third | 3529 | 2937 | 83 | 0.99 (0.83–1.18) | |
| Fourth (highest) | 3437 | 2916 | 85 | Reference |
—, not applicable.
P < .032.
In an adjusted analysis, appointment attendance significantly decreased for referrals for older children (aOR 0.45; 95% CI 0.37–0.56), children identifying as non-Hispanic African American (aOR 0.45; 95% CI 0.37–0.55), children with public insurance (aOR 0.62; 95% CI 0.54–0.71), and children from zip codes with the lowest median income (aOR 0.68; 95% CI 0.56–0.82). In an adjusted analysis, attended appointments decreased for referrals from urgent care (aOR 0.64; 95% CI 0.50–0.84) and with longer intervals from the referral to the scheduled appointment (61–90 days: aOR 0.09; 95% CI 0.07–0.12). A sensitivity analysis with a subspecialty-specific variable again yielded similar findings (Supplemental Table 5).
Because of the significant association between the time to the scheduled appointment and appointment attendance, we examined the median time to the scheduled appointment across referral and child characteristics (Supplemental Table 6). Shorter intervals from the referral to the scheduled appointment were associated with referral from urgent care (5 days versus 28 days for referral from primary care; P < .001), to surgical subspecialists (23 days versus 30 days for referral to medical subspecialties; P < .001), to small-capacity subspecialties (24 days versus 27 days for referrals to high-capacity subspecialties; P < .001), and from March to May (23 vs 32 days for referrals from December to February; P ≤ .001). Shorter intervals were also associated with referrals for children with specific patient characteristics, including younger age, (P < .001), Asian American and/or Pacific Islander race (P < .001), private insurance (P < .001), and 30- to 60-minute travel times (P < .001).
Discussion
In this large primary care network, only 51% of referrals resulted in a visit completed within 90 days. Approximately one-sixth of all referrals resulted in a scheduled but unattended appointment, and one-third of all referrals had no appointment scheduled within 90 days. This overall referral completion rate was lower than the 70% completion rate found in 1 previous pediatric study of referrals from 2 health centers19 but higher than a recent analysis in an adult health system, where 35% of referrals were completed.26 We also found that the rates of 2 separate steps, referral scheduling and visit attendance, varied significantly with patient and system factors.
We found significant variation in scheduling and attendance rates across multiple system factors. Among scheduled appointments, longer intervals from the referral to the scheduled appointment date decreased the likelihood of visit attendance. Previous work reported that attendance was improved in appointments scheduled within 60 days of the subspecialty referral, and a recent study also demonstrated lower no-show rates for ophthalmology appointments scheduled with a shorter lead time.19,27 In our adjusted analysis, even appointments 8 to 14 days after referral were substantially less likely to be attended than those scheduled for 0 to 7 days after. Our cross-sectional data cannot determine if shorter intervals from the referral to the scheduled appointment have a causal impact on visit attendance; an alternative explanation is that unmeasured variables increased both the likelihood of an earlier appointment and visit attendance (eg, increased clinical acuity, increased parental anxiety, and/or increased PCP advocacy for an earlier appointment). Future research could test whether interventions to decrease the interval to appointments improve appointment attendance. Although providing appointments within a week or even a month remains a substantial challenge for many pediatric subspecialties,5 specific interventions to decrease scheduling intervals (eg, open access), increase appointment capacity (eg, advanced practice providers or primary care physicians in subspecialty care clinics), or reduce referral burden (eg referral guidelines or electronic consultations) are worth examining for their impact on the visit interval and subsequently visit attendance, no-show rates, and subspecialty clinic efficiency.18,28,29
Among related systems factors, we observed that referrals from urgent care and to surgical specialties were more likely to result in a scheduled appointment in unadjusted analysis and were associated with more timely appointments. Although referral from urgent care was also associated with an increased likelihood of visit attendance in an unadjusted analysis, referral from urgent care was associated with a decreased likelihood of visit attendance after accounting for the timeliness of the appointment. These results further support the association between appointment timeliness and visit attendance while suggesting opportunities to learn from strategies employed by specific specialties and referrers to improve the timeliness of appointment scheduling.
Additional variables highlight the value of separately assessing appointment scheduling and visit attendance. As one might expect, subspecialties with lower appointment capacity had lower appointment scheduling, but appointment capacity was not significantly associated with visit attendance among scheduled appointments. Season of referral was also associated with scheduling but not attendance. The lowest seasonal rates of scheduling occurred in the springtime months. One possible explanation is that appointments generated from these referrals would fall in the summer months, when family and physician schedules may be disrupted by summer recesses, travel, and new trainees requiring higher levels of supervision. Understanding whether seasonal variation is driven more by shifts in family availability versus appointment availability could drive additional improvement strategies.
In addition to the above system factors, several patient factors were significantly associated with referral scheduling and visit attendance. Specifically, African American race, public insurance, and lower zip code median income were each associated with both unscheduled and unattended appointments. Because demographic factors were associated with both decreased scheduling of appointments and decreased attendance at scheduled appointments, there is potential value for interventions at both of these phases with a focus on specific populations’ needs to address disparities in referral completion. Of note, these factors (African American race, public insurance, and lower zip code median income) were also associated with longer time intervals to appointment scheduling. Notably, these disparities in scheduling were observed within a system in which subspecialist schedulers were tasked with actively calling each of these families, suggesting a need to explore alternative means of contacting families (eg, texting), address alternative barriers to scheduling (eg, unpredictable work schedules, limited health literacy, or previous experience of perceived prejudice or racism in health care settings), and consider possible implicit or explicit biases in appointment scheduling processes. Also, although extreme travel time was associated with decreased visit scheduling, we found that visit attendance among scheduled appointments did not differ across travel time. This suggests that families may account for the realities of travel time in their decisions to schedule, but those who are scheduling are equally likely to attend despite prolonged travel times. Interventions designed to address geographic barriers, such as telemedicine, might then have the largest impact on appointment scheduling (rather than attendance rates among scheduled appointments) for patients in these communities.
This study was limited by the data available. Most notably, these data did not include why patients declined to schedule or why they canceled or missed appointments. These data also did not include information on subspecialty visits that occurred outside the studied health system. However, we limited our analysis only to referrals routed specifically to CHP subspecialists for completion within the studied health system. Additionally, we note that institutional variation in subspecialty clinic structure and subspecialist supply may limit the generalizability of subspecialty-specific data. For this reason, we focused on more generalizable characteristics of subspecialties (medical versus surgical and high versus low capacity) in our main analysis while acknowledging subspecialty-specific variation and including more details on subspecialty-specific referral reasons and outcomes for specific subspecialties in our supplemental analyses. Next, we could not address the appropriateness or urgency of referral. Some unscheduled referrals and unattended appointments may represent unnecessary referrals that did not truly require a subspecialty visit or cases in which symptoms resolved so that the need for referral resolved. For this reason, we would not expect all referrals to be completed, but we do seek to address unwarranted variation in scheduling and completion. Finally, we note that our analysis occurred within the setting of a relatively unfragmented referral catchment area with an electronically transmitted referral system in place. These features likely make our results best case scenarios by simplifying scheduling choices and steps for families, which is consistent with previous work showing that the implementation of electronically transmitted referral improved referral completion.16 It is also possible that electronically transmitted referrals help reduce disparities by reaching out to families with communication barriers or time constraints, in which case other settings may also observe larger disparities across child characteristics. However, the presence of the electronically transmitted referrals within a large, relatively unfragmented catchment area also enhances the quality of our data for this analysis (by comprehensively capturing referrals and subsequent visits), thereby enhancing the internal validity of our findings.
Conclusions
We identified system-level factors associated with referral scheduling and appointment attendance among children referred for pediatric subspecialty care, including referral site, subspecialty type, subspecialty capacity, and interval from referral to scheduled appointment. We also identified inequities in appointment scheduling and visit attendance associated with patient sociodemographic characteristics. Targeting barriers to appointment scheduling and visit attendance may result in more efficient and equitable access to pediatric subspecialty care.
Dr Bohnhoff conceptualized and designed the study, analyzed the data, drafted the initial manuscript, and reviewed and revised the manuscript; Ms Taormina collected the data, assisted with interpretation, and critically revised the manuscript; Ms Ferrante and Dr Wolfson assisted with interpretation and critically revised the manuscript; Dr Ray supervised study conceptualization and design as well as analysis, interpreted results, and critically revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
FUNDING: No external funding.
COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2019-2975.
