Video Abstract

Video Abstract

Close modal
BACKGROUND

Telemedicine is widely used but has uncertain value. We assessed telemedicine to further improve outcomes and reduce costs of comprehensive care (CC) for medically complex children.

METHODS

We conducted a single-center randomized clinical trial comparing telemedicine with CC relative to CC alone for medically complex children in reducing care days outside the home (clinic, emergency department, or hospital; primary outcome), rate of children developing serious illnesses (causing death, ICU admission, or hospital stay >7 days), and health system costs. We used intent-to-treat Bayesian analyses with neutral prior assuming no benefit. All participants received CC, which included 24/7 phone access to primary care providers (PCPs), low patient-to-PCP ratio, and hospital consultation from PCPs. The telemedicine group also received remote audiovisual communication with the PCPs.

RESULTS

Between August 22, 2018, and March 23, 2020, we randomly assigned 422 medically complex children (209 to CC with telemedicine and 213 to CC alone) before meeting predefined stopping rules. The probability of a reduction with CC with telemedicine versus CC alone was 99% for care days outside the home (12.94 vs 16.94 per child-year; Bayesian rate ratio, 0.80 [95% credible interval, 0.66–0.98]), 95% for rate of children with a serious illness (0.29 vs 0.62 per child-year; rate ratio, 0.68 [0.43–1.07]) and 91% for mean total health system costs (US$33 718 vs US$41 281 per child-year; Bayesian cost ratio, 0.85 [0.67–1.08]).

CONCLUSION

The addition of telemedicine to CC likely reduced care days outside the home, serious illnesses, other adverse outcomes, and health care costs for medically complex children.

What’s Known of this subject:

Few previous studies have rigorously addressed the effectiveness or cost-effectiveness of telemedicine for direct patient‐provider communication as an alternative to face‐to‐face care for children with chronic conditions.

In this randomized trial, the Bayesian probability was 91% to 99% that telemedicine with comprehensive care reduced days of care in a medical setting (primary outcome, 12.94 vs 16.94 days per child-year with comprehensive care alone), children with serious illnesses, and health system costs.

Medically complex children1,2  account for only 0.4% of all children but are responsible for ∼40% of pediatric deaths3  and 53% of all pediatric hospital charges.4  All too often, their care is fragmented, ineffective, and inefficient.5,6

To promote prompt effective care for medically complex children at all hours, we developed an outpatient comprehensive care (CC) program at the University of Texas Health Science Center at Houston (UTH) that now includes a hospital consultation service. As a result, CC has substantially reduced serious illnesses, hospital days, PICU days, and health system costs for medically complex children in our center.68  Even so, their need for medical attention and the costs of care remain distressingly high.

Optimizing the outcomes and reducing the costs of medically complex children are likely to require an integrated approach to augment care in the home as well as the clinic and hospital. Use of telemedicine to augment their care in the home might be highly beneficial for families of medically complex children, many of whom face financial, transportation, and health system obstacles, particularly for technology-dependent medically complex children. Moreover, any time spent in a medical setting imposes a risk of acquiring serious, even life-threatening, infections. However, there is a dearth of research assessing the implementation and effects of telemedicine for medically complex children.

For these reasons, we conducted a randomized quality improvement (QI) trial comparing CC with telemedicine (treatment hereafter referred to as telemedicine) versus CC alone. Bayesian analyses913  were used to assess the overall probability and magnitude of benefit in reducing the total days of care outside the home (primary outcome), serious illnesses, other adverse outcomes, and health system costs.

Our institutional review board approved the trial as a QI trial1417  to verify the benefits of an intervention to increase access to care for medically complex children, promote better informed decisions by primary care providers (PCPs) than would otherwise be made over the telephone, and facilitate prompt recognition and treatment of serious illnesses while avoiding unnecessary clinic or emergency department (ED) visits and needless exposures, expense, inconvenience, school absences, and parental time off work.1820  As allowed under federal regulations for QI studies judged unlikely to increase patient risk,15,17  our institutional review board waived the consent requirements. As in many QI studies, parental consent was not obtained for the QI intervention (telemedicine) being assessed, although of course every parent was aware of their own child’s care, and no child received less care than would have been provided in the absence of the study.

This QI study had the design of a hybrid type 1 effectiveness-implementation trial21  in which we worked to fully implement, augment, and verify telemedicine benefits and cost-effectiveness.

We included patients treated in the High-Risk Children's Clinic (HRCC) at UTH. All had ≥2 hospitalizations, or ≥1 PICU admission in the year before clinic enrollment and a >50% estimated risk of hospitalization in the absence of CC as judged by the clinic’s medical director (R.M.) on the basis of the patient’s diagnoses and clinical course and prospectively validated.6,7  We excluded medically complex children with specific disorders given primary care outside our clinic (eg, serious unrepaired congenital heart disease) or a do-not-resuscitate order.

Partly because the number of routine visits depended on age, patients were stratified by age (<2 years or older) as well as estimated baseline risk (risk level 1 [positive-pressure ventilation], risk level 2 [above the expected median risk but not ventilator-dependent], and risk level 3 [at or below the expected median risk]), as judged by the clinic’s medical director (R.M.). All eligible patients already treated in the HRCC were randomly assigned to either telemedicine or CC at the outset of the trial. All later eligible patients were randomly assigned during their first HRCC visit except for infants receiving continued mechanical ventilation after discharge from the NICU, who were randomly assigned within 24 hours after discharge. A study nurse (M.P.), uninvolved in patient care, randomly assigned all patients using a computer-generated algorithm in the Research Electronic Data Capture with variable block sizes. Enrollment occurred between August 22, 2018, and March 23, 2020 (before coronavirus disease 2019 [COVID-19] was identified in the families of any of our patients).

CC was provided to all trial participants to promote prompt effective care at all hours. As previously detailed,68  CC included highly experienced PCPs and subspecialists working in the same clinic, 24/7 access to the clinic emergency cell phone that is rotated between the PCPs, same-day care on weekdays in the clinic for acute illnesses and complications, a low patient-to-PCP ratio (≤100:1), weekly scrutiny of the care received before any ED visit or hospitalization, and a hospital consultation service from the CC providers during hospitalizations in our primary teaching hospital, the Children’s Memorial Hermann Hospital (CMHH).8

Telemedicine was exclusively available to this group. Telemedicine was provided by Zoom for Healthcare, an application compliant to the Health Insurance Portability and Accountability Act that offered reliable connectivity through cellular data for families without Wi-Fi and could be downloaded to any smart phone at no charge. A bilingual staff member assisted each family in the telemedicine group in downloading the application, creating a Zoom account, and learning to use Zoom for audiovisual communication with our PCPs. Several meetings with the telemedicine parents were often required, particularly when new phones were obtained.

The parents in this treatment group were asked if they would like a telemedicine visit whenever they called the clinic for an acute illness. The PCPs chose whether to schedule return visits or health maintenance visits by telemedicine or to communicate with parents using telemedicine at nights and on weekends. At the study outset, telemedicine had not been used in any pediatric clinic in our center, and the parents often declined to have telemedicine visits. In addition, our PCPs who were all experienced in communicating with patients by telephone at all hours7,8  were initially skeptical that telemedicine would add much benefit to CC. In promoting use of telemedicine, the weekly meetings between the medical director (R.M.) and the PCPs and staff included a discussion of any difficulties encountered and potential ways to improve the use of telemedicine for both scheduled follow-up checks and sick calls. He also provided periodic updates of the data to assure the staff of its safety and reviewed the number of telemedicine visits by each PCP.

Although our providers and parents could not be blinded to the treatment provided, our statisticians (C.P. and K.L.) and health care economist (E.A.) were unaware of treatment group.

The staff tabulated all clinic visits (including vaccine-only appointments) and telemedicine visits for all study patients. The type of visit, reason for the telemedicine visit, and the intervention recommended after a telemedicine encounter were abstracted from the clinic electronic medical records (EMR).

These included the total number of days of care outside of the home (in the clinic, ED, or hospital) (primary outcome), office visits (excluding well-child checks), well-child checks, hospital admissions, readmissions (within 30 days of discharge), PICU admissions, and serious illnesses (causing death, PICU admission, or hospital stay >7 days). To identify clinic and telemedicine visits, we assessed the clinic EMR. Hospital days, ED visits, and hospitalizations were identified by assessing the CMHH EMR and claims data from the 14 hospitals in the Memorial Hermann Hospital System (MHHS). In addition, parents were also asked at each visit about any medical care since previous visit.

A health care economist (E.A.) conducted detailed evaluations of hospital and clinic costs from the health system perspective. For each ED visit and hospitalization in the MHHS, total hospital-specific costs were obtained from the cost-accounting system for these hospitals. Costs for physician services provided at CMHH were estimated by using UTH claims data and relative value units (RVUs) from the 2020 Medicare Fee Schedule.22  The costs at other hospitals outside MHHS (7% of total admissions and 6% of all ED visits) were estimated on the basis of the mean costs of these encounters at MHHS.

Clinic costs for CC were estimated by using the HRCC total expenditures to account for the low patient-to-PCP ratio and the provision of multiple nonbillable services not captured by conventional claims-based methods. The mean personnel time spent providing an office visit, a telemedicine visit, and an inpatient consultation was assessed by surveying the PCPs. The total time spent for each service was then multiplied by the observed number of such encounters and the unit time costs based on the PCPs’ salary, fringes, and overhead costs. The costs for telemedicine included the monthly fees for Zoom for Healthcare and the staff time assisting patients with Zoom installation and use. To capture the full costs of providing CC, the above tabulated costs were subtracted from the total HRCC expenditures and the otherwise unaccounted for CC costs were allocated to patients on the basis of their length of follow-up. Costs for outpatient services occurring at our institution but outside our clinic (eg, diagnostic imaging) were calculated by using the standard RVU-based methodology.22

All costs were inflated to 2020 on the basis of the Consumer Price Index for medical services.23

As in previous trials,8,24,25  we prespecified use of Bayesian analyses to avoid the limitations and the pervasive misinterpretation of P values24,25  and to address the issue of central importance to patients and providers: the probability that an intervention is effective based on previous knowledge or assumptions and the results of the trial.2628  Although telemedicine was considered likely to be beneficial, we conservatively assumed a neutral prior probability29  centered at a relative risk of 1.0 (no effect on days of care outside home) and a 95% credible interval of 0.3–3.3, encompassing the largest likely effect size for major outcomes in randomized trials. This neutral prior probability was also used for the secondary outcomes.

Intent-to-treat analyses were performed. Negative binomial models were used to assess total number of outcomes. Final models with the lowest k-fold information criterion were chosen after performing k-fold cross validation with k = 10.30  Binary outcomes were analyzed with logistic models and estimates of relative risk derived from the models’ predicted probabilities of outcomes.31  Costs were assessed by using generalized linear mixed models with log-link and γ distribution. To account for the patients who did not incur any hospital-related costs, a zero-inflated negative binomial distribution was used to analyze hospital costs. All models included age and baseline risk factor (stratifying variables) as covariates and length of follow-up as an offset variable. The models for total and clinic costs also included a random intercept for siblings (to account for within-family correlation); we note that for clinical outcomes and hospital costs there was not enough variability to estimate this extra parameter.

REDCap software was used to collect and manage the study data. All the analyses were performed by using R software version 3.6.2. (R Foundation for Statistical Computing) with the brms package.30

To assess the largest sample size feasible with the available resources, we originally planned to enroll almost all eligible patients for 2 years with a goal of accruing ∼400 patients. To avoid unduly prolonging this QI trial, our predefined stopping guideline32  was to discontinue enrollment if an interim Bayesian analyses at the end of the trial’s first year (using a neutral prior) identified a ≥75% posterior probability that telemedicine reduced total care days outside the home.

Enrollment occurred from August 22, 2018, to March 23, 2020, when the first interim analysis identified a 95% probability that telemedicine decreased total days of care outside the home. (Because telemedicine was implemented more slowly than expected [see below], the preplanned interim analysis was deferred from 12 to 19 months after the trial was started. The analysis at that point was prompted by concerns that we should make telemedicine available to all medically complex children during the impending COVID-19 pandemic.).

Excluding 63 ineligible children in our clinic, we randomly assigned 422 medically complex children, 209 to telemedicine and 213 to CC alone (Fig 1). Two medically complex children randomly assigned to telemedicine were unable to install the Zoom application because of lack of Internet access or a smart phone but were included in our intent-to-treat analysis. The 2 groups were similar at baseline (Table 1). A total of 259 child-years of telemedicine and 253 child-years of CC alone were provided. The mean length of follow-up per child was 1.24 years (95% confidence interval [CI], 1.17–1.30) for telemedicine and 1.19 years (95% CI, 1.12–1.26) for CC alone.

FIGURE 1

Patient recruitment, randomization, and follow-up. a1 patient had >1 exclusion criteria.

FIGURE 1

Patient recruitment, randomization, and follow-up. a1 patient had >1 exclusion criteria.

Close modal
TABLE 1

Baseline Characteristics by Treatment Group

Telemedicine and CC (n = 209)CC Alone (n = 213)
Age at enrollment, y, mean ± SD 6.2 ± 5.4 5.7 ± 4.5
Male sex, No. (%) 121 (58) 117 (55)
Race and ethnicity, No. (%)
Hispanic 109 (52) 88 (41)
Black 56 (27) 82 (38)
Non-Hispanic white 21 (10) 21 (10)
Other 23 (11) 22 (10)
Estimated baseline risk, No. (%)
Risk level I mechanical ventilation 52 (25) 52 (24)
Risk level II, nonventilated and above or at median expected risk 67 (32) 70 (33)
Risk level III, nonventilated and below median expected risk 90 (43) 91 (43)
Unique families, No. 197 200
Insurance status, No. (%)
Medicaid 182 (87) 188 (88)
Commercial plans 27 (13) 25 (12)
Maternal level of education, less than high school diploma 38 (18) 31 (15)
Prematurity, No. (%) 102 (49) 105 (49)
Chronic diseases, No. (%)
Cardiac disorder 27 (13) 18 (8)
Congenital disorder 111 (53) 123 (58)
ENT disorder 60 (29) 68 (32)
GI disorder 107 (51) 124 (58)
Immunologic disorder 3 (1) 3 (1)
Musculoskeletal disordera 27 (13) 35 (16)
Neurologic disorderb 97 (46) 99 (46)
Respiratory disorder 142 (68) 139 (65)
2 or more disorders 157 (75) 168 (79)
3 or more disorders 118 (56) 126 (59)
Telemedicine and CC (n = 209)CC Alone (n = 213)
Age at enrollment, y, mean ± SD 6.2 ± 5.4 5.7 ± 4.5
Male sex, No. (%) 121 (58) 117 (55)
Race and ethnicity, No. (%)
Hispanic 109 (52) 88 (41)
Black 56 (27) 82 (38)
Non-Hispanic white 21 (10) 21 (10)
Other 23 (11) 22 (10)
Estimated baseline risk, No. (%)
Risk level I mechanical ventilation 52 (25) 52 (24)
Risk level II, nonventilated and above or at median expected risk 67 (32) 70 (33)
Risk level III, nonventilated and below median expected risk 90 (43) 91 (43)
Unique families, No. 197 200
Insurance status, No. (%)
Medicaid 182 (87) 188 (88)
Commercial plans 27 (13) 25 (12)
Maternal level of education, less than high school diploma 38 (18) 31 (15)
Prematurity, No. (%) 102 (49) 105 (49)
Chronic diseases, No. (%)
Cardiac disorder 27 (13) 18 (8)
Congenital disorder 111 (53) 123 (58)
ENT disorder 60 (29) 68 (32)
GI disorder 107 (51) 124 (58)
Immunologic disorder 3 (1) 3 (1)
Musculoskeletal disordera 27 (13) 35 (16)
Neurologic disorderb 97 (46) 99 (46)
Respiratory disorder 142 (68) 139 (65)
2 or more disorders 157 (75) 168 (79)
3 or more disorders 118 (56) 126 (59)

ENT, ear, nose, and throat; GI, gastrointestinal.

a

Achondroplasia, chest wall malformations, congenital scoliosis, congenital ribs absences, hip dislocation.

b

Cerebral palsy, epilepsy, hypoxemia ischemic encephalopathy, severe developmental delay, mitochondrial disorder.

Of the 313 telemedicine visits, 51% were follow-up visits (for chronic conditions or after an acute illness, ED visit or hospitalization) and 49% were sick visits. Half the encounters (51%) did not result in any change in care. Of the remaining encounters, 56% involved a pharmacologic change, 20% a nonpharmacologic change, and 24% involved both.

With increasing acceptance of telemedicine by our PCPs and parents, total telemedicine visits increased gradually from 10 in the trial’s first month to 55 in its last month, with a mean of 16 telemedicine visits per month during the study period.

Table 2 presents clinical outcomes. Despite limited initial use of telemedicine, the probability of an overall reduction with telemedicine versus CC alone was 99% for care days outside the home (12.94 vs 16.94 days per child-year; Bayesian rate ratio [RR], 0.80 [95% credible interval, 0.66–0.98) (Fig 2) and 95% for rate of children with a serious illness (0.29 vs 0.62 per child-year; RR, 0.68 [0.43–1.07]). Except for in-person well-child visits, the findings for all other clinical outcomes consistently favored the telemedicine group with probabilities of a reduction ranging from 66% to 98%, including total hospital admissions (90%) and PICU admissions (97%).

FIGURE 2

Posterior probability of decreased days of care outside the home with the telemedicine intervention versus CC alone. The curves represent probability densities and are scaled so that the total area under the curve is 1, and the area between any 2 values on the x-axis equals the probability of observing a value in that range. The green line plots a neutral prior distribution centered at an RR of 1.0 (95% credible interval, 0.3–3.3) and indicates an equal number of children would be expected to benefit from either CC with telemedicine or CC alone. The posterior probability of intervention effect is derived by combining the prior distribution with the trial results. The posterior distribution (black line) is shifted to the left of an RR of 1.0 with a median of 0.80. The area under the curve that is less than an RR of 1.0 (blue) represents the posterior probability of any reduction in days of care outside the home (99% for this trial). The area under the curve that is greater than an RR of 1.0 (salmon pink) represents the posterior probability of an increase in total days of care outside the home in the telemedicine group (1% for this trial).

FIGURE 2

Posterior probability of decreased days of care outside the home with the telemedicine intervention versus CC alone. The curves represent probability densities and are scaled so that the total area under the curve is 1, and the area between any 2 values on the x-axis equals the probability of observing a value in that range. The green line plots a neutral prior distribution centered at an RR of 1.0 (95% credible interval, 0.3–3.3) and indicates an equal number of children would be expected to benefit from either CC with telemedicine or CC alone. The posterior probability of intervention effect is derived by combining the prior distribution with the trial results. The posterior distribution (black line) is shifted to the left of an RR of 1.0 with a median of 0.80. The area under the curve that is less than an RR of 1.0 (blue) represents the posterior probability of any reduction in days of care outside the home (99% for this trial). The area under the curve that is greater than an RR of 1.0 (salmon pink) represents the posterior probability of an increase in total days of care outside the home in the telemedicine group (1% for this trial).

Close modal
TABLE 2

Clinical Outcome Measures by Treatment Group

Telemedicine and CC (n = 209)aCC Alone (n = 213)bBayesian RR (95% CrI)dProbability of Reduction,d,e %
Observed Rate per Child-YearAdjusted Rate per Child-YearcObserved Rate per Child-YearAdjusted Rate per Child-Yearc
Days of care outside the home (in the clinic, ED, or hospital) 12.94 15.15 16.94 18.84 0.80 (0.66–0.98) 99
In-person clinic visits
Office visits (excluding well-child checks) 5.90 6.66 7.25 7.46 0.89f (0.77–1.04) 93
Well-child checks 0.99 1.06 0.88 1.02 1.05 (0.87–1.26) 32
Total ED visits 1.47 1.59 1.98 1.81 0.88f (0.67–1.15) 82
Total admissions 1.01 1.06 1.23 1.28 0.83 (0.62–1.11) 90
Admissions with LOS >7 d 0.26 0.24 0.27 0.28 0.86 (0.51–1.42) 73
30-d readmissions 0.24 0.27 0.29 0.31 0.86f (0.41–1.81) 66
Total hospital days 6.24 8.20 9.05 10.64 0.77f (0.50–1.18) 88
PICU admissions 0.38 0.39 0.67 0.60 0.66 (0.42–1.03) 97
Total days in PICU 1.92 3.10 4.04 5.02 0.61f(0.33–1.16) 94
Children with serious illnesses 0.29 0.24 0.62 0.30 0.68g (0.43–1.07) 95
Episodes of serious illnesses 0.46 0.46 0.91 0.71 0.66 (0.44–0.99) 98
Death 0.06 0.02 0.32 0.03 0.74g (0.32–1.69) 76
Telemedicine and CC (n = 209)aCC Alone (n = 213)bBayesian RR (95% CrI)dProbability of Reduction,d,e %
Observed Rate per Child-YearAdjusted Rate per Child-YearcObserved Rate per Child-YearAdjusted Rate per Child-Yearc
Days of care outside the home (in the clinic, ED, or hospital) 12.94 15.15 16.94 18.84 0.80 (0.66–0.98) 99
In-person clinic visits
Office visits (excluding well-child checks) 5.90 6.66 7.25 7.46 0.89f (0.77–1.04) 93
Well-child checks 0.99 1.06 0.88 1.02 1.05 (0.87–1.26) 32
Total ED visits 1.47 1.59 1.98 1.81 0.88f (0.67–1.15) 82
Total admissions 1.01 1.06 1.23 1.28 0.83 (0.62–1.11) 90
Admissions with LOS >7 d 0.26 0.24 0.27 0.28 0.86 (0.51–1.42) 73
30-d readmissions 0.24 0.27 0.29 0.31 0.86f (0.41–1.81) 66
Total hospital days 6.24 8.20 9.05 10.64 0.77f (0.50–1.18) 88
PICU admissions 0.38 0.39 0.67 0.60 0.66 (0.42–1.03) 97
Total days in PICU 1.92 3.10 4.04 5.02 0.61f(0.33–1.16) 94
Children with serious illnesses 0.29 0.24 0.62 0.30 0.68g (0.43–1.07) 95
Episodes of serious illnesses 0.46 0.46 0.91 0.71 0.66 (0.44–0.99) 98
Death 0.06 0.02 0.32 0.03 0.74g (0.32–1.69) 76

CrI, credible interval; LOS, length of stay.

a

There were 259 child-years in the telemedicine and CC group.

b

There were 253 child-years in the CC group.

c

The estimated marginal (conditional) rate of events were obtained from the Bayesian models.

d

A Bayesian negative binomial generalized linear regression model (with a log-link function) was used with adjustment for age strata, baseline risk, and length of follow-up (unless specified otherwise).

e

Posterior probability of RR <1 (rounded to nearest percent) obtained from Bayesian model with a neutral prior centered at RR of 1 with 95% previous credible interval of 0.3–3.3. Previous distributions for regression coefficients were normal with mean of 0 and SD of 0.67 for intervention group and 1.0 for all other covariates. Bayesian models were fitted using Markov chain Monte Carlo methods (3 chains with a burn-in of 2000 with an additional 10 000 iterations for each chain).

f

A Bayesian negative binomial regression model was used with adjustment for length of follow-up and with the shape parameter varying with intervention group, age strata, and baseline risk.

g

A Bayesian binary logistic regression model was used with adjustment for baseline risk, age strata, and length of follow-up.

Table 3 summarizes the study’s economic analyses. The probability that telemedicine reduced mean total health system costs per child-year was 91% ($33 718 with telemedicine vs$41 281 with CC alone; Bayesian cost ratio, 0.85 [0.67–1.08]), owing mainly to reduced costs for hospital and ED services ($26 759 with telemedicine vs$34 419 with CC alone; cost ratio, 0.77 [0.54–1.10]).

TABLE 3

Estimated Health System Costs per Child-Year by Treatment Group

Observed Costs per Child-Year, $, Mean (95% CI)Bayesian Cost Ratio (95% CrI)bProbability of Cost Reduction,c,d % Telemedicine and CC (n = 209)CC Alone (n = 213)a Total health system costs 33 718 (23 185–44 250) 41 281 (28 568–53 993) 0.85 (0.67–1.08) 91 Costs of hospital and ED services 26 759 (16 402–37 116) 34 419 (21 837–47 002) 0.77e (0.54–1.10) 93e Costs of outpatient clinic services 6958 (6507–7410) 6861 (6373–7350) 1.02 (0.96–1.09) 23 Observed Costs per Child-Year,$, Mean (95% CI)Bayesian Cost Ratio (95% CrI)bProbability of Cost Reduction,c,d %
Telemedicine and CC (n = 209)CC Alone (n = 213)a
Total health system costs 33 718 (23 185–44 250) 41 281 (28 568–53 993) 0.85 (0.67–1.08) 91
Costs of hospital and ED services 26 759 (16 402–37 116) 34 419 (21 837–47 002) 0.77e (0.54–1.10) 93e
Costs of outpatient clinic services 6958 (6507–7410) 6861 (6373–7350) 1.02 (0.96–1.09) 23

Values are inflated to 2020 US dollars. CrI, credible interval.

a

There were 259 child-years in the telemedicine and CC group.

b

There were 253 child-years in the usual CC group.

c

Unless specified otherwise, the Bayesian estimates of differences in costs were obtained by using multilevel mixed effects generalized linear models with gamma distribution and log-link, adjusting for age strata, baseline risk, length of follow-up, and within-family correlation.

d

Posterior probability of cost ratio <1 rounded to the nearest percent and obtained from Bayesian model with a skeptical prior centered at cost ratio of 1.0 (no cost difference) with 95% previous interval of 0.3–3.3. Previous distributions for regression coefficients were normal with mean of 0 and SD of 0.67 for intervention group and 1.0 for all other covariates. Bayesian models were fitted by using Markov chain Monte Carlo methods (3 chains with a burn-in of 2000 with an additional 10 000 iterations for each chain).

e

To account for the patients who did not incur hospital costs during the trial, the costs of hospital and ED services were analyzed by using mixed effects generalized linear models with a zero-inflated negative binomial distribution and log-link, adjusting for age strata, baseline risk, and length of follow-up.

The estimated total costs of telemedicine during the study period were $79 766 (or$308 per child-year), accounting for 4.4% of the total outpatient costs in the telemedicine group; 63% ($50 101) of the telemedicine costs was for the extensive personnel time needed to assist patients with Zoom (25% full-time equivalent [FTE] of a staff member, 11% FTE of a registered nurse, and 2.5% FTE of a research nurse), 33% ($26 498) was for the PCPs’ time providing telemedicine consultations (mean length of telemedicine consultations was 41 minutes), and 4% ($3167) was for expenditures on the Zoom for Healthcare account for 10 hosts. As recently emphasized by Menon and Belcher,33 telehealth has major potential benefits, particularly for disadvantaged children for whom health care may be least accessible. However, the value of telemedicine has been variable and disputed across a broad range of conditions and age groups,34,35 and few randomized trials have assessed the effectiveness or cost-effectiveness of telemedicine to improve the outcomes of children with chronic conditions. As in an increasing number of clinical trials,9,12,24,29,36 we used Bayesian analyses to estimate the (posterior) probability of benefit or harm from the intervention and to avoid the limitations of the dichotomous frequentist approach that requires either rejecting or failing to reject the null hypothesis based on an arbitrary significance level.914 ,2628 ,37 Recent publications,38 including a commentary in Nature with >800 signatories,27 have called for avoiding dichotomous thinking and abandoning the term statistical significance in the medical literature. We also used a hybrid effectiveness-implementation trial design that allowed us to identify and address the barriers to delivering telemedicine, in particular the initial hesitation among PCPs and the intense clinic staff support needed to ensure family comfort and familiarity with Zoom, while gradually augmenting it and verifying its overall benefits and cost-effectiveness. Our Bayesian analyses identified a 99% probability of reduced days of care in the clinic, ED, or hospital, a 95% probability of a reduced rate of children with serious illness, and a 91% probability of reduced health system costs. Almost all other outcomes favored the telemedicine group as well. Our adjusted rates and RRs can be considered conservative because the RR values are “shrunk” toward the null and closer to a 1.0 than in frequentist analyses39 because we used a prior Bayesian probability that assumed no treatment benefit (50/50 likelihood of benefit). Our findings indicate that telemedicine can be safely used with CC and is likely to be beneficial for low income, high-risk medically complex children like ours while providing more convenient and readily accessible care and reducing their exposures in medical settings. Reducing exposures is especially important during the current COVID-19 pandemic33 or other outbreaks of contagious and seasonal illnesses. Our previous trials of CC involving both outpatient and inpatient measures to promote prompt effective care at all times demonstrated reduced adverse outcomes and decreased health system costs among medically complex children.68 The current trial indicates that adding telemedicine to augment care in the home for medically complex children is likely to further increase these benefits and could have led to savings of$7563 per child-year for a total of \$1.9 million.

The observed benefits from telemedicine might be due to multiple factors, including an enhanced ability of the PCPs to assess medically complex children in their home, a closer PCP-family relationship, increased treatment adherence, reduced delays of the family in seeking medical attention when needed, avoidance of unnecessary exposures, and a greater awareness of the benefits of seeking care from the PCPs rather than ED physicians unfamiliar with their child and more likely to hospitalize medically complex children.

Limitations of this trial include the impossibility to blind the families and PCPs to the intervention. Our trial involved a single center, few clinicians, methods of care, and a population that may differ substantially from those in other centers. The benefits and cost savings with telemedicine can be expected to be less for populations at lower risk than in this trial. Our sample size, although larger than in most telemedicine trials, was limited, and we were unable to adjust for within-family correlation for the clinical outcomes. Finally, a Hawthorne effect40  during the trial may have resulted in greater benefits than if telemedicine was part of routine clinical care for all medically complex children. However, such a Hawthorne effect may have been partially avoided by not specifically informing the families about the trial.

The addition of telemedicine to augment CC for medically complex children in their home was likely to reduce days of treatment outside the home, serious illnesses, other outcomes, and health system costs while increasing access to PCPs. Other studies of telemedicine to augment outpatient care for broader populations of medically complex children are needed.

FUNDING: Supported in part from grant 5KL2TR000370 from the Center for Clinical and Translational Sciences, grant 5 UL1TR00371 from the National Center for Advancing Translational Sciences, a grant from the Texas Medical Center Health Policy Institute, a Learning Healthcare Scholar Award from the University of Texas Health Science Center at Houston, and support from the Network Access Improvement Program. The funding sources had no role in the design, conduct, analyses, interpretation, or the decision to submit results of this study. Partially funded by the National Institutes of Health (NIH).

COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2021-051429.

Dr Mosquera takes responsibility for the integrity of the data and accuracy of the data analysis, conceptualization and design of the study, acquisition, analysis, and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, obtaining funding, administrative, technical, and material support, and study supervision; Dr Avritscher takes responsibility for the integrity of the data and accuracy of the data analysis, conceptualization and design of the study, acquisition, analysis, and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, economic analysis, obtaining funding, administrative, technical, and material support, and study supervision; Dr Pedroza takes responsibility for the conceptualization and design of the study, acquisition, analysis, and interpretation of data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content, and statistical and economic analyses; Dr Lee takes responsibility for the acquisition, analysis, and interpretation of data, critical revision of the manuscript for important intellectual content, and statistical analyses; Drs Ramanathan, Harris, Eapen, Yadav, Caldas, Poe, Harting, Ottosen, Gonzalez, and Ms Martinez Castillo take responsibility for the acquisition and interpretation of data, administrative, technical, and material support, and critical revision of the manuscript for important intellectual content; Dr Tyson takes responsibility for conceptualization and design of the study, acquisition, analysis, and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, obtaining funding, administrative, technical, and material support, and study supervision; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

This trial has been registered at ClinicalTrials.gov (identifier NCT03590509).

Deidentified individual participant data will not be made available.

Preliminary results of this trial were presented in the Pediatric Academic Society Conference, March 2020; virtual and were also presented in the AcademyHealth Annual Research Meeting, May 10, 2021; virtual.

• CC

comprehensive care

•
• CI

confidence interval

•
• CMHH

Children’s Memorial Hermann Hospital

•
• COVID-19

coronavirus disease 2019

•
• ED

emergency department

•
• EMR

electronic medical record

•
• FTE

full-time equivalent

•
• HRCC

High-Risk Children’s Clinic

•
• MHHS

Memorial Hermann Hospital System

•
• PCP

primary care provider

•
• QI

quality improvement

•
• RR

rate ratio

•
• UTH

University of Texas Health Science Center at Houston

1
Altman
L
,
Zurynski
Y
,
Breen
C
,
Hoffmann
T
,
Woolfenden
S
.
A qualitative study of health care providers’ perceptions and experiences of working together to care for children with medical complexity (CMC)
.
BMC Health Serv Res
.
2018
;
18
(
1
):
70
2
Kuo
DZ
,
Cohen
E
,
Agrawal
R
,
Berry
JG
,
Casey
PH
.
A national profile of caregiver challenges among more medically complex children with special health care needs
.
.
2011
;
165
(
11
):
1020
1026
3
Simon
TD
,
Berry
J
,
Feudtner
C
, et al
.
Children with complex chronic conditions in inpatient hospital settings in the United States
.
Pediatrics
.
2010
;
126
(
4
):
647
655
4
Berry
JG
,
Hall
M
,
Hall
DE
, et al
.
Inpatient growth and resource use in 28 children’s hospitals: a longitudinal, multi-institutional study
.
JAMA Pediatr
.
2013
;
167
(
2
):
170
177
5
Kuo
DZ
,
McAllister
JW
,
Rossignol
L
,
Turchi
RM
,
Stille
CJ
.
Care Coordination for Children With Medical Complexity: Whose Care Is It, Anyway?
Pediatrics
.
2018
;
141
(
Suppl 3
):
S224
S232
6
Mosquera
RA
,
Avritscher
EB
,
Samuels
CL
, et al
.
Effect of an enhanced medical home on serious illness and cost of care among high-risk children with chronic illness: a randomized clinical trial
.
JAMA
.
2014
;
312
(
24
):
2640
2648
7
Avritscher
EBC
,
Mosquera
RA
,
Tyson
JE
, et al
.
Post-trial sustainability and scalability of the benefits of a medical home for high-risk children with medical complexity
.
J Pediatr
.
2019
;
206
:
232
239.e3
8
Mosquera
RAAE
,
Avritscher
EBC
,
Pedroza
C
, et al
.
Hospital consultation from outpatient clinicians for medically complex children: a randomized clinical trial
.
JAMA Pediatr
.
2021
;
175
(
1
):
e205026
10.1001/jamapediatrics.2020.5026
9
Quintana
M
,
Viele
K
,
Lewis
RJ
.
Bayesian analysis: using prior information to interpret the results of clinical trials
.
JAMA
.
2017
;
318
(
16
):
1605
1606
10
Wijeysundera
DN
,
Austin
PC
,
Hux
JE
,
Beattie
WS
,
Laupacis
A
.
Bayesian statistical inference enhances the interpretation of contemporary randomized controlled trials
.
J Clin Epidemiol
.
2009
;
62
(
1
):
13
21.e5
11
Lilford
RJ
,
Thornton
JG
,
Braunholtz
D
.
Clinical trials and rare diseases: a way out of a conundrum
.
BMJ
.
1995
;
311
(
7020
):
1621
1625
12
Lewis
RJ
,
Angus
DC
.
Time for clinicians to embrace their inner Bayesian?: Reanalysis of results of a clinical trial of extracorporeal membrane oxygenation
.
JAMA
.
2018
;
320
(
21
):
2208
2210
13
Gelman
A
.
Bayesian data analysis
. Third ed.
Boca Raton, FL
:
CRC Press
;
2014
.
14
Fan
E
,
Laupacis
A
,
Pronovost
PJ
,
Guyatt
GH
,
Needham
DM
.
How to use an article about quality improvement
.
JAMA
.
2010
;
304
(
20
):
2279
2287
15
Horwitz
LI
,
Kuznetsova
M
,
Jones
SA
.
Creating a learning health system through rapid-cycle, randomized testing
.
N Engl J Med
.
2019
;
381
(
12
):
1175
1179
16
RR
,
Beauchamp
TL
,
Kass
NE
.
Informed consent, comparative effectiveness, and learning health care
.
N Engl J Med
.
2014
;
370
(
8
):
766
768
17
Kass
NE
,
RR
,
Goodman
SN
,
Pronovost
P
,
Tunis
S
,
Beauchamp
TL
.
The research-treatment distinction: a problematic approach for determining which activities should have ethical oversight
.
Hastings Cent Rep
.
2013
;
43
(
s1
):
S4
S15
18
Marcin
JP
,
Rimsza
ME
,
Moskowitz
WB
;
COMMITTEE ON PEDIATRIC WORKFORCE
.
The use of telemedicine to address access and physician workforce shortages
.
Pediatrics
.
2015
;
136
(
1
):
202
209
19
Hooshmand
M
,
Yao
K
.
Challenges facing children with special healthcare needs and their families: telemedicine as a bridge to care
.
Telemed J E Health
.
2017
;
23
(
1
):
18
24
20
Notario
PM
,
Gentile
E
,
Amidon
M
,
Angst
D
,
Lefaiver
C
,
Webster
K
.
Home-based telemedicine for children with medical complexity
.
Telemed J E Health
.
2019
;
25
(
11
):
1123
1132
21
Curran
GM
,
Bauer
M
,
Mittman
B
,
Pyne
JM
,
Stetler
C
.
Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact
.
Med Care
.
2012
;
50
(
3
):
217
226
22
Centers for Medicare and Medicaid Services
.
Physician fee schedule search
. .
23
US Department of Labor
.
Bureau of Labor Statistics consumer price indexes
.
Available at: https://www.bls.gov/cpi/data.htm. Accessed May 15, 2020
24
Laptook
AR
,
Shankaran
S
,
Tyson
JE
, et al;
Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network
.
Effect of therapeutic hypothermia initiated after 6 hours of age on death or disability among newborns with hypoxic-ischemic encephalopathy: a randomized clinical trial [published erratum appears in JAMA. 2018;319(10):1051]
.
JAMA
.
2017
;
318
(
16
):
1550
1560
25
Morris
BH
,
Oh
W
,
Tyson
JE
, et al;
NICHD Neonatal Research Network
.
Aggressive vs. conservative phototherapy for infants with extremely low birth weight
.
N Engl J Med
.
2008
;
359
(
18
):
1885
1896
26
Wasserstein
RL
,
Lazar
NA
.
The ASA statement on p-values: context, process, and purpose
.
Am Stat
.
2016
;
70
(
2
):
129
133
27
Amrhein
V
,
Greenland
S
,
McShane
B
.
Scientists rise up against statistical significance
.
Nature
.
2019
;
567
(
7748
):
305
307
28
Diamond
GA
,
Kaul
S
.
Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials
.
J Am Coll Cardiol
.
2004
;
43
(
11
):
1929
1939
29
Goligher
EC
,
Tomlinson
G
,
Hajage
D
, et al
.
Extracorporeal membrane oxygenation for severe acute respiratory distress syndrome and posterior probability of mortality benefit in a post hoc Bayesian analysis of a randomized clinical trial
.
JAMA
.
2018
;
320
(
21
):
2251
2259
30
Burkner
PC
.
Brms: an R package for Bayesian multilevel models using Stan
.
J Stat Softw
.
2017
;
80
:
1
28
31
McNutt
LA
,
Wu
C
,
Xue
X
,
Hafner
JP
.
Estimating the relative risk in cohort studies and clinical trials of common outcomes
.
Am J Epidemiol
.
2003
;
157
(
10
):
940
943
32
Pedroza
C
,
Tyson
JE
,
Das
A
,
Laptook
A
,
Bell
EF
,
Shankaran
S
;
Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network
.
Advantages of Bayesian monitoring methods in deciding whether and when to stop a clinical trial: an example of a neonatal cooling trial
.
Trials
.
2016
;
17
(
1
):
335
33
Menon
DU
,
Belcher
HME
.
COVID-19 pandemic health disparities and pediatric health care-the promise of telehealth
.
JAMA Pediatr
.
2021
;
175
(
4
):
345
346
34
Mistry
H
.
Systematic review of studies of the cost-effectiveness of telemedicine and telecare. Changes in the economic evidence over twenty years
.
J Telemed Telecare
.
2012
;
18
(
1
):
1
6
35
Flodgren
G
,
Rachas
A
,
Farmer
AJ
,
Inzitari
M
,
Shepperd
S
.
Interactive telemedicine: effects on professional practice and health care outcomes
.
Cochrane Database Syst Rev
.
2015
;
2015
(
9
):
CD002098
36
Lee
JJ
,
Chu
CT
.
Bayesian clinical trials in action
.
Stat Med
.
2012
;
31
(
25
):
2955
2972
37
Greenland
S
,
Senn
SJ
,
Rothman
KJ
, et al
.
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
.
Eur J Epidemiol
.
2016
;
31
(
4
):
337
350
38
Wasserstein
RL
,
Schirm
A
,
Lazar
NA
.
Moving to a world beyond “p <0.05”
.
Am Stat
.
2019
;
73
(
suppl 1
):
1
19
39
Spiegelhalter
DJ
,
Myles
JP
,
Jones
DR
,
Abrams
KR
.
Methods in health service research. An introduction to bayesian methods in health technology assessment
.
BMJ
.
1999
;
319
(
7208
):
508
512
40
Sedgwick
P
,
Greenwood
N
.
Understanding the Hawthorne effect
.
BMJ
.
2015
;
351
:
h4672

## Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have no conflict of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.