The ability for our health care system to adapt with extraordinary speed under crisis has never been more evident than now as we face the coronavirus disease 2019 (COVID-19) pandemic. Ambulatory clinics and primary care providers have seen shifts in patient demand for their services, and elective and scheduled care at hospitals has been reduced to mitigate the spread of infection. Almost overnight, the health care system has shifted toward providing care through telehealth platforms to avoid the catastrophic consequences of “doing business as usual,” making telehealth a leading modality of health care delivery.1 

The telehealth alternative to in-person health care expanded rapidly as new federal and state legislation passed and payers reimbursed telehealth more broadly. Jefferson Health, one of the nation’s early pioneers in telehealth, reported a 10-fold surge in demand, scheduling up to 600 telehealth visits a day. As with many hospitals across the nation, the Children’s Hospital of Philadelphia saw telehealth visits increase from 5 to 10 per day to >1500 per day. More than 800 providers at the Ann & Robert H. Lurie Children’s Hospital of Chicago were trained and privileged in a matter of weeks to expand telehealth services. Now is a critical opportunity to systematically evaluate telehealth care delivery, identify patient cohorts that can benefit, and explore ways to incorporate telehealth into patient care workflows. Using this knowledge, we will evolve our health care system to improve how care is delivered now and during crises.

Telehealth features that distinguish it from conventional face-to-face methods of care delivery include the heavy reliance on technology for encounters and the inability to “lay hands on” patients. Because telehealth usage surged during COVID-19 and patients and providers are becoming accustomed with the technology, outcome information is largely disorganized and dispersed among organizations. Although a few measurement standards exist to guide the assessment of telehealth’s impact on the care delivered, current literature lacks a unified approach to evaluate telehealth in pediatric health care delivery. The National Quality Forum (NQF) framework offers a comprehensive guide for developing telehealth measures under 4 domains (access to care, financial impact and/or cost, experience, and effectiveness) and 53 measurement concepts.2  In the World Health Organization framework, similar measurement concepts are applied in an approach that considers the maturity stage of a telehealth program.3  For example, the focus of a pilot telehealth program may be feasibility, whereas the focus of a more mature program may be scalability and impact to the health system. In a technical brief from the Agency for Health Research and Quality (AHRQ), researchers describe available research on patient outcomes and telehealth use and therefore emphasize the criticality of including the impact on health outcomes in any telehealth evaluation.4 

Invaluable work done by the above organizations inspired Supporting Pediatric Research on Outcomes and Utilization of Telehealth (SPROUT), a part of the Section on Telehealth Care of the American Academy of Pediatrics, to synergize the above work into a single framework that researchers can use to study telehealth’s impact on patients. In the SPROUT Telehealth Evaluation and Measurement (STEM) profile (Table 1),5  the NQF, World Health Organization, and AHRQ concepts are reorganized into 4 measurement domains: (1) health outcomes; (2) health delivery (quality and cost); (3) experience, and (4) program implementation and key performance indicators (KPIs). The STEM profile is meant to communicate telehealth’s value to 4 key stakeholder groups: patients, providers, health systems, and payers.

Identifying health outcome targets early is equally important as identifying quality outcomes when implementing telehealth services. Health measures are available through various federal agencies, national committees, and academic societies, such as the NQF, which summarizes the work from >40 measure steward organizations, such as the AHRQ and the Centers for Medicare and Medicaid Services.6  Telehealth’s rapid adaptation creates an opportunity to compare outcome measures after telehealth and in-person visits among patients with chronic conditions, such as hemoglobin A1c (HbA1c) in children with type 1 diabetes, BMI percentile to monitor obesity, or psychometric questionnaires to assess anxiety disorder symptoms in children. Publicly available data on COVID-19 disease burden and policy changes in specific geographies7  can be used to compare endemic areas with and without telehealth.

Telehealth has the potential to impact the quality and cost of health care delivery, including areas of access, effectiveness, cost, safety, and equity. With “stay-at-home” mandates in many communities, telehealth can help increase access and minimize the need for in-person appointments. Such impact can be measured at the pandemic’s start in terms of the proportion of in-person appointments that were successfully converted to telehealth encounters. Conversely, access may be impacted negatively in populations without access to Internet services, computers, or smartphone technology necessary for telehealth visits. Effectiveness is known typically as the extent evidence-based care is delivered reliably and consistently to patients; this care is often described by clinical recommendations. Although the convenience of telehealth may afford providers more opportunities to improve patient compliance, providers must continue to follow recommended practices and not skip safeguards, as reported recently on the tendency to overprescribe antibiotics in treating upper respiratory infections when patients are seen via telemedicine.

Travel time and miles avoided by using telehealth can be translated to cost savings by using standard conversion formulas (eg, $0.57/mile as business travel for providers or $0.17/mile as medical purpose travel for patients). However, the recent telehealth surge has exposed an equity gap in families who cannot access telehealth because of a lack of technology resources.

The provider and patient experience with a telehealth encounter and the logistic impact on their daily lives are important considerations. The Telehealth Usability Questionnaire, Telemedicine Satisfaction and Usefulness Questionnaire, Patient Assessment of Communication during Telemedicine, and the Net Promotor Score are examples of assessment tools that can be used to assess satisfaction with the provider-patient communication, technology, and usefulness. The impact on provider’s workload and disruption of family routines (ie, work and school) are personal burdens that may affect appointment adherence and overall satisfaction. Other experiential factors like encounter duration, video and audio quality, and connectivity should also be tracked.

In a time of rapid health care system transformations, it is critical to monitor both the creation of new telehealth programs and how existing programs change. Future economic analyses will be possible if programs gather data on the costs of telehealth service expansion and delivery, such as time and resources spent to credential and prepare members of the care team, establish new billing procedures, and provide telehealth tools. As implementation science frameworks guide program execution, programs should track KPIs to self-assess and benchmark with peer institutions.

Telehealth offers valuable opportunities to improve the process of health care delivery that may translate to better health outcomes. Identifying and studying these opportunities require a measurement strategy that allows data to be aggregated across multiple health systems, making possible the study of rare conditions and comparisons of different locations and methods for delivering services via telehealth. The STEM profile offers a construct to define and organize telehealth measures in terms of health outcome, health delivery quality and cost, and individual experience, as well as emphasizes program implementation and benchmarks. Findings from rigorous telehealth program evaluation in these areas can be used to inform data-driven reimbursement and policy changes that encourage appropriate telehealth use, especially amid the explosion of telehealth services associated with the COVID-19 pandemic.

We thank the members of the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group for their efforts, particularly David McSwain, Dana Schinasi, and Kari Gali, for their work in applying the STEM framework for COVID-19.

Dr Chuo drafted the initial manuscript and table, reviewed and revised the manuscript, and led the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group in the development of the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Evaluation and Measurement (STEM) framework; Drs Macy reviewed and revised the manuscript and contributed to the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group in the application of the STEM framework to coronavirus disease 2019; Dr Lorch reviewed and revised the manuscript and advised on the STEM framework and its application to coronavirus disease 2019; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Supporting Pediatric Research on Outcomes and Utilization of Telehealth–Clinical Translational Science Awards Collaborative Telehealth Research Network and funded in part by the National Institutes of Health National Center for Advancing Translational Science grant U01TR002626. Funded by the National Institutes of Health (NIH).

AHRQ

Agency for Health Research and Quality

COVID-19

coronavirus disease 2019

HbA1c

hemoglobin A1c

KPI

key performance indicator

NQF

National Quality Forum

STEM

Supporting Pediatric Research on Outcomes and Utilization of Telehealth Evaluation and Measurement

1
Hollander
JE
,
Carr
BG
.
Virtually perfect? Telemedicine for Covid-19
.
N Engl J Med
.
2020
;
382
(
18
):
1679
1681
2.
National Quality Forum
.
Creating a framework to support measure development for telehealth.
2017
. Available at: https://www.qualityforum.org/Publications/2017/08/Creating_a_Framework_to_Support_Measure_Development_for_Telehealth.aspx. Accessed March 1, 2020
3
World Health Organization
.
Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment
.
Geneva, Switzerland
:
World Health Organization
;
2016
4
Totten
AM
,
Womack
DM
,
Eden
KB
, et al
.
Telehealth: Mapping the Evidence for Patient Outcomes From Systematic Reviews. Technical Brief No. 26. AHRQ Publication No. 16-EHC034-EF.
Rockville, MD
:
Agency for Healthcare Research and Quality
;
2016
5.
Medical University of South Carolina
.
SPROUT COVID-19 measure framework.
2020
. Available at: https://research.musc.edu/-/sm/research/resources/sctr/f/sprout-telehealth-measurement-framework-for-covid-19.ashx?la=en. Accessed March 1, 2020
6.
National Quality Forum
.
Measures, reports, and tools.
2020
. Available at: www.qualityforum.org/Measures_Reports_Tools.aspx. Accessed March 1, 2020
7
Mathematica
.
COVID-19 curated data, modeling, and policy resources
.
2020
. Available at: https://www.mathematica.org/features/covid-19-curated-data-modeling-and-policy-resources. Accessed March 30, 2020

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.