The ability for our health care system to adapt with extraordinary speed under crisis has never been more evident than now as we face the coronavirus disease 2019 (COVID-19) pandemic. Ambulatory clinics and primary care providers have seen shifts in patient demand for their services, and elective and scheduled care at hospitals has been reduced to mitigate the spread of infection. Almost overnight, the health care system has shifted toward providing care through telehealth platforms to avoid the catastrophic consequences of “doing business as usual,” making telehealth a leading modality of health care delivery.1 

The telehealth alternative to in-person health care expanded rapidly as new federal and state legislation passed and payers reimbursed telehealth more broadly. Jefferson Health, one of the nation’s early pioneers in telehealth, reported a 10-fold surge in demand, scheduling up to 600 telehealth visits a day. As with many hospitals across the nation, the Children’s Hospital of Philadelphia saw telehealth visits increase from 5 to 10 per day to >1500 per day. More than 800 providers at the Ann & Robert H. Lurie Children’s Hospital of Chicago were trained and privileged in a matter of weeks to expand telehealth services. Now is a critical opportunity to systematically evaluate telehealth care delivery, identify patient cohorts that can benefit, and explore ways to incorporate telehealth into patient care workflows. Using this knowledge, we will evolve our health care system to improve how care is delivered now and during crises.

Telehealth features that distinguish it from conventional face-to-face methods of care delivery include the heavy reliance on technology for encounters and the inability to “lay hands on” patients. Because telehealth usage surged during COVID-19 and patients and providers are becoming accustomed with the technology, outcome information is largely disorganized and dispersed among organizations. Although a few measurement standards exist to guide the assessment of telehealth’s impact on the care delivered, current literature lacks a unified approach to evaluate telehealth in pediatric health care delivery. The National Quality Forum (NQF) framework offers a comprehensive guide for developing telehealth measures under 4 domains (access to care, financial impact and/or cost, experience, and effectiveness) and 53 measurement concepts.2  In the World Health Organization framework, similar measurement concepts are applied in an approach that considers the maturity stage of a telehealth program.3  For example, the focus of a pilot telehealth program may be feasibility, whereas the focus of a more mature program may be scalability and impact to the health system. In a technical brief from the Agency for Health Research and Quality (AHRQ), researchers describe available research on patient outcomes and telehealth use and therefore emphasize the criticality of including the impact on health outcomes in any telehealth evaluation.4 

Invaluable work done by the above organizations inspired Supporting Pediatric Research on Outcomes and Utilization of Telehealth (SPROUT), a part of the Section on Telehealth Care of the American Academy of Pediatrics, to synergize the above work into a single framework that researchers can use to study telehealth’s impact on patients. In the SPROUT Telehealth Evaluation and Measurement (STEM) profile (Table 1),5  the NQF, World Health Organization, and AHRQ concepts are reorganized into 4 measurement domains: (1) health outcomes; (2) health delivery (quality and cost); (3) experience, and (4) program implementation and key performance indicators (KPIs). The STEM profile is meant to communicate telehealth’s value to 4 key stakeholder groups: patients, providers, health systems, and payers.

TABLE 1

Four Measurement Domains of STEM

DomainExample of Potential MeasuresDetails and Specifications for Examples
Health outcomes (measuring the individual or population level related to physiology, mental health, and quality of life; these measures may come from diagnostic tests and encounter records or from patient-reported outcomes) Mortality No. deaths from COVID-19 divided by the No. patients with positive test results for COVID-19 
BMI and percentile Patient’s BMI over time, >95% (obesity) 
Type 1 diabetes blood glucose management: monitoring and treatment (HbA1c target [<7.5%], with individualized considerations) Percentage of children and adolescents with type 1 diabetes whose most recent HbA1c level is <7.5% (American Diabetes Association guidelines) 
Health delivery (quality and cost) (measuring timely access, effectiveness of care, costs, safety, and equity) Access: children and young adults aged 12 mo to 19 y who had a visit with a primary care practitioner An HEDIS measure: https://www.ncqa.org/hedis/measures/children-and-adolescents-access-to-primary-care-practitioners-cap/ 
Access: follow-up care for children aged 6–12 y who were prescribed an ADHD medication An HEDIS measure: https://www.ncqa.org/hedis/measures/follow-up-care-for-children-prescribed-adhd-medication/ 
Effectiveness: preventive care guidelines Compliance with preventive care guidelines (ie, counseling on nutrition and physical activity for children and adolescents done through either an in-person or a telemedicine visit) 
Effectiveness: vaccination rate Vaccination coverage in specific pediatric age cohorts 
Safety: antibiotic prescription rate Rate of children aged 3 mo to 18 y who were given a diagnosis of upper respiratory infection and were not dispensed an antibiotic prescription (HEDIS): https://www.ncqa.org/hedis/measures/appropriate-treatment-for-children-with-upper-respiratory-infection/ 
Safety of telehealth visits: percentage of unanticipated ED visits No. completed telehealth visits with unanticipated, subsequent ED visits divided by the No. completed telehealth visits 
Patient cost: average miles saved from not traveling to the provider’s office Miles from home to clinic or hospital setting: tracking distance allows for calculations of time saved and dollars saved 
Equity: percentage of clinical encounters delivered via telehealth in communities with low and high incomes No. telehealth visits divided by the No. clinic encounters (telehealth and in-person visits) among patients from different income levels 
Individual experience (measuring the quality and characteristics of the telehealth encounter itself and its impact on the individual patient or provider in terms of workload burden, satisfaction, and issues experienced with technology and logistics) Net Promotor Score A common customer-experience metric used to measure customer loyalty to a company that provides a specific service for that customer, often an estimator of customer satisfaction 
Patient centeredness: responses to survey questions about patient centeredness How much did the provider include you in the decision-making? How much did you feel that the provider heard all your concerns? 
Staff wellness and burnout Examples tools: Maslach Burnout Inventory, Oldenburg Burnout Inventory, Physician Work Life Study single-item measure, Stanford Professional Fulfillment Index, and Well-Being Index 
Patient burden reduction: saved workdays or reduced No. missed school days No. lost workdays or missed school days due to health appointments 
Usability Did you have difficulties getting to your appointment (either traveling to the clinic or using the telehealth system)? Did you have any technical difficulties? 
Program implementation and KPIs (measuring program performance, implementation process, and benchmarks with peer programs; measuring how the system adapted human resources, processes, and tools) System changes No. and type of changes to the system to accommodate the program (eg, hiring remote diabetes educators, purchasing devices, investing in technology support services, and creating telehealth platforms) 
KPIs No. telehealth visits completed within timeliness of care targets set by the program 
DomainExample of Potential MeasuresDetails and Specifications for Examples
Health outcomes (measuring the individual or population level related to physiology, mental health, and quality of life; these measures may come from diagnostic tests and encounter records or from patient-reported outcomes) Mortality No. deaths from COVID-19 divided by the No. patients with positive test results for COVID-19 
BMI and percentile Patient’s BMI over time, >95% (obesity) 
Type 1 diabetes blood glucose management: monitoring and treatment (HbA1c target [<7.5%], with individualized considerations) Percentage of children and adolescents with type 1 diabetes whose most recent HbA1c level is <7.5% (American Diabetes Association guidelines) 
Health delivery (quality and cost) (measuring timely access, effectiveness of care, costs, safety, and equity) Access: children and young adults aged 12 mo to 19 y who had a visit with a primary care practitioner An HEDIS measure: https://www.ncqa.org/hedis/measures/children-and-adolescents-access-to-primary-care-practitioners-cap/ 
Access: follow-up care for children aged 6–12 y who were prescribed an ADHD medication An HEDIS measure: https://www.ncqa.org/hedis/measures/follow-up-care-for-children-prescribed-adhd-medication/ 
Effectiveness: preventive care guidelines Compliance with preventive care guidelines (ie, counseling on nutrition and physical activity for children and adolescents done through either an in-person or a telemedicine visit) 
Effectiveness: vaccination rate Vaccination coverage in specific pediatric age cohorts 
Safety: antibiotic prescription rate Rate of children aged 3 mo to 18 y who were given a diagnosis of upper respiratory infection and were not dispensed an antibiotic prescription (HEDIS): https://www.ncqa.org/hedis/measures/appropriate-treatment-for-children-with-upper-respiratory-infection/ 
Safety of telehealth visits: percentage of unanticipated ED visits No. completed telehealth visits with unanticipated, subsequent ED visits divided by the No. completed telehealth visits 
Patient cost: average miles saved from not traveling to the provider’s office Miles from home to clinic or hospital setting: tracking distance allows for calculations of time saved and dollars saved 
Equity: percentage of clinical encounters delivered via telehealth in communities with low and high incomes No. telehealth visits divided by the No. clinic encounters (telehealth and in-person visits) among patients from different income levels 
Individual experience (measuring the quality and characteristics of the telehealth encounter itself and its impact on the individual patient or provider in terms of workload burden, satisfaction, and issues experienced with technology and logistics) Net Promotor Score A common customer-experience metric used to measure customer loyalty to a company that provides a specific service for that customer, often an estimator of customer satisfaction 
Patient centeredness: responses to survey questions about patient centeredness How much did the provider include you in the decision-making? How much did you feel that the provider heard all your concerns? 
Staff wellness and burnout Examples tools: Maslach Burnout Inventory, Oldenburg Burnout Inventory, Physician Work Life Study single-item measure, Stanford Professional Fulfillment Index, and Well-Being Index 
Patient burden reduction: saved workdays or reduced No. missed school days No. lost workdays or missed school days due to health appointments 
Usability Did you have difficulties getting to your appointment (either traveling to the clinic or using the telehealth system)? Did you have any technical difficulties? 
Program implementation and KPIs (measuring program performance, implementation process, and benchmarks with peer programs; measuring how the system adapted human resources, processes, and tools) System changes No. and type of changes to the system to accommodate the program (eg, hiring remote diabetes educators, purchasing devices, investing in technology support services, and creating telehealth platforms) 
KPIs No. telehealth visits completed within timeliness of care targets set by the program 

ADHD, attention-deficit/hyperactivity disorder; ED, emergency department; HEDIS, Healthcare Effectiveness Data and Information Set.

Identifying health outcome targets early is equally important as identifying quality outcomes when implementing telehealth services. Health measures are available through various federal agencies, national committees, and academic societies, such as the NQF, which summarizes the work from >40 measure steward organizations, such as the AHRQ and the Centers for Medicare and Medicaid Services.6  Telehealth’s rapid adaptation creates an opportunity to compare outcome measures after telehealth and in-person visits among patients with chronic conditions, such as hemoglobin A1c (HbA1c) in children with type 1 diabetes, BMI percentile to monitor obesity, or psychometric questionnaires to assess anxiety disorder symptoms in children. Publicly available data on COVID-19 disease burden and policy changes in specific geographies7  can be used to compare endemic areas with and without telehealth.

Telehealth has the potential to impact the quality and cost of health care delivery, including areas of access, effectiveness, cost, safety, and equity. With “stay-at-home” mandates in many communities, telehealth can help increase access and minimize the need for in-person appointments. Such impact can be measured at the pandemic’s start in terms of the proportion of in-person appointments that were successfully converted to telehealth encounters. Conversely, access may be impacted negatively in populations without access to Internet services, computers, or smartphone technology necessary for telehealth visits. Effectiveness is known typically as the extent evidence-based care is delivered reliably and consistently to patients; this care is often described by clinical recommendations. Although the convenience of telehealth may afford providers more opportunities to improve patient compliance, providers must continue to follow recommended practices and not skip safeguards, as reported recently on the tendency to overprescribe antibiotics in treating upper respiratory infections when patients are seen via telemedicine.

Travel time and miles avoided by using telehealth can be translated to cost savings by using standard conversion formulas (eg, $0.57/mile as business travel for providers or $0.17/mile as medical purpose travel for patients). However, the recent telehealth surge has exposed an equity gap in families who cannot access telehealth because of a lack of technology resources.

The provider and patient experience with a telehealth encounter and the logistic impact on their daily lives are important considerations. The Telehealth Usability Questionnaire, Telemedicine Satisfaction and Usefulness Questionnaire, Patient Assessment of Communication during Telemedicine, and the Net Promotor Score are examples of assessment tools that can be used to assess satisfaction with the provider-patient communication, technology, and usefulness. The impact on provider’s workload and disruption of family routines (ie, work and school) are personal burdens that may affect appointment adherence and overall satisfaction. Other experiential factors like encounter duration, video and audio quality, and connectivity should also be tracked.

In a time of rapid health care system transformations, it is critical to monitor both the creation of new telehealth programs and how existing programs change. Future economic analyses will be possible if programs gather data on the costs of telehealth service expansion and delivery, such as time and resources spent to credential and prepare members of the care team, establish new billing procedures, and provide telehealth tools. As implementation science frameworks guide program execution, programs should track KPIs to self-assess and benchmark with peer institutions.

Telehealth offers valuable opportunities to improve the process of health care delivery that may translate to better health outcomes. Identifying and studying these opportunities require a measurement strategy that allows data to be aggregated across multiple health systems, making possible the study of rare conditions and comparisons of different locations and methods for delivering services via telehealth. The STEM profile offers a construct to define and organize telehealth measures in terms of health outcome, health delivery quality and cost, and individual experience, as well as emphasizes program implementation and benchmarks. Findings from rigorous telehealth program evaluation in these areas can be used to inform data-driven reimbursement and policy changes that encourage appropriate telehealth use, especially amid the explosion of telehealth services associated with the COVID-19 pandemic.

We thank the members of the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group for their efforts, particularly David McSwain, Dana Schinasi, and Kari Gali, for their work in applying the STEM framework for COVID-19.

Dr Chuo drafted the initial manuscript and table, reviewed and revised the manuscript, and led the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group in the development of the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Evaluation and Measurement (STEM) framework; Drs Macy reviewed and revised the manuscript and contributed to the Supporting Pediatric Research on Outcomes and Utilization of Telehealth Metrics Topic Working Group in the application of the STEM framework to coronavirus disease 2019; Dr Lorch reviewed and revised the manuscript and advised on the STEM framework and its application to coronavirus disease 2019; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Supporting Pediatric Research on Outcomes and Utilization of Telehealth–Clinical Translational Science Awards Collaborative Telehealth Research Network and funded in part by the National Institutes of Health National Center for Advancing Translational Science grant U01TR002626. Funded by the National Institutes of Health (NIH).

     
  • AHRQ

    Agency for Health Research and Quality

  •  
  • COVID-19

    coronavirus disease 2019

  •  
  • HbA1c

    hemoglobin A1c

  •  
  • KPI

    key performance indicator

  •  
  • NQF

    National Quality Forum

  •  
  • STEM

    Supporting Pediatric Research on Outcomes and Utilization of Telehealth Evaluation and Measurement

1
Hollander
JE
,
Carr
BG
.
Virtually perfect? Telemedicine for Covid-19
.
N Engl J Med
.
2020
;
382
(
18
):
1679
1681
2.
National Quality Forum
.
Creating a framework to support measure development for telehealth.
2017
. Available at: https://www.qualityforum.org/Publications/2017/08/Creating_a_Framework_to_Support_Measure_Development_for_Telehealth.aspx. Accessed March 1, 2020
3
World Health Organization
.
Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment
.
Geneva, Switzerland
:
World Health Organization
;
2016
4
Totten
AM
,
Womack
DM
,
Eden
KB
, et al
.
Telehealth: Mapping the Evidence for Patient Outcomes From Systematic Reviews. Technical Brief No. 26. AHRQ Publication No. 16-EHC034-EF.
Rockville, MD
:
Agency for Healthcare Research and Quality
;
2016
5.
Medical University of South Carolina
.
SPROUT COVID-19 measure framework.
2020
. Available at: https://research.musc.edu/-/sm/research/resources/sctr/f/sprout-telehealth-measurement-framework-for-covid-19.ashx?la=en. Accessed March 1, 2020
6.
National Quality Forum
.
Measures, reports, and tools.
2020
. Available at: www.qualityforum.org/Measures_Reports_Tools.aspx. Accessed March 1, 2020
7
Mathematica
.
COVID-19 curated data, modeling, and policy resources
.
2020
. Available at: https://www.mathematica.org/features/covid-19-curated-data-modeling-and-policy-resources. Accessed March 30, 2020

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.