OBJECTIVES:

Noninvasive positive pressure ventilation (NIPPV) is increasingly used in critically ill pediatric patients, despite limited data on safety and efficacy. Administrative data may be a good resource for observational studies. Therefore, we sought to assess the performance of the International Classification of Diseases, Ninth Revision procedure code for NIPPV.

METHODS:

Patients admitted to the PICU requiring NIPPV or heated high-flow nasal cannula (HHFNC) over the 11-month study period were identified from the Virtual PICU System database. The gold standard was manual review of the electronic health record to verify the use of NIPPV or HHFNC among the cohort. The presence or absence of a NIPPV procedure code was determined by using administrative data. Test characteristics with 95% confidence intervals (CIs) were generated, comparing administrative data with the gold standard.

RESULTS:

Among the cohort (n = 562), the majority were younger than 5 years, and the most common primary diagnosis was bronchiolitis. Most (82%) required NIPPV, whereas 18% required only HHFNC. The NIPPV code had a sensitivity of 91.1% (95% CI: 88.2%–93.6%) and a specificity of 57.6% (95% CI: 47.2%–67.5%), with a positive likelihood ratio of 2.15 (95% CI: 1.70–2.71) and negative likelihood ratio of 0.15 (95% CI: 0.11–0.22).

CONCLUSIONS:

Among our critically ill pediatric cohort, NIPPV procedure codes had high sensitivity but only moderate specificity. On the basis of our study results, there is a risk of misclassification, specifically failure to identify children who require NIPPV, when using administrative data to study the use of NIPPV in this population.

Children hospitalized with respiratory illness may require admission to the PICU for escalation in respiratory support, including noninvasive positive pressure ventilation (NIPPV). NIPPV allows for the provision of respiratory support without use of an endotracheal tube and has been applied in numerous chronic and acute respiratory disorders over the past few decades.1  In addition, NIPPV has been shown to improve survival and reduce complications in certain adult patient populations with acute respiratory failure.2  However, pediatric data on the safety and efficacy of NIPPV remains lacking despite its extensive use and a growing number of yearly publications on the topic.3  Given the challenges of pediatric clinical trials, administrative data have been used for conducting observational studies on the use of NIPPV.4,5 

Currently, there is limited information about the performance of the administrative procedure code for NIPPV for a critically ill pediatric population.

Furthermore, the International Classification of Diseases, Ninth Revision (ICD-9) for noninvasive ventilation specifies the use of bilevel positive airway pressure (BPAP) or continuous positive airway pressure (CPAP). However, the use of heated high-flow nasal cannula (HHFNC) has been included in the definition of NIPPV by some authors in previous pediatric studies.6 

The purpose of our study was to evaluate the accuracy of the ICD-9 procedure code for NIPPV by testing its ability to discriminate between NIPPV and HHFNC within a local cohort of children admitted to the PICU.

The study protocol was approved by the institutional review board, with a waiver of consent. Data were obtained from 3 sources: Virtual PICU Systems LLC (VPS) database, the Children’s Hospital Colorado electronic health records (EHRs), and the Pediatric Health Information System. The study cohort was composed of subjects admitted to the PICU from August 2014 through June 2015, older than 30 days, and who required NIPPV or HHFNC for more than 24 hours during their PICU admission. NIPPV use could be newly initiated or a baseline requirement. The full details of cohort development have been previously published.7  Briefly, the cohort was identified from VPS data as requiring at least 1 of 3 levels of respiratory support: CPAP, BPAP, or HHFNC. Subjects who did not require any respiratory support during the PICU admission were excluded. The original VPS data were manually abstracted directly from the EHR, not from administrative data, by trained PICU research nurses who undergo regular interrater reliability testing to ensure ongoing accurate data entry. Any unusual patterns are flagged by VPS, and site coordinators are notified to perform revalidation of flagged data. The type, timing, and duration of CPAP, BPAP, or HHFNC was then reverified by 1 of the physician team members through review of the EHR and categorized into either NIPPV or HHFNC support only. This reverified data on the required level of respiratory support served as the final gold standard for comparison with administrative coding.

The presence or absence of the ICD-9 procedure code for noninvasive mechanical ventilation (93.9×) for the cohort was identified from the Pediatric Health Information System database by using the medical record number and date of hospital admission. Subjects who had a procedure code for NIPPV but only required HHFNC were considered false-positives, whereas those who did not have a procedure code for NIPPV but required CPAP or BPAP were considered false-negatives. The test characteristics generated included sensitivity, specificity, positive and negative likelihood ratios (LRs) with 95% confidence intervals (95% CIs). As an exploratory analysis, we also performed separate analyses by age group (age <1 year and age ≥1 year). Descriptive statistics were used to characterize the study cohort. All analyses were performed by using Stata SE, version 14.0 (Stata Corp, College Station, TX).

During the study period, 562 patients met the inclusion criteria. Cohort characteristics are summarized in Table 1. The majority of patients were <5 years of age, and the most common primary diagnosis was bronchiolitis. Most patients required CPAP or BPAP during their ICU admission. Ninety-nine patients (18%) required HHFNC only. The median length of time on NIPPV was 2.0 days (interquartile range [IQR] 2.0–4.0 days), and the median ICU length of stay was 2.8 days (IQR: 2.0–4.3 days).

TABLE 1

Characteristics of the Cohort

Characteristic
Age category, y, n (%)  
 <1 131 (23) 
 1–5 260 (46) 
 5–10 72 (13) 
 >10 99 (18) 
Male sex, n (%) 312 (56) 
Primary diagnosis, n (%)  
 Bronchiolitis 269 (48) 
 Viral pneumonia 92 (16) 
 Status asthmaticus 103 (18) 
 Bacterial pneumonia 37 (7) 
 Septic shock 9 (2) 
 Other 52 (9) 
Maximum respiratory support required during ICU stay, n (%)  
 HHFNC only 99 (18) 
 CPAP or BPAP 463 (82) 
Length of time on NIPPV, d, mean (IQR) 2.0 (2.0–4.0) 
ICU length of stay, d, mean (IQR) 2.8 (2.0–4.3) 
Characteristic
Age category, y, n (%)  
 <1 131 (23) 
 1–5 260 (46) 
 5–10 72 (13) 
 >10 99 (18) 
Male sex, n (%) 312 (56) 
Primary diagnosis, n (%)  
 Bronchiolitis 269 (48) 
 Viral pneumonia 92 (16) 
 Status asthmaticus 103 (18) 
 Bacterial pneumonia 37 (7) 
 Septic shock 9 (2) 
 Other 52 (9) 
Maximum respiratory support required during ICU stay, n (%)  
 HHFNC only 99 (18) 
 CPAP or BPAP 463 (82) 
Length of time on NIPPV, d, mean (IQR) 2.0 (2.0–4.0) 
ICU length of stay, d, mean (IQR) 2.8 (2.0–4.3) 

Administrative codes for NIPPV had a sensitivity of 91.1% (95% CI: 88.2%–93.6%) and a specificity of 57.6% (95% CI: 47.2%–67.5%) for identifying patients requiring CPAP or BPAP, compared with the EHR-based gold standard (Table 2). The positive LR was 2.15 (95% CI: 1.70–2.71), and the negative LR was 0.15 (95% CI: 0.11–0.22). The test characteristics did not substantially differ by age group. For infants, sensitivity was 92.6% (95% CI: 85.3%–97%), specificity was 59.5% (95% CI: 42.1%–75.2%), positive LR was 2.28 (95% CI: 1.54–3.39), and negative LR was 0.13 (95% CI: 0.06–0.27). For noninfants, sensitivity was 90.8% (95% CI: 87.4%–93.5%), specificity was 56.5% (95% CI: 43.3%–69%), positive LR was 2.08 (95% CI: 1.57–2.77), and negative LR was 0.16 (95% CI: 0.11–0.24).

TABLE 2

Accuracy of Administrative Codes for NIPPV

EHR-Confirmed NIPPV UseTotal
NIPPV UseHHFNC Only
NIPPV code in administrative data 
 Present 422 42 464 
 Absent 41 57 98 
Total 463 99 — 
EHR-Confirmed NIPPV UseTotal
NIPPV UseHHFNC Only
NIPPV code in administrative data 
 Present 422 42 464 
 Absent 41 57 98 
Total 463 99 — 

Sensitivity was 91.1% (95% CI: 88.2%–93.6%), specificity was 57.6% (95% CI: 47.2%–67.5%), the positive LR was 2.15 (95% CI: 1.70–2.71), and the negative LR was 0.15 (95% CI: 0.11–0.22). —, not applicable.

Using a local cohort of PICU patients requiring NIPPV or HHFNC, we found that the ICD-9 code for noninvasive mechanical ventilation had a high sensitivity but only moderate specificity. To the best of our knowledge, this represents the first attempt to assess the performance of the administrative code for NIPPV in the critically ill pediatric population.

Conducting prospective clinical studies in the PICU is challenging because of smaller sample size and often financial and ethical limitations. Administrative databases serve as an attractive alternative data source because of their large sample sizes, lower costs, and often broad coverage of populations of interest. However, because these data are not collected for research purposes, they can have variable accuracy.8  Therefore, validation of administrative codes is an important step in rigorous observational research, yet is often not performed.9,10  Administrative data have been used to characterize NIPPV use and outcomes in adult and pediatric patients.4,5,1114  Although the accuracy of administrative codes for NIPPV has not been reported in children, the reported performance of the ICD-9 code for NIPPV in adults demonstrated high sensitivity and specificity, 86% and 92%, respectively.14  The lower specificity in our study may be due to different underlying populations and/or coding practices for pediatric patients.

For this study, HHFNC was used as the negative control to capture a population at high risk of NIPPV support but did not receive it. However, use of HHFNC in children continues to increase both within and outside of the ICU and has been shown to deliver variable amounts of positive airway pressure.1519  As a result, some authors combine both NIPPV and HHFNC when studying noninvasive respiratory support.20,21  The overlap in definitions, physiologic effects, and patient populations who require escalated respiratory support in the form of HHFNC and NIPPV makes discriminating between the modalities for research purposes increasingly difficult.

There are several limitations to our study that should be acknowledged. Local coding practices may differ among institutions, yielding higher or lower test characteristics. The cohort used as a gold standard only included nonneonatal, critically ill children who required, at minimum, HHFNC support, with 82% of the population requiring NIPPV during admission. With a more general population, including those who did not require either forms of respiratory support, we might anticipate lower sensitivity and higher specificity. However, we believe the cohort represents a population of children who would be of particular interest to the critical care research community. Additionally, we excluded children with <24 hours of NIPPV support, given the primary aims of the original study.7  Although direction of potential bias is uncertain, we would suspect greater misclassification due to administrative charges generally created on a daily basis (ie, children with shorter NIPPV duration on a single day may not receive a charge for NIPPV). Finally, we did not examine the performance of codes based on the International Classification of Diseases, 10th Revision given the time frame of our cohort. With more specific coding of CPAP versus BPAP, the performance of the administrative codes may be improved.

On the basis of our study results, there is a risk of misclassification when using administrative data to study the use of NIPPV in this population. Authors of future studies should either perform separate validation studies or use this information to perform sensitivity analyses to assess the impact on their results.

Dr Good conceptualized and designed the study and drafted the initial manuscript; Dr Leroue conceptualized and designed the study; Dr Czaja conceptualized and designed the study and conducted the initial analyses; and all authors reviewed and revised the manuscript and approved the final manuscript as submitted.

FUNDING: No external funding.

1
Hull
J
.
The value of non-invasive ventilation
.
Arch Dis Child
.
2014
;
99
(
11
):
1050
1054
2
Nava
S
,
Hill
N
.
Non-invasive ventilation in acute respiratory failure
.
Lancet
.
2009
;
374
(
9685
):
250
259
3
Najaf-Zadeh
A
,
Leclerc
F
.
Noninvasive positive pressure ventilation for acute respiratory failure in children: a concise review
.
Ann Intensive Care
.
2011
;
1
(
1
):
15
4
Kaiser
SV
,
Bakel
LA
,
Okumura
MJ
,
Auerbach
AD
,
Rosenthal
J
,
Cabana
MD
.
Risk factors for prolonged length of stay or complications during pediatric respiratory hospitalizations
.
Hosp Pediatr
.
2015
;
5
(
9
):
461
473
5
Keele
L
,
Keenan
HT
,
Sheetz
J
,
Bratton
SL
.
Differences in characteristics of dying children who receive and do not receive palliative care
.
Pediatrics
.
2013
;
132
(
1
):
72
78
6
Canarie
MF
,
Barry
S
,
Carroll
CL
, et al
;
Northeast Pediatric Critical Care Research Consortium
.
Risk factors for delayed enteral nutrition in critically ill children
.
Pediatr Crit Care Med
.
2015
;
16
(
8
):
e283
e289
7
Leroue
MK
,
Good
RJ
,
Skillman
HE
,
Czaja
AS
.
Enteral nutrition practices in critically ill children requiring noninvasive positive pressure ventilation
.
Pediatr Crit Care Med
.
2017
;
18
(
12
):
1093
1098
8
Garland
A
,
Gershengorn
HB
,
Marrie
RA
,
Reider
N
,
Wilcox
MEA
.
A practical, global perspective on using administrative data to conduct intensive care unit research
.
Ann Am Thorac Soc
.
2015
;
12
(
9
):
1373
1386
9
van Walraven
C
,
Austin
P
.
Administrative database research has unique characteristics that can risk biased results
.
J Clin Epidemiol
.
2012
;
65
(
2
):
126
131
10
van Walraven
C
,
Bennett
C
,
Forster
AJ
.
Administrative database research infrequently used validated diagnostic or procedural codes
.
J Clin Epidemiol
.
2011
;
64
(
10
):
1054
1059
11
Stefan
MS
,
Shieh
MS
,
Pekow
PS
, et al
.
Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey
.
J Hosp Med
.
2013
;
8
(
2
):
76
82
12
Vasquez
MM
,
McClure
LA
,
Sherrill
DL
, et al
.
Positive airway pressure therapies and hospitalization in chronic obstructive pulmonary disease
.
Am J Med
.
2017
;
130
(
7
):
809
818
13
Walkey
AJ
,
Wiener
RS
.
Use of noninvasive ventilation in patients with acute respiratory failure, 2000-2009: a population-based study
.
Ann Am Thorac Soc
.
2013
;
10
(
1
):
10
17
14
Lindenauer
PK
,
Stefan
MS
,
Shieh
MS
,
Pekow
PS
,
Rothberg
MB
,
Hill
NS
.
Outcomes associated with invasive and noninvasive ventilation among patients hospitalized with exacerbations of chronic obstructive pulmonary disease
.
JAMA Intern Med
.
2014
;
174
(
12
):
1982
1993
15
Kubicka
ZJ
,
Limauro
J
,
Darnall
RA
.
Heated, humidified high-flow nasal cannula therapy: yet another way to deliver continuous positive airway pressure?
Pediatrics
.
2008
;
121
(
1
):
82
88
16
Betters
KA
,
Gillespie
SE
,
Miller
J
,
Kotzbauer
D
,
Hebbar
KB
.
High flow nasal cannula use outside of the ICU; factors associated with failure
.
Pediatr Pulmonol
.
2017
;
52
(
6
):
806
812
17
Al-Alaiyan
S
,
Dawoud
M
,
Al-Hazzani
F
.
Positive distending pressure produced by heated, humidified high flow nasal cannula as compared to nasal continuous positive airway pressure in premature infants
.
J Neonatal Perinatal Med
.
2014
;
7
(
2
):
119
124
18
Spoletini
G
,
Alotaibi
M
,
Blasi
F
,
Hill
NS
.
Heated humidified high-flow nasal oxygen in adults: mechanisms of action and clinical implications
.
Chest
.
2015
;
148
(
1
):
253
261
19
Nielsen
KR
,
Ellington
LE
,
Gray
AJ
,
Stanberry
LI
,
Smith
LS
,
DiBlasi
RM
.
Effect of high-flow nasal cannula on expiratory pressure and ventilation in infant, pediatric, and adult models
.
Respir Care
.
2018
;
63
(
2
):
147
157
20
Cummings
JJ
,
Polin
RA
;
Committee on Fetus and Newborn, American Academy of Pediatrics
.
Noninvasive respiratory support
.
Pediatrics
.
2016
;
137
(
1
):
e20153758
21
Romans
RA
,
Schwartz
SM
,
Costello
JM
, et al
.
Epidemiology of noninvasive ventilation in pediatric cardiac ICUs
.
Pediatr Crit Care Med
.
2017
;
18
(
10
):
949
957

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.