OBJECTIVES:

Diagnoses extracted from physician notes are used to calculate hospital quality metrics; failure to document high-risk diagnoses may lead to the appearance of worse-than-expected outcomes for complex patients. Academic hospitals often rely on documentation authored by trainees, yet residents receive little training in this regard. In this study, we evaluate inpatient pediatric resident notes to determine which high-risk diagnoses are commonly missed and assess the efficacy of a multitiered intervention to improve the documentation of these diagnoses.

METHODS:

In a baseline review of 220 charts, 13 frequently missed high-risk diagnoses were identified in 2013. Interventions began in 2014, including physician education and reference cards. The intervention also included note template prompts for 4 of the diagnoses. Using a standardized rubric, we reviewed charts for 3 years (2013, 2014, and 2015). The average within-disease probability of missed high-risk diagnoses was compared across time.

RESULTS:

There was a decrease in the probability of undocumented target high-risk diagnoses after the intervention (52% vs 36% in 2014 [odds ratio = 0.51; P < .001] and 37% in 2015 [odds ratio = 0.50; P < .001]). Documentation of diagnoses prompted by the note template was not significantly better than those targeted by the other interventions alone (P = .55).

CONCLUSIONS:

Pediatric residents were significantly less likely to omit a high-risk diagnosis in their notes after implementation of our documentation improvement program, suggesting that curriculum development is an effective method of improving documentation, with the goal of improving the accuracy of health systems performance indices.

As pediatric hospital staffs work toward greater quality and patient safety, accurate outcome data are vital. Many children’s hospital staffs now submit these data to external sources as part of quality of care reporting and are moving toward increased transparency for patients and consumers, making outcome data for a given institution, department, or even individual provider increasingly available to the public.14  Importantly, the information used to calculate quality metrics, such as the mortality index, is collected from physician documentation by certified medical coders as they capture diagnoses for billing. If the coded diagnoses fail to illustrate the severity of a patient’s illness because of incomplete physician documentation, aggregated quality data will not appropriately reflect the patient population’s complexity, leading to an appearance of inappropriately high mortality rates.2,5  On the basis of established mathematical models, each International Classification of Diseases diagnosis is weighted according to the magnitude of its effect on hospital resource use and on an individual patient’s severity of illness and risk of mortality.1,2,6,7  If a patient has a greater number of high-risk diagnoses, such as sepsis, pancytopenia, and thousands of others, that patient will be assigned a higher expected mortality risk. However, coders are not medically trained and cannot infer diagnoses that have not been explicitly documented by providers. For example, a physician may write that a patient is febrile, tachycardic, and has a positive blood culture result, but if the term “sepsis” is not stated, it cannot be coded and will not be reflected in the patient’s estimated mortality risk.

Clinical documentation improvement (CDI) programs, which are designed to ensure accurate documentation of these high-risk diagnoses, are now common in internal medicine and surgical fields, largely driven by clear financial incentives under Medicare payment structures.814  However, these financial incentives may not translate to pediatrics given the regional pay structure differences among children’s hospitals, so many pediatricians may not be supported by formal CDI initiatives.15  Additionally, residents often receive no formal training on CDI concepts despite being responsible for most documentation at academic medical centers.1618 

To date, there are no published studies on pediatric CDI efforts. Our goal with this study was to establish a clinical documentation curriculum at our institution and to examine its efficacy in improving documentation of high-risk diagnoses, with a greater aim of improving the accuracy of our quality data.

We performed this single-center time series study at a tertiary care academic children’s hospital within a hospital. The study was determined to be exempt from human subject research review by the Institutional Review Board at the University of California, Los Angeles.

To assess current documentation practices, we reviewed 200 charts that were randomly selected from all 1119 pediatric ward and ICU discharges between July 1 and December 31, 2013. We also reviewed all 20 mortalities from the calendar year because these patients were presumed more likely to have high-risk diagnoses. This resulted in a total of 220 baseline charts reviewed. Eighty-four (38%) of these 220 charts were ICU patients as defined by spending time in the ICU at any point during that encounter. Four pediatric hospitalist attending physicians were trained in reviewing charts, specifically looking for missed high-risk diagnoses as defined by the Centers for Medicare and Medicaid Services.19  Only notes written by residents were reviewed. Of the baseline charts reviewed, 67% had 1 or more missing high-risk diagnosis. Diagnoses that were missed 5 or more times were targeted for intervention, resulting in the following list of 13 diagnoses: acidosis, alkalosis, acute kidney injury, chronic renal failure, epilepsy, heart failure, hypertension, malnutrition, neutropenia, anemia, pancytopenia, sepsis, and shock.

Interventions began in July 2014 at the start of the new academic year. Educational conferences on the importance of documenting specific diagnoses in terms that coders will recognize were presented to faculty and residents. These lectures were repeated over the course of several months in resident orientation, noon conferences, departmental quality meetings, and faculty meetings. A reference card (Fig 1) defining the target diagnoses was distributed electronically to residents and posted in their workrooms to educate them about when these diagnoses apply. The diagnoses were defined by using established literature; when no clear guidelines were available, we relied on expert consensus among subspecialists at our institution. We also revised the electronic medical record (EMR) note templates, which are used by all our pediatric residents, to include 4 of the target diagnoses that were easiest to incorporate without cluttering the note or overly encumbering note writers. Drop-down lists that were already present within the templates were modified to specifically mention the diagnoses of acute kidney injury, anemia, pancytopenia, and sepsis to prompt residents to include them when applicable.

FIGURE 1

Resident reference card for high-risk diagnoses. a Adapted from Goldstein B, Giroir B, Randolph A; International Consensus Conference on Pediatric Sepsis. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med. 2005;6(1):2–8. b Adapted from Tobin MJ. Principles and Practice of Mechanical Ventilation. 2nd ed. New York, New York: McGraw-Hill Medical Publishing Division; 2006. c Adapted from Jobe AH, Bancalari E. Bronchopulmonary dysplasia. Am J Respir Crit Care Med. 2001;163(7):1723. d Adapted from Chameides L, et al eds. Pediatric Advanced Life Support Provider Manual. Dallas, TX: American Heart Association Subcommittee on Pediatric Resuscitation; 2006:85. e Adapted from Akcan-Arikan A1, Zappitelli M, Loftis LL, Washburn KK, Jefferson LS, Goldstein SL. Modified RIFLE criteria in critically ill children with acute kidney injury. Kidney Int. 2007;71(10):1028–1035. f Adapted from the National Kidney Foundation Practice Guidelines for Chronic Kidney Disease. g Adapted from National High Blood Pressure Education Program Working Group on High Blood Pressure in Children and Adolescents. The fourth report on the diagnosis, evaluation, and treatment of high blood pressure in children and adolescents. Pediatrics. 2004;114(2, suppl 4th report):555. h Adapted from Engel J Jr; International League Against Epilepsy (ILAE). A proposed diagnostic scheme for people with epileptic seizures and with epilepsy: report of the ILAE task force on classification and terminology. Epilepsia. 2001;42(6):796–803. i Adapted from Posner JB, Saper CB, Schiff ND, Plum F. Diagnosis of Stupor and Coma. 4th ed. New York, NY: Oxford University Press; 2007:401 pp, illustrated. j World Health Organization. Global Database on Child Growth and Malnutrition. Available at: http://www.who.int/nutgrowthdb/about/introduction/en/index5.html. Accessed June 7, 2014. AMS, altered mental status; ARDS, acute respiratory distress syndrome; ARF, acute renal failure; CO, cardiac output; Cr, creatinine; DCM, dilated cardiomyopathy; DIC, disseminated intravascular coagulation; eCCI, estimated creatinine clearance; ESRD, end stage renal disease; GFR, glomerular filtration rate; HCM, hypertrophic cardiomyopathy; HR, heart rate; L, left; pRIFLE, pediatric risk, injury, failure, loss, end stage renal disease; R, right; RCM, restrictive cardiomyopathy; RR, respiratory rate; sCR, serum creatinine; SIRS, systematic inflammatory response syndrome; SvO2, mixed venous oxygen saturation; SVR, systemic vascular resistance; UOP, urine output; WBC, white blood cell; %ile, percentile.

FIGURE 1

Resident reference card for high-risk diagnoses. a Adapted from Goldstein B, Giroir B, Randolph A; International Consensus Conference on Pediatric Sepsis. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med. 2005;6(1):2–8. b Adapted from Tobin MJ. Principles and Practice of Mechanical Ventilation. 2nd ed. New York, New York: McGraw-Hill Medical Publishing Division; 2006. c Adapted from Jobe AH, Bancalari E. Bronchopulmonary dysplasia. Am J Respir Crit Care Med. 2001;163(7):1723. d Adapted from Chameides L, et al eds. Pediatric Advanced Life Support Provider Manual. Dallas, TX: American Heart Association Subcommittee on Pediatric Resuscitation; 2006:85. e Adapted from Akcan-Arikan A1, Zappitelli M, Loftis LL, Washburn KK, Jefferson LS, Goldstein SL. Modified RIFLE criteria in critically ill children with acute kidney injury. Kidney Int. 2007;71(10):1028–1035. f Adapted from the National Kidney Foundation Practice Guidelines for Chronic Kidney Disease. g Adapted from National High Blood Pressure Education Program Working Group on High Blood Pressure in Children and Adolescents. The fourth report on the diagnosis, evaluation, and treatment of high blood pressure in children and adolescents. Pediatrics. 2004;114(2, suppl 4th report):555. h Adapted from Engel J Jr; International League Against Epilepsy (ILAE). A proposed diagnostic scheme for people with epileptic seizures and with epilepsy: report of the ILAE task force on classification and terminology. Epilepsia. 2001;42(6):796–803. i Adapted from Posner JB, Saper CB, Schiff ND, Plum F. Diagnosis of Stupor and Coma. 4th ed. New York, NY: Oxford University Press; 2007:401 pp, illustrated. j World Health Organization. Global Database on Child Growth and Malnutrition. Available at: http://www.who.int/nutgrowthdb/about/introduction/en/index5.html. Accessed June 7, 2014. AMS, altered mental status; ARDS, acute respiratory distress syndrome; ARF, acute renal failure; CO, cardiac output; Cr, creatinine; DCM, dilated cardiomyopathy; DIC, disseminated intravascular coagulation; eCCI, estimated creatinine clearance; ESRD, end stage renal disease; GFR, glomerular filtration rate; HCM, hypertrophic cardiomyopathy; HR, heart rate; L, left; pRIFLE, pediatric risk, injury, failure, loss, end stage renal disease; R, right; RCM, restrictive cardiomyopathy; RR, respiratory rate; sCR, serum creatinine; SIRS, systematic inflammatory response syndrome; SvO2, mixed venous oxygen saturation; SVR, systemic vascular resistance; UOP, urine output; WBC, white blood cell; %ile, percentile.

Close modal

A standardized rubric was created to specifically look for target diagnoses (Supplemental Fig 3). Using this rubric, the 4 reviewers achieved >90% interrater agreement on a test sample of 10 charts before initiation of data collection. A total of 100 charts were then randomly selected from ward and ICU discharges between July 1, 2013, and December 31, 2013, for our preintervention data and 2014 and 2015 for postintervention data. The 100 charts selected from 2013 were unique for this phase of the study because we were now reviewing charts that were focused only on the 13 diagnoses targeted after the baseline review. This sample size was selected on the basis of an a priori power analysis (80% power, 5% α) to detect a 20% reduction in missed diagnoses. Additionally, all mortalities during those intervals were reviewed. Newborn nursery, NICU, and cardiothoracic ICU charts were excluded because these populations have a unique set of diagnoses that are less generalizable to other patients. Patients admitted for 2 or fewer calendar days were also excluded because they were less likely to have high-risk diagnoses. Charts that were found to meet exclusion criteria after chart review began were omitted without being replaced. This resulted in a total of 97 reviewed charts in 2013, 117 in 2014, and 114 in 2015 after applying inclusion and exclusion criteria. Each target diagnosis was coded as “present and documented,” “present but not documented,” or “not present” within each chart.

The primary outcome of this study was a reduction in undocumented target high-risk diagnoses after the intervention. Secondary outcomes were the efficacy of EMR templates and whether improved documentation would lead to an increase in the expected mortality (ie, the percent of patients expected to die on the basis of illness severity) as measured by Vizient, formerly known as the University Health-System Consortium.20  The average within-disease probability of missed high-risk diagnoses was compared across time (preintervention and postintervention) by using a mixed effects logistic regression model with disease random intercept. An interaction P value for the differences between years was used to assess the utility of the note template. Differences in expected mortality were compared by using an independent samples test of proportions.

Overall in 2013, there was a 52% probability of a target diagnosis being “present but not documented” (95% confidence interval [CI]: 0.47 to 0.57), which decreased to 36% in 2014 (95% CI: 0.32 to 0.40) after implementation of the interventions (odds ratio = 0.51; P < .001; Table 1). This reduction was sustained in 2015 with a 37% probability (P < .001; 95% CI: 0.32 to 0.42). As for the individual diagnoses, there was a reduction in missed documentation for 3 of the 13 target diagnoses (Fig 2). The results for the remainder of the diagnoses are available in Supplemental Table 2.

TABLE 1

Analysis of Missed Target Diagnoses and Expected Mortality Over Time

201320142015P (2013 vs 2014)P (2013 vs 2015)P (2014 vs 2015)
No. missed diagnoses per charta 1.93 ± 1.13 (1–5) 1.72 ± 1.04 (1–4) 1.53 ± 0.84 (1–4) .18 .03 .46 
Probability of missed diagnosisb       
 Overall 0.52 0.36 0.37 <.001 <.001 .94 
 ICU 0.49 0.43 0.40 .35 .13 .59 
 Ward 0.59 0.38 0.39 <.001 <.001 .81 
Expected mortalityc 0.76% 0.78% 0.93% .93 .49 .54 
201320142015P (2013 vs 2014)P (2013 vs 2015)P (2014 vs 2015)
No. missed diagnoses per charta 1.93 ± 1.13 (1–5) 1.72 ± 1.04 (1–4) 1.53 ± 0.84 (1–4) .18 .03 .46 
Probability of missed diagnosisb       
 Overall 0.52 0.36 0.37 <.001 <.001 .94 
 ICU 0.49 0.43 0.40 .35 .13 .59 
 Ward 0.59 0.38 0.39 <.001 <.001 .81 
Expected mortalityc 0.76% 0.78% 0.93% .93 .49 .54 
a

Values are expressed as average ± SD (range). P calculated by using Wilcoxon rank-sum test.

b

P calculated by using mixed effects logistic regression model with disease random intercept.

c

P calculated by using 2 sample test of proportions.

FIGURE 2

Reduction in missed target diagnoses over time. Compared with 2013, statistically significant reduction in missed diagnoses was found for all target diagnoses combined (2014 [P < .001] and 2015 [P < .001]), acidosis (2015 only [P = .04]), acute kidney injury (2014 [P = .006] and 2015 [P = .03]), and anemia (2014 only [P = .04]).

FIGURE 2

Reduction in missed target diagnoses over time. Compared with 2013, statistically significant reduction in missed diagnoses was found for all target diagnoses combined (2014 [P < .001] and 2015 [P < .001]), acidosis (2015 only [P = .04]), acute kidney injury (2014 [P = .006] and 2015 [P = .03]), and anemia (2014 only [P = .04]).

Close modal

The percent of the total charts reviewed per year that were mortalities remained consistent over the study interval (13%, 15%, and 16% in 2013, 2014, and 2015, respectively; P = .89). The percent of patients with an ICU stay also remained the same (42%, 33%, and 39% in 2013, 2014, and 2015, respectively; P = .32). When further analyzed by ICU status, high-risk diagnoses were more prevalent in ICU patients compared with ward patients (26% vs 15.7%; P < .001). There was sustained improvement in missed diagnoses in ward charts, which went from 59% in 2013 to 38% in 2014 (P < .001) and 39% in 2015 (P < .001), but the improvement in ICU charts was not statistically significant.

For our secondary outcome measures, documentation of diagnoses included in the note template was not significantly better than those targeted by educational interventions alone (P = .55). Expected mortality rates did increase by 22% over the study interval, but this was not statistically significant.

Our study revealed that before our interventions, the probability of missing a target diagnosis was >50%, suggesting that our patient population had a higher risk of mortality than our coded data would indicate. The educational intervention was effective at improving inpatient pediatric resident documentation of high-risk diagnoses and sustaining improvement with ongoing education. This approach may be particularly useful for pediatric institutions without funding for formal CDI programs that may need to rely on physician-driven initiatives.

As expected, ICU patients had more high-risk diagnoses than ward patients, but the 2 groups were similarly susceptible to omissions in documentation; in fact, only the ward patient group showed statistically significant improvement. Thus, on the basis of our data, CDI efforts should include lower-acuity patients as well as ICU patients. We had anticipated that an increased ability to capture high-risk diagnoses would lead to an increased expected mortality, but this was not statistically significant.

Similar measures such as lecture series and reference cards have been used in publications in adult literature to achieve a 43% increase in expected mortality and increases in reimbursement up to 16% to 24%.21,22  However, many of these institutions also retain the services of a CDI team that audits notes in real time and queries physicians regarding missed diagnoses that may need to be added to their documentation.11  In fact, at the time of this study, our institution’s CDI specialists were reviewing all adult inpatient documentation but not pediatric charts because of funding limitations and the lack of perceived benefits. As a result of our work, our institution will be providing resources for a CDI team that will use our reference card to query documentation errors in real time, further bridging the gap between what physicians document and what coders interpret.

A notable strength of our study was that we evaluated undocumented and documented diagnoses as a percent of the total number of instances in which the diagnosis was clinically applicable. In this way, we were able to control for changes in patient complexity over time, in contrast to previous studies in which the authors only looked at absolute numbers of documented high-risk diagnoses, which may be confounded by increases in the prevalence of studied diagnoses over time.

A limitation of our study was its single-center design that was focused solely on resident documentation, so our data may not be generalizable to nonteaching institutions. Additionally, because our educational interventions consisted of multiple elements, we cannot comment on the effectiveness of any single component. The retrospective nature of the review made it impossible to determine the “false-positive” rate, that is, the diagnoses that were documented but may not in fact have been applicable. The phenomenon of manipulating documentation to maximize reimbursement has been labeled “diagnosis-related group creep” and is a potential adverse consequence of CDI efforts.23  Finally, although we as physician reviewers had strong interrater agreement with each other, we were unable to correlate our findings with our coders because they review all physician notes, whereas in our study, we only reviewed resident documentation.

Our educational interventions successfully decreased the probability of missing diagnoses by 16%; however, there are still significant opportunities for further improvements to ensure that our quality metrics reflect the complexity of our patient population. Future work will expand on the current intervention to target additional diagnoses and to include attending and fellow documentation. There also may be more applications of EMR technology to assist in documentation on the basis of existing data in other areas of the chart, such as importing physician-generated “problem lists” into daily notes to make it accessible to coders. In further studies, authors may also assess the effect of these interventions on resident experience. Encouraging use of more precise language may help trainees recognize and treat high-risk diagnoses more promptly, but it is important to identify and prevent any adverse effects on resident workflow.

Drs Kulkarni, Crummey, Heath, and Kosack all conceptualized and designed the study, performed a substantial portion of the data collection, contributed to drafting the initial manuscript, and reviewed and revised the manuscript; Mr Jackson helped design the study, performed all statistical analysis, and contributed to drafting the initial manuscript; and all authors approved the final manuscript as submitted.

FUNDING: Although no direct funding was required, statistical support was received through the National Institutes of Health National Center for Advancing Translational Science University of California Los Angeles Clinical and Translational Science Institute grant UL1TR001881.

1
University Health System Consortium
.
Mortality risk adjustment methodology for University Health System’s clinical data base
.
2008
.
2
Iezzoni
LI
.
Assessing quality using administrative data
.
Ann Intern Med
.
1997
;
127
(
8, pt 2
):
666
674
3
U.S. Centers for Medicare and Medicaid Services
.
Hospital compare
.
Available at: https://www.medicare.gov/hospitalcompare/search.html. Accessed November 26, 2017
4
The Joint Commission
.
Quality check
.
Available at: http://www.healthcarequalitydata.org/. Accessed November 26, 2017
5
Zuidema
GD
,
Dans
PE
,
Dunlap
ED
.
Documentation of care and prospective payment. One hospital’s experience
.
Ann Surg
.
1984
;
199
(
5
):
515
521
6
Rosenthal
GE
,
Harper
DL
,
Quinn
LM
,
Cooper
GS
.
Severity-adjusted mortality and length of stay in teaching and nonteaching hospitals. Results of a regional study
.
JAMA
.
1997
;
278
(
6
):
485
490
7
Aron
DC
,
Harper
DL
,
Shepardson
LB
,
Rosenthal
GE
.
Impact of risk-adjusting cesarean delivery rates when reporting hospital performance
.
JAMA
.
1998
;
279
(
24
):
1968
1972
8
Falcone
RE
,
Wanamaker
SR
,
Monk
J
,
Carey
LC
,
Valenziano
C
.
Physician review improves hospital DRG reimbursement in injury
.
J Trauma
.
1992
;
33
(
3
):
370
373; discussion 373–374
9
Hoffman
GD
,
Jones
DK
.
Prebilling DRG training can increase hospital reimbursement
.
Healthc Financ Manage
.
1993
;
47
(
9
):
58
, 60, 62
10
Barnes
SL
,
Waterman
M
,
Macintyre
D
,
Coughenour
J
,
Kessel
J
.
Impact of standardized trauma documentation to the hospital’s bottom line
.
Surgery
.
2010
;
148
(
4
):
793
797; discussion 797–798
11
Frazee
RC
,
Matejicka
AV
 II
,
Abernathy
SW
, et al
.
Concurrent chart review provides more accurate documentation and increased calculated case mix index, severity of illness, and risk of mortality
.
J Am Coll Surg
.
2015
;
220
(
4
):
652
656
12
Kittinger
BJ
,
Matejicka
A
 II
,
Mahabir
RC
.
Surgical precision in clinical documentation connects patient safety, quality of care, and reimbursement
.
Perspect Health Inf Manag
.
2016
;
13
:
1f
13
Momin
SR
,
Lorenz
RR
,
Lamarre
ED
.
Effect of a documentation improvement program for an academic otolaryngology practice
.
JAMA Otolaryngol Head Neck Surg
.
2016
;
142
(
6
):
533
537
14
Reyes
C
,
Greenbaum
A
,
Porto
C
,
Russell
JC
.
Implementation of a clinical documentation improvement curriculum improves quality metrics and hospital charges in an academic surgery department
.
J Am Coll Surg
.
2017
;
224
(
3
):
301
309
15
Long
MJ
,
Dreachslin
JL
,
Fisher
J
.
Should children’s hospitals have special consideration in reimbursement policy?
Health Care Financ Rev
.
1986
;
8
(
1
):
55
63
16
Pines
JM
,
Braithwaite
S
.
Documentation and coding education in emergency medicine residency programs: a national survey of residents and program directors
.
Cal J Emerg Med
.
2004
;
5
(
1
):
3
8
17
Fakhry
SM
,
Robinson
L
,
Hendershot
K
,
Reines
HD
.
Surgical residents’ knowledge of documentation and coding for professional services: an opportunity for a focused educational offering
.
Am J Surg
.
2007
;
194
(
2
):
263
267
18
Yount
KW
,
Reames
BN
,
Kensinger
CD
, et al
.
Resident awareness of documentation requirements and reimbursement: a multi-institutional survey
.
Ann Thorac Surg
.
2014
;
97
(
3
):
858
864
19
Centers for Medicare and Medicaid Services
.
Draft ICD-10-CM/PCS MS-DRGv31.0 definitions manual
.
20
VIZIENT Clinical Data Base/Resource Manager
.
Patient outcomes by hospital/hospital system
.
21
Spellberg
B
,
Harrington
D
,
Black
S
,
Sue
D
,
Stringer
W
,
Witt
M
.
Capturing the diagnosis: an internal medicine education program to improve documentation
.
Am J Med
.
2013
;
126
(
8
):
739
743.e1
22
Zalatimo
O
,
Ranasinghe
M
,
Harbaugh
RE
,
Iantosca
M
.
Impact of improved documentation on an academic neurosurgical practice
.
J Neurosurg
.
2014
;
120
(
3
):
756
763
23
Simborg
DW
.
DRG creep: a new hospital-acquired disease
.
N Engl J Med
.
1981
;
304
(
26
):
1602
1604

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data