Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System

I disagree that the statement "... observed an unexpected increase in mortality coincident with CPOE implementation" is not accurate. It is exactly what was seen in their population of patients. Moreover, the word "coincident" implies only correlation, not causality, and I find this wording quite appropriate.

That there were problems with the implementation is true, and I would posit, an understatement. That a hospital should not make system changes (e.g. not allowing access to a chart (of which I consider order entry an integral part), forcing orders to go through nursing, not allowing pre-writing of orders or pre-registration of patients, elimination of satellite pharmacies) coincident with implementation of a new documentation system (another system change, after all) is clear from this study, if it was not previously self-evident.

Some of the events that the authors correctly point out to be system changes may in fact have been forced by the CPOE software, though this is not very clear from the original article. Any mortality from such CPOE-forced changes would validly be blamed on the CPOE. However, not having order sets pre-programmed on day 1 of implementation (for example) is clearly a fault of the hospital, not of the CPOE system. The fact that other hospitals had few or no problems may be an indicator of the superiority of their chosen CPOE systems, and not merely superiority of their planning and implementation.

I do not see that the difference in pre-CPOE and post-CPOE periods is necessarily a problem. I feel that it is valid and, in fact, important to look even at transient effects. Optimally, there would be no change in, and hopefully improvement of, such variables both in the short and in the long term.

I also do not see the study of CPOE in a special patient population, in this case, acutely/emergently ill children, to be necessarily a problem, rather, to be a valid object of study in its own right. This would obviously result in a smaller patient population. Analogously, one does not confirm carpal tunnel syndrome by measuring the median nerve conduction velocity over the entire arm, because the large amount of normal nerve will 'dilute' the findings from the abnormal segment. A CPOE system may wonderfully solve lots of problems in a nonemergent setting but fail miserably when stressed in this way.

Now putting on my programmer hat, I find it appalling, though sadly not surprising, that such an obtuse user interface (UI) would have been created by the CPOE manufacturer. Programmer time is expensive, and it is far easier to push the product out the door and declare it to be functional rather than to spend the time watching users use it. Though programmers hate it, counting mouse clicks is important. New systems should be no worse than the system they are replacing in all respects, unless an extremely good reason can be provided to the contrary. Did the manufacturer study this issue? Something tells me it was not.

My own experience has been that naive users are the best testers. The best UI is that which allows such a user to start right in without a manual and perform correctly. Moreover, a good UI encourages good behavior by the user and helps avoid dangerous or stupid actions. If one cannot order warfarin 6.5 mg qpm directly, but only by first ordering the 5 mg dose, then separately ordering the 1 mg dose and overriding the system's warning that you've already ordered warfarin (the nerve!), and rechecking all relevant drug interactions, then ordering the 0.5 mg dose, again overriding the system's warnings and rechecking all relevant drug interactions, efficiency will be reduced. But this is the least of the problems: humans being humans, some other, faster, and likely less safe approach will be found around this awkwardness. 'Human Factors Engineering' is the name for this field, which has its own growing body of literature. Unfortunately, this is to be found mainly within the engineering journals, as the study of HFE within medicine is in an embryonic, or worse, zygotal phase.

Any programmer who has tried to teach the use of any reasonably complex program also knows that one session 3 months ahead of time is abjectly inadequate. Moreover, the best learning is 'spiral' learning, i.e., where concepts are returned to in more detail as time goes on. Implementation on a whole-hospital scale over several days should also be seen as potentially dangerous, barring hard evidence to the contrary (implementations in other hospitals). Implementations should be progressive, starting with the least critical patient units (rehab, psych), growing over time, with review of the software and other system changes along the entire path. Needless to say, any standard techniques, forms, orders, etc. should be in place ahead of time to the extent possible, at the very least those that are time-critical.

One final issue, not mentioned in the original article, must be brought up. Was there any financial relationship between the hospital and the CPOE manufacturer? Just such a relationship has been the cause for choosing one CPOE (or EHR) over another (usually better, but also more expensive) one on more than one occasion. Physicians are accused of being inordinately influenced by drug manufacturers' sales forces. Let us not forget that hospital administrations are hardly immune to this issue.

Mark D. Bej, M.D., FACNS

Conflict of Interest:

None declared

I generally agree with the comments of Phibbs et al and Levick with respect to the controversial article by Han et al on Computerized Physician Order Entry (CPOE), but would add three crucial observations:

(1) While many advocates of CPOE may be quick to implicate factors other than the CPOE system itself for the paper's results, the inadequacies of the system, or the implementation of the system, must be considered. Without commenting specifically on the design of the Cerner CPOE system, many current CPOE systems are abysmal in design. Their user interfaces, using branching menus, have changed little from that of the Technicon system introduced in 1974, despite many subsequent innovations in user interface technology. They are slow, cumbersome, and unintuitive. One of the more startling observations in the paper was that entry of an order took 1 to 2 minutes. This is simply unacceptable in a busy ICU setting. Commercial information technology vendors must rethink their CPOE designs, and hospital IT departments must not foist poor designs off on their physician users.

(2) I was president of Emtek Healthcare Systems, a company that developed critical and acute care charting systems, for many years. During that time, I was constantly amazed by the disparity between our most and least successful customers, all using the same software. Obviously, at some level, this implicates customers. At another level, however, system vendors must do a better job of transferring the configurations and processes of successful customers to their customer bases. Customers, for their parts, must do far more to learn and adopt best implementation practices.

(3) One of the paper's fuzziest points is the relationship between introducing the CPOE system and changing pharmacy practices. It stated that other researchers in the same facility simultaneeously found a decrease in adverse drug events (ADE). The increase in mortality and the decrease in ADEs may both be true, if an ADE does not consider the time delay between when a physician conceives the need for an order and when, for example, the order is received in the pharmacy, or, more important, when the medication is available for administrationn in the unit. One could observe no ADEs while being habitually "delayed" in actually administering the medication. Presumably, such delays matter in critical care. While measuring such delays may be difficult, they must be considered as a contributing factor in this study.

Conflict of Interest:

None declared

Dear Editors: In response to the article, “Unexpected Incresed Mortality after Implementation of a Commercially Sold Computerized Physician Order Entry System,” (Han, Carcillo, et.al.; Pediatrics vol 116, No. 6, December 2005) I would like to address several issues with the study and the conclusions.

Most hospitals that have implemented CPOE would probably concur that five months post go-live is inadequate a period to fully evaluate the morbidity/mortality impact of the system. Certainly if process, communication and infrastructure issues are still present during that period, outcomes may be even more suspect.

The issue with CPOE is usually not in the software, but in the process change that is required to successfully implement such a complex system. These challenges were well documented in the article. Many of the issues presented are similar to those outlined by Koppel, Metlya, Cohen, et.al. in the JAMA article, “Role of computerized physician order entry systems in facilitating medication errors” (2005:293:1197-2003). The rebuttals to that article are clearly applicable to the issues raised by Han, et. al. and do not bear repeating here. But rather than conclude that work process and infrastructure issues must be completely understood, investigated, and resolved prior to implementation, the authors conclude that hospitals should monitor mortality rates after CPOE implementation.

I believe the authors most accurately summarized their view in the Study Limitations, where they stated, “Without an organized systems approach to this problem, simple-minded physician investigators can provide only conjecture.” The authors clearly are guilty of simple-minded conjecture, supported by their own long list of study limitations.

Don Levick, M.D., MBA President Medical Staff Physician Liaison Information Services Lehigh Valley Hospital

Conflict of Interest:

None declared

We are writing in response to the paper by Han et al on Computerized Physician Order Entry (CPOE).(1) This paper concludes that the implementation of a CPOE system resulted in an unintended increase in mortality; this conclusion is not supported by the analysis presented in the paper. As the authors describe in the paper, the hospital simultaneously made other system changes and experienced significant implementation problems which the statistical analyses didn’t control for. Thus, the authors’ estimate of the effect of CPOE on mortality is biased and this bias may well be large. While the authors included many caveats in their discussion, these are not reflected in the conclusion where they stated in both the abstract and the text that they “observed an unexpected increase in mortality coincident with CPOE implementation.” This statement is not an accurate reflection of the actual findings; while the increase in mortality coincided with the implementation of CPOE, it is unclear whether CPOE itself caused the increase.

A more accurate summary of the findings is that there were significant problems with the implementation process for CPOE at this hospital and that the hospital simultaneously instituted other system changes that may have accounted for adverse effects. The combined effects of CPOE, the implementation problems the hospital experienced, and other simultaneous system changes was an increase in mortality, and it is not possible to assign any causality. The real lesson from this study is that there can be unintended adverse effects if hospitals don’t carefully plan for and implement major clinical transformations like CPOE.

While it was implemented at the same time, CPOE does not require that all medications be concentrated in a centralized pharmacy; this was a separate decision made by the hospital. This simultaneous change was the likely cause of a significant part of the observed increase in mortality. The authors’ analysis didn’t allow them to separate the effect of the centralization of medications from that due to CPOE, but they have attributed both to CPOE. Thus, the authors have reported a biased estimate of CPOE on mortality.

There were several process issues associated with the hospital’s implementation of CPOE that led to delays in the administration of medications. These included: • There was inadequate wireless bandwidth for the computer network. • The pharmacy could not process medication orders until they were separately activated by a nurse. • Physicians could not enter additional medication orders when a pharmacist was processing a previous order. • Transport patients could not be pre-registered and orders couldn’t be entered before a patient actually arrived at the hospital. • Even though the CPOE system allowed pre-programmed order sets for commonly used medications, these were not ready when the system went live. • The dosing schedules for repeat medications were not based on the time of the initial dose.

While these are important issues that contributed to delays in the administration of medications, they are issues related to a poor implementation process of the CPOE system at this hospital, not with CPOE itself. Other hospitals, including Childrens’ Hospital of Seattle, have developed adequate solutions to all of these problems and experienced smooth CPOE implementation without any detected adverse clinical impact. Neither Brigham and Women's Hospital nor Wishard Memorial Hospital experienced a change in mortality with implementation of CPOE (personal comment, David Bates and William Tierney). The authors themselves noted in their discussion that subsequent changes solved many of these problems. Thus, a well-planned implementation of CPOE could have prevented these problems. The speed of the implementation was also unusual—it is not good practice to implement in 7 days, as this does not leave time for mid- course corrections. All of these potentially preventable implementation problems are embedded in the authors’ estimate of the effect of CPOE on mortality, which further biases the estimate.

We have also noted several issues with how the authors designed the study that could also have influenced their estimate of the CPOE effect. • The pre CPOE period was much longer than the post implementation period, 13 vs. 5 months. No justification is made for the different sample frames. • The short post-implementation period did not allow the investigators to differentially test for problems associated with the start-up phase of CPOE with longer-term effects. • The risk model is grossly inadequate. There is only limited description of the variables used in the risk-adjustment model. Most of the variables in the risk model represent very wide ranges of added risk (for example, the risk associated with prematurity ranges from virtually no elevated risk at 36 weeks to almost 100% mortality <24 weeks). The failure to control for this heterogeneity of risk is a serious limitation of the risk model. Relatively modest changes in this uncontrolled risk over the two time periods could easily bias the findings. • Several of the risk factors in the model relate to neonatal admissions, yet the authors used the PRISM as a risk adjustment score. The PRISM is designed to adjust for the risk of older pediatric patients, not neonates. • The authors acknowledge the recent study from their institution that reported that the CPOE system reduced adverse drug events.(2) It is not clear why in this study the authors chose to only study a much smaller (1942 vs. 8619 patients) group of patients admitted via inter-hospital transfer instead of all admissions to the hospital. This resulted in only 75 deaths in the study sample, which magnifies the other limitations of the study design. No information is presented on how the study patients compare to other hospital patients or if CPOE had a mortality effect on these other patients. The lack of this addition information makes it difficult to reconcile the inconsistent findings across the two papers.

In conclusion, although the findings the authors observed are concerning, it is not possible to assign any causation. There are sufficient flaws with the authors’ analysis that it is not possible to conclude that risk-adjusted mortality actually increased. Further, if risk-adjusted mortality did increase, it was likely related to the way CPOE was implemented at this hospital, not CPOE itself.

Ciaran S. Phibbs, Ph.D. VA Palo Alto Health Care System and Stanford University School of Medicine

Arnold Milstein, M.D., M.P.H. Medical Director, Pacific Business Group on Health

Suzanne F. Delbanco, Ph.D. CEO, The Leapfrog Group

David Bates, M.D. Professor of Medicine, Harvard Medical School

1. Han YY, Carcillo JA, Venkataraman ST, Clark RSB, Watson RS, Nguyen TC, Bayir H, Orr RA. Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System. Pediatrics, 2005;116:1506-1512. 2. Upperman JS, Staley P, Rriend K. et al. The Impact of Hospitalwide Computerized Physician Order Entry on Medical Errors in a Pediatric Hospital. J Pediar Surg, 2005;40:57-59.

Conflict of Interest:

None declared