Kevin Heard, BS

Insulin therapy in the hospital setting can cause hypoglycemia, which may lead to increased mortality and length of stay (LOS).[1, 2, 3] Hypoglycemia is associated with cardiovascular, cerebrovascular, and patient fall events.[4, 5] The Centers for Medicare and Medicaid Services have designated both severe hypoglycemia (SH) with harm and diabetic ketoacidosis as hospital acquired conditions (HAC) or never events. The Society for Hospital Medicine (SHM) defines SH in the hospital as a blood glucose (BG) <40 mg/dL. Minimizing episodes of SH is important for patient health outcomes, patient safety, and for healthcare facilities' safety metrics.

Many factors contribute to SH including excessive insulin doses, medication errors, inappropriate timing of insulin doses with food intake, changes in nutritional status, impaired renal function, and changes in medications such as steroids.[6] As part of a multiyear project in patient safety, an inpatient hypoglycemia alert algorithm was developed based on a multivariate analysis of individual patient demographic, pharmacy, laboratory, and glucometric data. The algorithm was previously shown to have a 75% sensitivity to predict episodes of SH.[7] In this study, we tested whether a predictive real‐time informatics hypoglycemia alert based on the tested algorithm, along with trained nurses, would result in a decreased frequency of SH events compared to usual care. We hypothesized that this alert would result in a reduction of SH events in those patients at high risk for hypoglycemia.

METHODS

Study Design and Population

This prospective cohort‐intervention study involved inpatients admitted to Barnes‐Jewish Hospital in St. Louis, Missouri, the academic hospital of Washington University School of Medicine (WUSM), from August 2011 through December 2011. Fourteen floors, including 10 internal medicine and 4 cardiology medicine floors, were selected based upon a high frequency of severe hypoglycemic events noted in 2010. Six of the internal medicine floors were designated as intervention floors, and 8 were designated as control floors, including the 4 cardiology units. The study population consisted of patients receiving diabetic medications on study floors who had a BG <90 mg/dL during their hospital stay (Figure 1). The study was approved by the WUSM institutional review board and included a waiver of consent for individual patients.

Figure 1

The pharmacy informatics system was programmed with the previously developed hypoglycemia alert to prospectively identify those patients at high risk of hypoglycemia based on real‐time patient information.[7] Patients were identified as high risk on study floors if insulin or an oral antihyperglycemic agent was prescribed and if their hypoglycemia informatics generated risk score was >35 within 24 hours of having a capillary or venous BG<90 mg/dL. The risk score of 35 corresponded to a 50% sensitivity for a subsequent BG <60 mg/dL and a 75% sensitivity for a BG <40 mg/dL. Patients who generated an alert once during their hospital stay were assigned to 1 of 3 categories based on their admission division and risk score: high‐risk intervention (HR‐I), high‐risk control (HR‐C), or low risk (LR). LR patients also had a BG <90 mg/dL during their stay, but a risk score of <35.

The electronic alert for HR‐I patients was sent by pager to division‐specific charge nurses. Fourteen charge nurses on intervention divisions were trained to assess the alert, interview the patient, identify an alternate dosing strategy, and collaborate with the patient's physicians. HR‐C patients were identified on control divisions based on the same criteria as intervention patients, but no alert was generated. Control patients' charts were reviewed and evaluated upon discharge by the research team‐certified diabetes nurse educator to determine whether the treating physician had identified the SH risk and had changed insulin orders.

Nurses and physicians caring for patients on study divisions provided informed consent to participate in the study. Nurses' satisfaction with the alert process and physician interaction was assessed with a collaboration scale that was completed after each alert (see Supporting Information, Appendix A, in the online version of this article).[8]

RESULTS

DISCUSSION

Hospitals are accountable for safe and effective care of patients with hyperglycemia, which includes prevention of medication‐induced hypoglycemia. We have developed a predictive informatics hypoglycemia risk alert that, when tested in a real‐world situation, significantly reduced the rate of SH in hospitalized patients without increasing severe hyperglycemia. The alert algorithm correctly identified patients who were at high risk for hypoglycemia and allowed caretakers the opportunity to lower that risk. The positive predictive value of the alert was low but acceptable at 9.7%, owing to the overall low rate of hypoglycemia in the patient population.

The alert model tested involved 3 components for success: the automated alert, trained charge nurse responders, and an interaction between the nurse responder and the care provider. HR‐I patients were interviewed and assessed for problems associated with oral intake, dietary habits, medication compliance, and hypoglycemia at home prior to communicating with physicians. The extensive training and proficiency in patient assessment and SBAR communication process required by nurses was paramount in the success of the alert. However, the alert provided a definitive risk assessment that was actionable, versus more global instruction, which has not had the same impact in risk reduction. Based on feedback collected from nurses at the study end, they felt the alert process was within their scope of practice and was not unduly burdensome. They also found that the training in diabetes management and SBAR communication techniques, in addition to the alert system, were useful in protecting patients from medication harm.

Physicians for HR‐C patients missed many opportunities to effectively intervene and thereby reduce the likelihood of an SH event. Our assumption is that the clinicians did not ascertain the risk of SH, which was reflected by the fact that orders were changed in 40.7% of HR‐I patients versus only 20.5% in the HR‐C group. Having alerts go directly to nurses rather than physicians permitted inclusion of additional information, such as caloric intake and testing schedules, so that changes in orders would have greater context, and the importance of mild hypoglycemia would not be overlooked.[16] Glycemic control is challenging for providers in the inpatient setting, as there is little time to test and titrate doses of insulin to achieve control. Tight glycemic control has become the primary focus of diabetes management in the outpatient setting to reduce long‐term risks of microvascular complications.[17, 18] However, establishing glycemic targets in the inpatient setting has been difficult because the risk for hypoglycemia increases with tighter control.[19, 20] Inpatient hypoglycemia has been associated with increased mortality, particularly in critically ill patients.[21, 22] Many factors contribute to hypoglycemia including low creatinine clearance, low body weight, untested insulin doses, errors in insulin administration, unexpected dietary changes, changes in medications affecting BG levels, poor communication during times of patient transfer to different care teams, and poor coordination of BG testing with insulin administration at meal times. A multifaceted approach aimed at improving both clinician and nurse awareness, and providing real‐time risk assessment is clearly required to insure patient safety.[6, 13, 23, 24]

There are significant economic benefits to avoiding SH in the hospital given the adverse outcomes associated with HACs and the extra cost associated with these conditions. In hospitalized patients, hypoglycemia worsens outcomes leading to higher costs due to longer LOS (by 3 days), higher inpatient charges (38.9%), and higher risk of discharge to a skilled nursing facility.[1, 3, 25, 26] Conversely, improved glycemic control can reduce surgical site infections, perioperative morbidity, and hospital LOS.[27] The high prevalence of insulin use among inpatients, many of whom have high‐risk characteristics, creates a milieu for both hyper‐ and hypoglycemia. Other groups have described a drop in hypoglycemia rates related to the use of standardized diabetes order sets and nurse and physician education, but this is the first study that used informatics in a prospective manner to identify patients who are at high risk for developing hypoglycemia and then specifically targeted those patients.[28] The alert process was modeled after a similar alert developed in our institution for identifying medicine patients at risk for sepsis.[29] Given the paucity of data related to inpatient glycemia risk reduction, this study is particularly relevant for improving patient safety.

The major limitation of this study is that it was not randomized at the patient level. Patients were assigned to intervention and control groups based on their occupancy on specific hospital floors to avoid treatment bias. Bias was assessed due to this nonrandom assignment by comparing demographic and clinical factors of HR patients between intervention and control floors, and found significant differences in HbA1c and admitting diagnosis. As the control group had lower HbA1c values than the intervention group, and it is known from the Diabetes Control and Complications Trial and Action to Control Cardiovascular Risk in Diabetes trial that lower HbA1c increases the risk of hypoglycemia, our results may be biased by the level of glucose control on admission.[30, 31] Admitting diagnoses differed significantly between intervention and control patients as did the Charlson Comorbidity Index score; however, the hypoglycemia alert system does not include patient diagnoses or comorbidities, and as such provided equipoise with regard to risk reduction regardless of presenting illness. This study included trained nurses, which may be beyond the scope of every institution and thereby limit the effectiveness of the alert in reducing risk. However, as a result of this study, the alert was expanded to other acute care floors at our hospital as well as other hospitals in the Barnes‐Jewish Hospital system.

In summary, this study showed a 68% decrease in episodes of SH in a high‐risk patient cohort on diabetic medications using a hypoglycemia alert system. The results of this study demonstrate the validity of a systems‐based approach to reduce SH in high‐risk inpatients.

Insulin therapy in the hospital setting can cause hypoglycemia, which may lead to increased mortality and length of stay (LOS).[1, 2, 3] Hypoglycemia is associated with cardiovascular, cerebrovascular, and patient fall events.[4, 5] The Centers for Medicare and Medicaid Services have designated both severe hypoglycemia (SH) with harm and diabetic ketoacidosis as hospital acquired conditions (HAC) or never events. The Society for Hospital Medicine (SHM) defines SH in the hospital as a blood glucose (BG) <40 mg/dL. Minimizing episodes of SH is important for patient health outcomes, patient safety, and for healthcare facilities' safety metrics.

Many factors contribute to SH including excessive insulin doses, medication errors, inappropriate timing of insulin doses with food intake, changes in nutritional status, impaired renal function, and changes in medications such as steroids.[6] As part of a multiyear project in patient safety, an inpatient hypoglycemia alert algorithm was developed based on a multivariate analysis of individual patient demographic, pharmacy, laboratory, and glucometric data. The algorithm was previously shown to have a 75% sensitivity to predict episodes of SH.[7] In this study, we tested whether a predictive real‐time informatics hypoglycemia alert based on the tested algorithm, along with trained nurses, would result in a decreased frequency of SH events compared to usual care. We hypothesized that this alert would result in a reduction of SH events in those patients at high risk for hypoglycemia.

METHODS

Study Design and Population

This prospective cohort‐intervention study involved inpatients admitted to Barnes‐Jewish Hospital in St. Louis, Missouri, the academic hospital of Washington University School of Medicine (WUSM), from August 2011 through December 2011. Fourteen floors, including 10 internal medicine and 4 cardiology medicine floors, were selected based upon a high frequency of severe hypoglycemic events noted in 2010. Six of the internal medicine floors were designated as intervention floors, and 8 were designated as control floors, including the 4 cardiology units. The study population consisted of patients receiving diabetic medications on study floors who had a BG <90 mg/dL during their hospital stay (Figure 1). The study was approved by the WUSM institutional review board and included a waiver of consent for individual patients.

Figure 1

The pharmacy informatics system was programmed with the previously developed hypoglycemia alert to prospectively identify those patients at high risk of hypoglycemia based on real‐time patient information.[7] Patients were identified as high risk on study floors if insulin or an oral antihyperglycemic agent was prescribed and if their hypoglycemia informatics generated risk score was >35 within 24 hours of having a capillary or venous BG<90 mg/dL. The risk score of 35 corresponded to a 50% sensitivity for a subsequent BG <60 mg/dL and a 75% sensitivity for a BG <40 mg/dL. Patients who generated an alert once during their hospital stay were assigned to 1 of 3 categories based on their admission division and risk score: high‐risk intervention (HR‐I), high‐risk control (HR‐C), or low risk (LR). LR patients also had a BG <90 mg/dL during their stay, but a risk score of <35.

The electronic alert for HR‐I patients was sent by pager to division‐specific charge nurses. Fourteen charge nurses on intervention divisions were trained to assess the alert, interview the patient, identify an alternate dosing strategy, and collaborate with the patient's physicians. HR‐C patients were identified on control divisions based on the same criteria as intervention patients, but no alert was generated. Control patients' charts were reviewed and evaluated upon discharge by the research team‐certified diabetes nurse educator to determine whether the treating physician had identified the SH risk and had changed insulin orders.

Nurses and physicians caring for patients on study divisions provided informed consent to participate in the study. Nurses' satisfaction with the alert process and physician interaction was assessed with a collaboration scale that was completed after each alert (see Supporting Information, Appendix A, in the online version of this article).[8]

RESULTS

DISCUSSION

Hospitals are accountable for safe and effective care of patients with hyperglycemia, which includes prevention of medication‐induced hypoglycemia. We have developed a predictive informatics hypoglycemia risk alert that, when tested in a real‐world situation, significantly reduced the rate of SH in hospitalized patients without increasing severe hyperglycemia. The alert algorithm correctly identified patients who were at high risk for hypoglycemia and allowed caretakers the opportunity to lower that risk. The positive predictive value of the alert was low but acceptable at 9.7%, owing to the overall low rate of hypoglycemia in the patient population.

The alert model tested involved 3 components for success: the automated alert, trained charge nurse responders, and an interaction between the nurse responder and the care provider. HR‐I patients were interviewed and assessed for problems associated with oral intake, dietary habits, medication compliance, and hypoglycemia at home prior to communicating with physicians. The extensive training and proficiency in patient assessment and SBAR communication process required by nurses was paramount in the success of the alert. However, the alert provided a definitive risk assessment that was actionable, versus more global instruction, which has not had the same impact in risk reduction. Based on feedback collected from nurses at the study end, they felt the alert process was within their scope of practice and was not unduly burdensome. They also found that the training in diabetes management and SBAR communication techniques, in addition to the alert system, were useful in protecting patients from medication harm.

Physicians for HR‐C patients missed many opportunities to effectively intervene and thereby reduce the likelihood of an SH event. Our assumption is that the clinicians did not ascertain the risk of SH, which was reflected by the fact that orders were changed in 40.7% of HR‐I patients versus only 20.5% in the HR‐C group. Having alerts go directly to nurses rather than physicians permitted inclusion of additional information, such as caloric intake and testing schedules, so that changes in orders would have greater context, and the importance of mild hypoglycemia would not be overlooked.[16] Glycemic control is challenging for providers in the inpatient setting, as there is little time to test and titrate doses of insulin to achieve control. Tight glycemic control has become the primary focus of diabetes management in the outpatient setting to reduce long‐term risks of microvascular complications.[17, 18] However, establishing glycemic targets in the inpatient setting has been difficult because the risk for hypoglycemia increases with tighter control.[19, 20] Inpatient hypoglycemia has been associated with increased mortality, particularly in critically ill patients.[21, 22] Many factors contribute to hypoglycemia including low creatinine clearance, low body weight, untested insulin doses, errors in insulin administration, unexpected dietary changes, changes in medications affecting BG levels, poor communication during times of patient transfer to different care teams, and poor coordination of BG testing with insulin administration at meal times. A multifaceted approach aimed at improving both clinician and nurse awareness, and providing real‐time risk assessment is clearly required to insure patient safety.[6, 13, 23, 24]

There are significant economic benefits to avoiding SH in the hospital given the adverse outcomes associated with HACs and the extra cost associated with these conditions. In hospitalized patients, hypoglycemia worsens outcomes leading to higher costs due to longer LOS (by 3 days), higher inpatient charges (38.9%), and higher risk of discharge to a skilled nursing facility.[1, 3, 25, 26] Conversely, improved glycemic control can reduce surgical site infections, perioperative morbidity, and hospital LOS.[27] The high prevalence of insulin use among inpatients, many of whom have high‐risk characteristics, creates a milieu for both hyper‐ and hypoglycemia. Other groups have described a drop in hypoglycemia rates related to the use of standardized diabetes order sets and nurse and physician education, but this is the first study that used informatics in a prospective manner to identify patients who are at high risk for developing hypoglycemia and then specifically targeted those patients.[28] The alert process was modeled after a similar alert developed in our institution for identifying medicine patients at risk for sepsis.[29] Given the paucity of data related to inpatient glycemia risk reduction, this study is particularly relevant for improving patient safety.

The major limitation of this study is that it was not randomized at the patient level. Patients were assigned to intervention and control groups based on their occupancy on specific hospital floors to avoid treatment bias. Bias was assessed due to this nonrandom assignment by comparing demographic and clinical factors of HR patients between intervention and control floors, and found significant differences in HbA1c and admitting diagnosis. As the control group had lower HbA1c values than the intervention group, and it is known from the Diabetes Control and Complications Trial and Action to Control Cardiovascular Risk in Diabetes trial that lower HbA1c increases the risk of hypoglycemia, our results may be biased by the level of glucose control on admission.[30, 31] Admitting diagnoses differed significantly between intervention and control patients as did the Charlson Comorbidity Index score; however, the hypoglycemia alert system does not include patient diagnoses or comorbidities, and as such provided equipoise with regard to risk reduction regardless of presenting illness. This study included trained nurses, which may be beyond the scope of every institution and thereby limit the effectiveness of the alert in reducing risk. However, as a result of this study, the alert was expanded to other acute care floors at our hospital as well as other hospitals in the Barnes‐Jewish Hospital system.

In summary, this study showed a 68% decrease in episodes of SH in a high‐risk patient cohort on diabetic medications using a hypoglycemia alert system. The results of this study demonstrate the validity of a systems‐based approach to reduce SH in high‐risk inpatients.

Insulin therapy in the hospital setting can cause hypoglycemia, which may lead to increased mortality and length of stay (LOS).[1, 2, 3] Hypoglycemia is associated with cardiovascular, cerebrovascular, and patient fall events.[4, 5] The Centers for Medicare and Medicaid Services have designated both severe hypoglycemia (SH) with harm and diabetic ketoacidosis as hospital acquired conditions (HAC) or never events. The Society for Hospital Medicine (SHM) defines SH in the hospital as a blood glucose (BG) <40 mg/dL. Minimizing episodes of SH is important for patient health outcomes, patient safety, and for healthcare facilities' safety metrics.

Many factors contribute to SH including excessive insulin doses, medication errors, inappropriate timing of insulin doses with food intake, changes in nutritional status, impaired renal function, and changes in medications such as steroids.[6] As part of a multiyear project in patient safety, an inpatient hypoglycemia alert algorithm was developed based on a multivariate analysis of individual patient demographic, pharmacy, laboratory, and glucometric data. The algorithm was previously shown to have a 75% sensitivity to predict episodes of SH.[7] In this study, we tested whether a predictive real‐time informatics hypoglycemia alert based on the tested algorithm, along with trained nurses, would result in a decreased frequency of SH events compared to usual care. We hypothesized that this alert would result in a reduction of SH events in those patients at high risk for hypoglycemia.

METHODS

Study Design and Population

This prospective cohort‐intervention study involved inpatients admitted to Barnes‐Jewish Hospital in St. Louis, Missouri, the academic hospital of Washington University School of Medicine (WUSM), from August 2011 through December 2011. Fourteen floors, including 10 internal medicine and 4 cardiology medicine floors, were selected based upon a high frequency of severe hypoglycemic events noted in 2010. Six of the internal medicine floors were designated as intervention floors, and 8 were designated as control floors, including the 4 cardiology units. The study population consisted of patients receiving diabetic medications on study floors who had a BG <90 mg/dL during their hospital stay (Figure 1). The study was approved by the WUSM institutional review board and included a waiver of consent for individual patients.

Figure 1

The pharmacy informatics system was programmed with the previously developed hypoglycemia alert to prospectively identify those patients at high risk of hypoglycemia based on real‐time patient information.[7] Patients were identified as high risk on study floors if insulin or an oral antihyperglycemic agent was prescribed and if their hypoglycemia informatics generated risk score was >35 within 24 hours of having a capillary or venous BG<90 mg/dL. The risk score of 35 corresponded to a 50% sensitivity for a subsequent BG <60 mg/dL and a 75% sensitivity for a BG <40 mg/dL. Patients who generated an alert once during their hospital stay were assigned to 1 of 3 categories based on their admission division and risk score: high‐risk intervention (HR‐I), high‐risk control (HR‐C), or low risk (LR). LR patients also had a BG <90 mg/dL during their stay, but a risk score of <35.

The electronic alert for HR‐I patients was sent by pager to division‐specific charge nurses. Fourteen charge nurses on intervention divisions were trained to assess the alert, interview the patient, identify an alternate dosing strategy, and collaborate with the patient's physicians. HR‐C patients were identified on control divisions based on the same criteria as intervention patients, but no alert was generated. Control patients' charts were reviewed and evaluated upon discharge by the research team‐certified diabetes nurse educator to determine whether the treating physician had identified the SH risk and had changed insulin orders.

Nurses and physicians caring for patients on study divisions provided informed consent to participate in the study. Nurses' satisfaction with the alert process and physician interaction was assessed with a collaboration scale that was completed after each alert (see Supporting Information, Appendix A, in the online version of this article).[8]

RESULTS

DISCUSSION

Hospitals are accountable for safe and effective care of patients with hyperglycemia, which includes prevention of medication‐induced hypoglycemia. We have developed a predictive informatics hypoglycemia risk alert that, when tested in a real‐world situation, significantly reduced the rate of SH in hospitalized patients without increasing severe hyperglycemia. The alert algorithm correctly identified patients who were at high risk for hypoglycemia and allowed caretakers the opportunity to lower that risk. The positive predictive value of the alert was low but acceptable at 9.7%, owing to the overall low rate of hypoglycemia in the patient population.

The alert model tested involved 3 components for success: the automated alert, trained charge nurse responders, and an interaction between the nurse responder and the care provider. HR‐I patients were interviewed and assessed for problems associated with oral intake, dietary habits, medication compliance, and hypoglycemia at home prior to communicating with physicians. The extensive training and proficiency in patient assessment and SBAR communication process required by nurses was paramount in the success of the alert. However, the alert provided a definitive risk assessment that was actionable, versus more global instruction, which has not had the same impact in risk reduction. Based on feedback collected from nurses at the study end, they felt the alert process was within their scope of practice and was not unduly burdensome. They also found that the training in diabetes management and SBAR communication techniques, in addition to the alert system, were useful in protecting patients from medication harm.

Physicians for HR‐C patients missed many opportunities to effectively intervene and thereby reduce the likelihood of an SH event. Our assumption is that the clinicians did not ascertain the risk of SH, which was reflected by the fact that orders were changed in 40.7% of HR‐I patients versus only 20.5% in the HR‐C group. Having alerts go directly to nurses rather than physicians permitted inclusion of additional information, such as caloric intake and testing schedules, so that changes in orders would have greater context, and the importance of mild hypoglycemia would not be overlooked.[16] Glycemic control is challenging for providers in the inpatient setting, as there is little time to test and titrate doses of insulin to achieve control. Tight glycemic control has become the primary focus of diabetes management in the outpatient setting to reduce long‐term risks of microvascular complications.[17, 18] However, establishing glycemic targets in the inpatient setting has been difficult because the risk for hypoglycemia increases with tighter control.[19, 20] Inpatient hypoglycemia has been associated with increased mortality, particularly in critically ill patients.[21, 22] Many factors contribute to hypoglycemia including low creatinine clearance, low body weight, untested insulin doses, errors in insulin administration, unexpected dietary changes, changes in medications affecting BG levels, poor communication during times of patient transfer to different care teams, and poor coordination of BG testing with insulin administration at meal times. A multifaceted approach aimed at improving both clinician and nurse awareness, and providing real‐time risk assessment is clearly required to insure patient safety.[6, 13, 23, 24]

There are significant economic benefits to avoiding SH in the hospital given the adverse outcomes associated with HACs and the extra cost associated with these conditions. In hospitalized patients, hypoglycemia worsens outcomes leading to higher costs due to longer LOS (by 3 days), higher inpatient charges (38.9%), and higher risk of discharge to a skilled nursing facility.[1, 3, 25, 26] Conversely, improved glycemic control can reduce surgical site infections, perioperative morbidity, and hospital LOS.[27] The high prevalence of insulin use among inpatients, many of whom have high‐risk characteristics, creates a milieu for both hyper‐ and hypoglycemia. Other groups have described a drop in hypoglycemia rates related to the use of standardized diabetes order sets and nurse and physician education, but this is the first study that used informatics in a prospective manner to identify patients who are at high risk for developing hypoglycemia and then specifically targeted those patients.[28] The alert process was modeled after a similar alert developed in our institution for identifying medicine patients at risk for sepsis.[29] Given the paucity of data related to inpatient glycemia risk reduction, this study is particularly relevant for improving patient safety.

The major limitation of this study is that it was not randomized at the patient level. Patients were assigned to intervention and control groups based on their occupancy on specific hospital floors to avoid treatment bias. Bias was assessed due to this nonrandom assignment by comparing demographic and clinical factors of HR patients between intervention and control floors, and found significant differences in HbA1c and admitting diagnosis. As the control group had lower HbA1c values than the intervention group, and it is known from the Diabetes Control and Complications Trial and Action to Control Cardiovascular Risk in Diabetes trial that lower HbA1c increases the risk of hypoglycemia, our results may be biased by the level of glucose control on admission.[30, 31] Admitting diagnoses differed significantly between intervention and control patients as did the Charlson Comorbidity Index score; however, the hypoglycemia alert system does not include patient diagnoses or comorbidities, and as such provided equipoise with regard to risk reduction regardless of presenting illness. This study included trained nurses, which may be beyond the scope of every institution and thereby limit the effectiveness of the alert in reducing risk. However, as a result of this study, the alert was expanded to other acute care floors at our hospital as well as other hospitals in the Barnes‐Jewish Hospital system.

In summary, this study showed a 68% decrease in episodes of SH in a high‐risk patient cohort on diabetic medications using a hypoglycemia alert system. The results of this study demonstrate the validity of a systems‐based approach to reduce SH in high‐risk inpatients.

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.

Patients deemed suitable for care on a general hospital unit are not expected to deteriorate; however, triage systems are not perfect, and some patients on general nursing units do develop critical illness during their hospitalization. Fortunately, there is mounting evidence that deteriorating patients exhibit measurable pathologic changes that could possibly be used to identify them prior to significant adverse outcomes, such as cardiac arrest.[1, 2, 3] Given the evidence that unplanned intensive care unit (ICU) transfers of patients on general units result in worse outcomes than more controlled ICU admissions,[1, 4, 5, 6] it is logical to assume that earlier identification of a deteriorating patient could provide a window of opportunity to prevent adverse outcomes.

The most commonly proposed systematic solution to the problem of identifying and stabilizing deteriorating patients on general hospital units includes some combination of an early warning system (EWS) to detect the deterioration and a rapid response team (RRT) to deal with it.[7, 8, 9, 10] We previously demonstrated that a relatively simple hospital‐specific method for generating EWS alerts derived from the electronic medical record (EMR) database is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general inpatient medicine units.[11, 12, 13, 14] However, our data also showed that simply providing the EWS alerts to these nursing units did not result in any demonstrable improvement in patient outcomes.[14] Therefore, we set out to determine whether linking real‐time EWS alerts to an intervention and notification of the RRT for patient evaluation could improve the outcomes of patients cared for on general inpatient units.

METHODS

RESULTS

Between January 15, 2013 and May 9, 2013, there were 4731 consecutive patients admitted to the 8 inpatient units and electronically screened as the base population for this investigation. Five hundred seventy‐one (12.1%) patients triggered an alert and were enrolled into the study (Figure 1). There were 286 patients assigned to the intervention group and 285 assigned to the control group. No patients were lost to follow‐up. Demographics, reason for hospital admission, and comorbidities of the 2 groups were similar (Table 1). The number of patients having a separate RRT call by the primary nursing team on the hospital units within 24 hours of generating an alert was greater for the intervention group but did not reach statistical significance (19.9% vs 16.5%; odds ratio: 1.260; 95% confidence interval [CI]: 0.8231.931). Table 2 provides the new diagnostic and therapeutic interventions initiated within 24 hours after a EWS alert was generated. Patients in the intervention group were significantly more likely to have their primary care team physician notified by an RRT nurse regarding medical condition issues and to have oximetry and telemetry started, whereas control patients were significantly more likely to have new antibiotic orders written within 24 hours of generating an alert.

Figure 1

Fifty‐one patients (17.8%) randomly assigned to the intervention group required ICU transfer compared with 52 of 285 patients (18.2%) in the control group (odds ratio: 0.972; 95% CI: 0.6351.490; P=0.898) (Table 3). Twenty‐one patients (7.3%) randomly assigned to the intervention group expired during their hospitalization compared with 22 of 285 patients (7.7%) in the control group (odds ratio: 0.947; 95%CI: 0.5091.764; P=0.865). Hospital length of stay was 8.49.5 days (median, 4.5 days; interquartile range, 2.311.4 days) for patients randomized to the intervention group and 9.411.1 days (median, 5.3 days; interquartile range, 3.211.2 days) for patients randomized to the control group (P=0.038). The ICU length of stay was 4.86.6 days (median, 2.9 days; interquartile range, 1.76.5 days) for patients randomized to the intervention group and 5.86.4 days (median, 2.9 days; interquartile range, 1.57.4) for patients randomized to the control group (P=0.812).The number of patients requiring transfer to a nursing home or long‐term acute care hospital was similar for patients in the intervention and control groups (26.9% vs 26.3%; odds ratio: 1.032; 95% CI: 0.7121.495; P=0.870). Similarly, the number of patients requiring hospital readmission before 30 days and 180 days, respectively, was similar for the 2 treatment groups (Table 3). For the combined study population, the EWS alerts were triggered 94138 hours (median, 27 hours; interquartile range, 7132 hours) prior to ICU transfer and 250204 hours (median200 hours; interquartile range, 54347 hours) prior to hospital mortality. The number of RRT calls for the 8 medicine units studied progressively increased from the start of the RRT program in 2009 through 2013 (121 in 2009, 194 in 2010, 298 in 2011, 415 in 2012, 415 in 2013; P<0.001 for the trend).

DISCUSSION

We demonstrated that a real‐time EWS alert sent to a RRT nurse was associated with a modest reduction in hospital length of stay, but similar rates of hospital mortality, ICU transfer, and subsequent need for placement in a long‐term care setting compared with usual care. We also found the number of RRT calls to have increased progressively from 2009 to the present on the study units examined.

Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[6] Bapoje et al. evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[19] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in an EWS.[20, 21] Organizations like the Institute for Healthcare Improvement have called for the development and routine implementation of EWSs to direct the activities of RRTs and improve outcomes.[22] However, a recent systematic review found that much of the evidence in support of EWSs and emergency response teams is of poor quality and lacking prospective randomized trials.[23]

Our earlier experience demonstrated that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our EWS.[14] Previous investigations have also had difficulty in demonstrating consistent outcome improvements with the use of EWSs and RRTs.[24, 25, 26, 27, 28, 29, 30, 31, 32] As a result of mandates from quality improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[33, 34] Linking RRT actions with a validated real‐time alert may represent a way of improving the overall effectiveness of such teams for monitoring general hospital units, short of having all hospitalized patients in units staffed and monitored to provide higher levels of supervision (eg, ICUs, step‐down units).[9, 35]

An alternative approach to preventing patient deterioration is to provide closer overall monitoring. This has been accomplished by employing nursing personnel to increase monitoring, or with the use of automated monitoring equipment. Bellomo et al. showed that the deployment of electronic automated vital sign monitors on general hospital units was associated with improved utilization of RRTs, increased patient survival, and decreased time for vital sign measurement and recording.[36] Laurens and Dwyer found that implementation of medical emergency teams (METs) to respond to predefined MET activation criteria as observed by hospital staff resulted in reduced hospital mortality and reduced need for ICU transfer.[37] However, other investigators have observed that imperfect implementation of nursing‐performed observational monitoring resulted in no demonstrable benefit, illustrating the limitations of this approach.[38] Our findings suggest that nursing care of patients on general hospital units may be enhanced with the use of an EWS alert sent to the RRT. This is supported by the observation that communications between the RRT and the primary care teams was greater as was the use of telemetry and oximetry in the intervention arm. Moreover, there appears to have been a learning effect for the nursing staff that occurred on our study units, as evidenced by the increased number of RRT calls that occurred between 2009 and 2013. This change in nursing practices on these units certainly made it more difficult for us to observe outcome differences in our current study with the prescribed intervention, reinforcing the notion that evaluating an already established practice is a difficult proposition.[39]

Our study has several important limitations. First, the EWS alert was developed and validated at Barnes‐Jewish Hospital.[11, 12, 13, 14] We cannot say whether this alert will perform similarly in another hospital. Second, the EWS alert only contains data from medical patients. Development and validation of EWS alerts for other hospitalized populations, including surgical and pediatric patients, are needed to make such systems more generalizable. Third, the primary clinical outcome employed for this trial was problematic. Transfer to an ICU may not be an optimal outcome variable, as it may be desirable to transfer alerted patients to an ICU, which can be perceived to represent a soft landing for such patients once an alert has been generated. A better measure could be 30‐day all‐cause mortality, which would not be subject to clinician biases. Finally, we could not specifically identify explanations for the greater use of antibiotics in the control group despite similar rates of infection for both study arms. Future studies should closely evaluate the ability of EWS alerts to alter specific therapies (eg, reduce antibiotic utilization).

In summary, we have demonstrated that an EWS alert linked to a RRT likely contributed to a modest reduction in hospital length of stay, but no reductions in hospital mortality and ICU transfer. These findings suggest that inpatient deterioration on general hospital units can be identified and linked to a specific intervention. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general hospital units but also intervene to improve their outcomes. We are moving forward with the development of a 2‐tiered EWS utilizing both EMR data and real‐time streamed vital sign data, to determine if we can further improve the prediction of clinical deterioration and potentially intervene in a more clinically meaningful manner.