Affiliations
Divsion of Hospital Medicine, University of California, San Francisco, San Francisco, California
Given name(s)
Anil N.
Family name
Makam
Degrees
MD, MAS

Incidence, predictors, and outcomes of hospital-acquired anemia

Article Type
Changed
Fri, 12/14/2018 - 08:21
Display Headline
Incidence, predictors, and outcomes of hospital-acquired anemia

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

METHODS

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

 

 

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Incidence of Adverse Outcomes by Severity of HAA
Figure

RESULTS

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Baseline Characteristics of Hospitalized Medicine Patients
Table 1

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

 

 

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).

Predictors of Developing Moderate or Severe HAA
Table 2

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

Association of HAA and 30-Day Postdischarge Adverse Outcomes
Table 3

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

DISCUSSION

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

Files
References

1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed

Article PDF
Issue
Journal of Hospital Medicine 12(5)
Publications
Topics
Page Number
317-322
Sections
Files
Files
Article PDF
Article PDF

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

METHODS

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

 

 

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Incidence of Adverse Outcomes by Severity of HAA
Figure

RESULTS

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Baseline Characteristics of Hospitalized Medicine Patients
Table 1

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

 

 

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).

Predictors of Developing Moderate or Severe HAA
Table 2

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

Association of HAA and 30-Day Postdischarge Adverse Outcomes
Table 3

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

DISCUSSION

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

METHODS

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

 

 

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Incidence of Adverse Outcomes by Severity of HAA
Figure

RESULTS

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Baseline Characteristics of Hospitalized Medicine Patients
Table 1

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

 

 

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).

Predictors of Developing Moderate or Severe HAA
Table 2

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

Association of HAA and 30-Day Postdischarge Adverse Outcomes
Table 3

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

DISCUSSION

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

References

1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed

References

1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed

Issue
Journal of Hospital Medicine 12(5)
Issue
Journal of Hospital Medicine 12(5)
Page Number
317-322
Page Number
317-322
Publications
Publications
Topics
Article Type
Display Headline
Incidence, predictors, and outcomes of hospital-acquired anemia
Display Headline
Incidence, predictors, and outcomes of hospital-acquired anemia
Sections
Article Source

© 2017 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Anil N. Makam, MD, MAS, 5323 Harry Hines Blvd., Dallas, TX 75390-9169; Telephone: 214-648-3272; Fax: 214-648-3232; E-mail: anil.makam@utsouthwestern.edu
Content Gating
Open Access (article Unlocked/Open Access)
Alternative CME
Use ProPublica
Article PDF Media
Media Files

Predicting 30-day pneumonia readmissions using electronic health record data

Article Type
Changed
Fri, 12/14/2018 - 08:24
Display Headline
Predicting 30-day pneumonia readmissions using electronic health record data

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.1 Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

METHODS

Study Design, Population, and Data Sources

 

 

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.20 For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),10,11,21,22 the pneumonia severity index (PSI),16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.25 For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.19 To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,14 we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,10 and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.16 Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.10 The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.26 The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.27 Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.27 This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.7 Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

 

 

RESULTS

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Baseline Characteristics of Patients Hospitalized with Pneumonia
Table 1

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

Final Pneumonia-Specific EHR Risk-Prediction Models for Readmissions
Table 2

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

Model Performance and Comparison of Pneumonia-Specific EHR Readmissions Models vs Other Models
Table 3

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Comparison of the calibration of different readmission models
Figure 1

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

DISCUSSION

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.19 With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).7 Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,28 based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.7

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,13,30 income,11,31 prior hospitalizations,11,13 thrombocytosis,32-34 and vital sign instabilities on discharge.17 Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Files
References

1. Centers for Disease Control and Prevention. Pneumonia. http://www.cdc.gov/nchs/fastats/pneumonia.htm. Accessed January 26, 2016.
2. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009;364(16):1582. PubMed
3. van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391-E402. PubMed
4. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433-440. PubMed
5. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520-528. PubMed
6. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality, US Department of Health and Human Services;March 2013. PubMed
7. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):998-1005. PubMed
8. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):1148-1154. PubMed
9. Hebert C, Shivade C, Foraker R, et al. Diagnosis-specific readmission risk prediction using electronic health data: a retrospective cohort study. BMC Med Inform Decis Mak. 2014;14:65. PubMed
10. Lindenauer PK, Normand SL, Drye EE, et al. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. J Hosp Med. 2011;6(3):142-150. PubMed
11. Mather JF, Fortunato GJ, Ash JL, Davis MJ, Kumar A. Prediction of pneumonia 30-day readmissions: a single-center attempt to increase model performance. Respir Care. 2014;59(2):199-208. PubMed
12. O’Brien WJ, Chen Q, Mull HJ, et al. What is the value of adding Medicare data in estimating VA hospital readmission rates? Health Serv Res. 2015;50(1):40-57. PubMed
13. Tang VL, Halm EA, Fine MJ, Johnson CS, Anzueto A, Mortensen EM. Predictors of rehospitalization after admission for pneumonia in the veterans affairs healthcare system. J Hosp Med. 2014;9(6):379-383. PubMed
14. Weinreich M, Nguyen OK, Wang D, et al. Predicting the risk of readmission in pneumonia: a systematic review of model performance. Ann Am Thorac Soc. 2016;13(9):1607-1614. PubMed
15. Kwok CS, Loke YK, Woo K, Myint PK. Risk prediction models for mortality in community-acquired pneumonia: a systematic review. Biomed Res Int. 2013;2013:504136. PubMed
16. Loke YK, Kwok CS, Niruban A, Myint PK. Value of severity scales in predicting mortality from community-acquired pneumonia: systematic review and meta-analysis. Thorax. 2010;65(10):884-890. PubMed
17. Halm EA, Fine MJ, Kapoor WN, Singer DE, Marrie TJ, Siu AL. Instability on hospital discharge and the risk of adverse outcomes in patients with pneumonia. Arch Intern Med. 2002;162(11):1278-1284. PubMed
18. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
19. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
20. Lindenauer PK, Lagu T, Shieh MS, Pekow PS, Rothberg MB. Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009. JAMA. 2012;307(13):1405-1413. PubMed
21. Ahmedani BK, Solberg LI, Copeland LA, et al. Psychiatric comorbidity and 30-day readmissions after hospitalization for heart failure, AMI, and pneumonia. Psychiatr Serv. 2015;66(2):134-140. PubMed
22. Jasti H, Mortensen EM, Obrosky DS, Kapoor WN, Fine MJ. Causes and risk factors for rehospitalization of patients hospitalized with community-acquired pneumonia. Clin Infect Dis. 2008;46(4):550-556. PubMed
23. Capelastegui A, España Yandiola PP, Quintana JM, et al. Predictors of short-term rehospitalization following discharge of patients hospitalized with community-acquired pneumonia. Chest. 2009;136(4):1079-1085. PubMed
24. Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243-250. PubMed
25. Vittinghoff E, Glidden D, Shiboski S, McCulloch C. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models (Statistics for Biology and Health). New York City, NY: Springer; 2012.
26. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-112. PubMed
27. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med. 2014;160(2):122-131. PubMed
28. Krumholz HM. Post-hospital syndrome--an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100-102. PubMed
29. Micek ST, Lang A, Fuller BM, Hampton NB, Kollef MH. Clinical implications for patients treated inappropriately for community-acquired pneumonia in the emergency department. BMC Infect Dis. 2014;14:61. PubMed
30. Metersky ML, Fine MJ, Mortensen EM. The effect of marital status on the presentation and outcomes of elderly male veterans hospitalized for pneumonia. Chest. 2012;142(4):982-987. PubMed
31. Calvillo-King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269-282. PubMed
32. Mirsaeidi M, Peyrani P, Aliberti S, et al. Thrombocytopenia and thrombocytosis at time of hospitalization predict mortality in patients with community-acquired pneumonia. Chest. 2010;137(2):416-420. PubMed
33. Prina E, Ferrer M, Ranzani OT, et al. Thrombocytosis is a marker of poor outcome in community-acquired pneumonia. Chest. 2013;143(3):767-775. PubMed

34. Violi F, Cangemi R, Calvieri C. Pneumonia, thrombosis and vascular disease. J Thromb Haemost. 2014;12(9):1391-1400. PubMed
35. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):1441-1447. PubMed
36. Field TS, Ogarek J, Garber L, Reed G, Gurwitz JH. Association of early post-discharge follow-up by a primary care physician and 30-day rehospitalization among older adults. J Gen Intern Med. 2015;30(5):565-571. PubMed
37. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862-869. PubMed
38. Brooke BS, Stone DH, Cronenwett JL, et al. Early primary care provider follow-up and readmission after high-risk surgery. JAMA Surg. 2014;149(8):821-828. PubMed
39. Adamuz J, Viasus D, Campreciós-Rodriguez P, et al. A prospective cohort study of healthcare visits and rehospitalizations after discharge of patients with community-acquired pneumonia. Respirology. 2011;16(7):1119-1126. PubMed
40. Shorr AF, Zilberberg MD, Reichley R, et al. Readmission following hospitalization for pneumonia: the impact of pneumonia type and its implication for hospitals. Clin Infect Dis. 2013;57(3):362-367. PubMed

Article PDF
Issue
Journal of Hospital Medicine 12(4)
Publications
Topics
Page Number
209-216
Sections
Files
Files
Article PDF
Article PDF

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.1 Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

METHODS

Study Design, Population, and Data Sources

 

 

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.20 For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),10,11,21,22 the pneumonia severity index (PSI),16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.25 For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.19 To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,14 we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,10 and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.16 Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.10 The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.26 The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.27 Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.27 This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.7 Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

 

 

RESULTS

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Baseline Characteristics of Patients Hospitalized with Pneumonia
Table 1

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

Final Pneumonia-Specific EHR Risk-Prediction Models for Readmissions
Table 2

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

Model Performance and Comparison of Pneumonia-Specific EHR Readmissions Models vs Other Models
Table 3

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Comparison of the calibration of different readmission models
Figure 1

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

DISCUSSION

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.19 With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).7 Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,28 based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.7

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,13,30 income,11,31 prior hospitalizations,11,13 thrombocytosis,32-34 and vital sign instabilities on discharge.17 Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.1 Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

METHODS

Study Design, Population, and Data Sources

 

 

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.20 For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),10,11,21,22 the pneumonia severity index (PSI),16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.25 For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.19 To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,14 we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,10 and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.16 Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.10 The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.26 The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.27 Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.27 This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.7 Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

 

 

RESULTS

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Baseline Characteristics of Patients Hospitalized with Pneumonia
Table 1

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

Final Pneumonia-Specific EHR Risk-Prediction Models for Readmissions
Table 2

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

Model Performance and Comparison of Pneumonia-Specific EHR Readmissions Models vs Other Models
Table 3

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Comparison of the calibration of different readmission models
Figure 1

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

DISCUSSION

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.19 With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).7 Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,28 based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.7

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,13,30 income,11,31 prior hospitalizations,11,13 thrombocytosis,32-34 and vital sign instabilities on discharge.17 Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

 

 

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

References

1. Centers for Disease Control and Prevention. Pneumonia. http://www.cdc.gov/nchs/fastats/pneumonia.htm. Accessed January 26, 2016.
2. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009;364(16):1582. PubMed
3. van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391-E402. PubMed
4. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433-440. PubMed
5. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520-528. PubMed
6. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality, US Department of Health and Human Services;March 2013. PubMed
7. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):998-1005. PubMed
8. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):1148-1154. PubMed
9. Hebert C, Shivade C, Foraker R, et al. Diagnosis-specific readmission risk prediction using electronic health data: a retrospective cohort study. BMC Med Inform Decis Mak. 2014;14:65. PubMed
10. Lindenauer PK, Normand SL, Drye EE, et al. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. J Hosp Med. 2011;6(3):142-150. PubMed
11. Mather JF, Fortunato GJ, Ash JL, Davis MJ, Kumar A. Prediction of pneumonia 30-day readmissions: a single-center attempt to increase model performance. Respir Care. 2014;59(2):199-208. PubMed
12. O’Brien WJ, Chen Q, Mull HJ, et al. What is the value of adding Medicare data in estimating VA hospital readmission rates? Health Serv Res. 2015;50(1):40-57. PubMed
13. Tang VL, Halm EA, Fine MJ, Johnson CS, Anzueto A, Mortensen EM. Predictors of rehospitalization after admission for pneumonia in the veterans affairs healthcare system. J Hosp Med. 2014;9(6):379-383. PubMed
14. Weinreich M, Nguyen OK, Wang D, et al. Predicting the risk of readmission in pneumonia: a systematic review of model performance. Ann Am Thorac Soc. 2016;13(9):1607-1614. PubMed
15. Kwok CS, Loke YK, Woo K, Myint PK. Risk prediction models for mortality in community-acquired pneumonia: a systematic review. Biomed Res Int. 2013;2013:504136. PubMed
16. Loke YK, Kwok CS, Niruban A, Myint PK. Value of severity scales in predicting mortality from community-acquired pneumonia: systematic review and meta-analysis. Thorax. 2010;65(10):884-890. PubMed
17. Halm EA, Fine MJ, Kapoor WN, Singer DE, Marrie TJ, Siu AL. Instability on hospital discharge and the risk of adverse outcomes in patients with pneumonia. Arch Intern Med. 2002;162(11):1278-1284. PubMed
18. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
19. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
20. Lindenauer PK, Lagu T, Shieh MS, Pekow PS, Rothberg MB. Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009. JAMA. 2012;307(13):1405-1413. PubMed
21. Ahmedani BK, Solberg LI, Copeland LA, et al. Psychiatric comorbidity and 30-day readmissions after hospitalization for heart failure, AMI, and pneumonia. Psychiatr Serv. 2015;66(2):134-140. PubMed
22. Jasti H, Mortensen EM, Obrosky DS, Kapoor WN, Fine MJ. Causes and risk factors for rehospitalization of patients hospitalized with community-acquired pneumonia. Clin Infect Dis. 2008;46(4):550-556. PubMed
23. Capelastegui A, España Yandiola PP, Quintana JM, et al. Predictors of short-term rehospitalization following discharge of patients hospitalized with community-acquired pneumonia. Chest. 2009;136(4):1079-1085. PubMed
24. Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243-250. PubMed
25. Vittinghoff E, Glidden D, Shiboski S, McCulloch C. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models (Statistics for Biology and Health). New York City, NY: Springer; 2012.
26. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-112. PubMed
27. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med. 2014;160(2):122-131. PubMed
28. Krumholz HM. Post-hospital syndrome--an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100-102. PubMed
29. Micek ST, Lang A, Fuller BM, Hampton NB, Kollef MH. Clinical implications for patients treated inappropriately for community-acquired pneumonia in the emergency department. BMC Infect Dis. 2014;14:61. PubMed
30. Metersky ML, Fine MJ, Mortensen EM. The effect of marital status on the presentation and outcomes of elderly male veterans hospitalized for pneumonia. Chest. 2012;142(4):982-987. PubMed
31. Calvillo-King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269-282. PubMed
32. Mirsaeidi M, Peyrani P, Aliberti S, et al. Thrombocytopenia and thrombocytosis at time of hospitalization predict mortality in patients with community-acquired pneumonia. Chest. 2010;137(2):416-420. PubMed
33. Prina E, Ferrer M, Ranzani OT, et al. Thrombocytosis is a marker of poor outcome in community-acquired pneumonia. Chest. 2013;143(3):767-775. PubMed

34. Violi F, Cangemi R, Calvieri C. Pneumonia, thrombosis and vascular disease. J Thromb Haemost. 2014;12(9):1391-1400. PubMed
35. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):1441-1447. PubMed
36. Field TS, Ogarek J, Garber L, Reed G, Gurwitz JH. Association of early post-discharge follow-up by a primary care physician and 30-day rehospitalization among older adults. J Gen Intern Med. 2015;30(5):565-571. PubMed
37. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862-869. PubMed
38. Brooke BS, Stone DH, Cronenwett JL, et al. Early primary care provider follow-up and readmission after high-risk surgery. JAMA Surg. 2014;149(8):821-828. PubMed
39. Adamuz J, Viasus D, Campreciós-Rodriguez P, et al. A prospective cohort study of healthcare visits and rehospitalizations after discharge of patients with community-acquired pneumonia. Respirology. 2011;16(7):1119-1126. PubMed
40. Shorr AF, Zilberberg MD, Reichley R, et al. Readmission following hospitalization for pneumonia: the impact of pneumonia type and its implication for hospitals. Clin Infect Dis. 2013;57(3):362-367. PubMed

References

1. Centers for Disease Control and Prevention. Pneumonia. http://www.cdc.gov/nchs/fastats/pneumonia.htm. Accessed January 26, 2016.
2. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009;364(16):1582. PubMed
3. van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391-E402. PubMed
4. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433-440. PubMed
5. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30-day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520-528. PubMed
6. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality, US Department of Health and Human Services;March 2013. PubMed
7. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real-time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):998-1005. PubMed
8. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):1148-1154. PubMed
9. Hebert C, Shivade C, Foraker R, et al. Diagnosis-specific readmission risk prediction using electronic health data: a retrospective cohort study. BMC Med Inform Decis Mak. 2014;14:65. PubMed
10. Lindenauer PK, Normand SL, Drye EE, et al. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. J Hosp Med. 2011;6(3):142-150. PubMed
11. Mather JF, Fortunato GJ, Ash JL, Davis MJ, Kumar A. Prediction of pneumonia 30-day readmissions: a single-center attempt to increase model performance. Respir Care. 2014;59(2):199-208. PubMed
12. O’Brien WJ, Chen Q, Mull HJ, et al. What is the value of adding Medicare data in estimating VA hospital readmission rates? Health Serv Res. 2015;50(1):40-57. PubMed
13. Tang VL, Halm EA, Fine MJ, Johnson CS, Anzueto A, Mortensen EM. Predictors of rehospitalization after admission for pneumonia in the veterans affairs healthcare system. J Hosp Med. 2014;9(6):379-383. PubMed
14. Weinreich M, Nguyen OK, Wang D, et al. Predicting the risk of readmission in pneumonia: a systematic review of model performance. Ann Am Thorac Soc. 2016;13(9):1607-1614. PubMed
15. Kwok CS, Loke YK, Woo K, Myint PK. Risk prediction models for mortality in community-acquired pneumonia: a systematic review. Biomed Res Int. 2013;2013:504136. PubMed
16. Loke YK, Kwok CS, Niruban A, Myint PK. Value of severity scales in predicting mortality from community-acquired pneumonia: systematic review and meta-analysis. Thorax. 2010;65(10):884-890. PubMed
17. Halm EA, Fine MJ, Kapoor WN, Singer DE, Marrie TJ, Siu AL. Instability on hospital discharge and the risk of adverse outcomes in patients with pneumonia. Arch Intern Med. 2002;162(11):1278-1284. PubMed
18. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
19. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
20. Lindenauer PK, Lagu T, Shieh MS, Pekow PS, Rothberg MB. Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003-2009. JAMA. 2012;307(13):1405-1413. PubMed
21. Ahmedani BK, Solberg LI, Copeland LA, et al. Psychiatric comorbidity and 30-day readmissions after hospitalization for heart failure, AMI, and pneumonia. Psychiatr Serv. 2015;66(2):134-140. PubMed
22. Jasti H, Mortensen EM, Obrosky DS, Kapoor WN, Fine MJ. Causes and risk factors for rehospitalization of patients hospitalized with community-acquired pneumonia. Clin Infect Dis. 2008;46(4):550-556. PubMed
23. Capelastegui A, España Yandiola PP, Quintana JM, et al. Predictors of short-term rehospitalization following discharge of patients hospitalized with community-acquired pneumonia. Chest. 2009;136(4):1079-1085. PubMed
24. Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243-250. PubMed
25. Vittinghoff E, Glidden D, Shiboski S, McCulloch C. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models (Statistics for Biology and Health). New York City, NY: Springer; 2012.
26. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-112. PubMed
27. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med. 2014;160(2):122-131. PubMed
28. Krumholz HM. Post-hospital syndrome--an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100-102. PubMed
29. Micek ST, Lang A, Fuller BM, Hampton NB, Kollef MH. Clinical implications for patients treated inappropriately for community-acquired pneumonia in the emergency department. BMC Infect Dis. 2014;14:61. PubMed
30. Metersky ML, Fine MJ, Mortensen EM. The effect of marital status on the presentation and outcomes of elderly male veterans hospitalized for pneumonia. Chest. 2012;142(4):982-987. PubMed
31. Calvillo-King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269-282. PubMed
32. Mirsaeidi M, Peyrani P, Aliberti S, et al. Thrombocytopenia and thrombocytosis at time of hospitalization predict mortality in patients with community-acquired pneumonia. Chest. 2010;137(2):416-420. PubMed
33. Prina E, Ferrer M, Ranzani OT, et al. Thrombocytosis is a marker of poor outcome in community-acquired pneumonia. Chest. 2013;143(3):767-775. PubMed

34. Violi F, Cangemi R, Calvieri C. Pneumonia, thrombosis and vascular disease. J Thromb Haemost. 2014;12(9):1391-1400. PubMed
35. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):1441-1447. PubMed
36. Field TS, Ogarek J, Garber L, Reed G, Gurwitz JH. Association of early post-discharge follow-up by a primary care physician and 30-day rehospitalization among older adults. J Gen Intern Med. 2015;30(5):565-571. PubMed
37. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862-869. PubMed
38. Brooke BS, Stone DH, Cronenwett JL, et al. Early primary care provider follow-up and readmission after high-risk surgery. JAMA Surg. 2014;149(8):821-828. PubMed
39. Adamuz J, Viasus D, Campreciós-Rodriguez P, et al. A prospective cohort study of healthcare visits and rehospitalizations after discharge of patients with community-acquired pneumonia. Respirology. 2011;16(7):1119-1126. PubMed
40. Shorr AF, Zilberberg MD, Reichley R, et al. Readmission following hospitalization for pneumonia: the impact of pneumonia type and its implication for hospitals. Clin Infect Dis. 2013;57(3):362-367. PubMed

Issue
Journal of Hospital Medicine 12(4)
Issue
Journal of Hospital Medicine 12(4)
Page Number
209-216
Page Number
209-216
Publications
Publications
Topics
Article Type
Display Headline
Predicting 30-day pneumonia readmissions using electronic health record data
Display Headline
Predicting 30-day pneumonia readmissions using electronic health record data
Sections
Article Source

© 2017 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Anil N. Makam, MD, MAS; 5323 Harry Hines Blvd., Dallas, TX, 75390-9169; Telephone: 214-648-3272; Fax: 214-648-3232; E-mail: anil.makam@utsouthwestern.edu
Content Gating
Open Access (article Unlocked/Open Access)
Alternative CME
Use ProPublica
Article PDF Media
Media Files

Sneak Peek: Journal of Hospital Medicine

Article Type
Changed
Fri, 09/14/2018 - 12:00
Predicting 30-day pneumonia readmissions using electronic health record data.

 

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

OBJECTIVE: To develop pneumonia-specific readmission risk-prediction models using EHR data from the first day and from the entire hospital stay (“full stay”).

DESIGN: Observational cohort study using backward-stepwise selection and cross validation.

SUBJECTS: Consecutive pneumonia hospitalizations from six diverse hospitals in north Texas from 2009 to 2010.

MEASURES: All-cause, nonelective, 30-day readmissions, ascertained from 75 regional hospitals.

 

 

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM
 

Publications
Topics
Sections
Predicting 30-day pneumonia readmissions using electronic health record data.
Predicting 30-day pneumonia readmissions using electronic health record data.

 

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

OBJECTIVE: To develop pneumonia-specific readmission risk-prediction models using EHR data from the first day and from the entire hospital stay (“full stay”).

DESIGN: Observational cohort study using backward-stepwise selection and cross validation.

SUBJECTS: Consecutive pneumonia hospitalizations from six diverse hospitals in north Texas from 2009 to 2010.

MEASURES: All-cause, nonelective, 30-day readmissions, ascertained from 75 regional hospitals.

 

 

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM
 

 

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

OBJECTIVE: To develop pneumonia-specific readmission risk-prediction models using EHR data from the first day and from the entire hospital stay (“full stay”).

DESIGN: Observational cohort study using backward-stepwise selection and cross validation.

SUBJECTS: Consecutive pneumonia hospitalizations from six diverse hospitals in north Texas from 2009 to 2010.

MEASURES: All-cause, nonelective, 30-day readmissions, ascertained from 75 regional hospitals.

 

 

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM
 

Publications
Publications
Topics
Article Type
Sections
Disallow All Ads
Content Gating
No Gating (article Unlocked/Free)
Alternative CME

Predicting Readmissions from EHR Data

Article Type
Changed
Mon, 01/02/2017 - 19:34
Display Headline
Predicting all‐cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison

Unplanned hospital readmissions are frequent, costly, and potentially avoidable.[1, 2] Due to major federal financial readmissions penalties targeting excessive 30‐day readmissions, there is increasing attention to implementing hospital‐initiated interventions to reduce readmissions.[3, 4] However, universal enrollment of all hospitalized patients into such programs may be too resource intensive for many hospitals.[5] To optimize efficiency and effectiveness, interventions should be targeted to individuals most likely to benefit.[6, 7] However, existing readmission risk‐prediction models have achieved only modest discrimination, have largely used administrative claims data not available until months after discharge, or are limited to only a subset of patients with Medicare or a specific clinical condition.[8, 9, 10, 11, 12, 13, 14] These limitations have precluded accurate identification of high‐risk individuals in an all‐payer general medical inpatient population to provide actionable information for intervention prior to discharge.

Approaches using electronic health record (EHR) data could allow early identification of high‐risk patients during the index hospitalization to enable initiation of interventions prior to discharge. To date, such strategies have relied largely on EHR data from the day of admission.[15, 16] However, given that variation in 30‐day readmission rates are thought to reflect the quality of in‐hospital care, incorporating EHR data from the entire hospital stay to reflect hospital care processes and clinical trajectory may more accurately identify at‐risk patients.[17, 18, 19, 20] Improved accuracy in risk prediction would help better target intervention efforts in the immediate postdischarge period, an interval characterized by heightened vulnerability for adverse events.[21]

To help hospitals target transitional care interventions more effectively to high‐risk individuals prior to discharge, we derived and validated a readmissions risk‐prediction model incorporating EHR data from the entire course of the index hospitalization, which we termed the full‐stay EHR model. We also compared the full‐stay EHR model performance to our group's previously derived prediction model based on EHR data on the day of admission, termed the first‐day EHR model, as well as to 2 other validated readmission models similarly intended to yield near real‐time risk predictions prior to or shortly after hospital discharge.[9, 10, 15]

METHODS

Study Design, Population, and Data Sources

We conducted an observational cohort study using EHR data from 6 hospitals in the DallasFort Worth metroplex between November 1, 2009 and October 30, 2010 using the same EHR system (Epic Systems Corp., Verona, WI). One site was a university‐affiliated safety net hospital; the remaining 5 sites were teaching and nonteaching community sites.

We included consecutive hospitalizations among adults 18 years old discharged alive from any medicine inpatient service. For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization, were transferred to another acute care facility, left against medical advice, or who died outside of the hospital within 30 days of discharge. For model derivation, we randomly split the sample into separate derivation (50%) and validation cohorts (50%).

Outcomes

The primary outcome was 30‐day hospital readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100‐mile radius of Dallas, ascertained from an all‐payer regional hospitalization database. Nonelective hospitalizations included all hospitalizations classified as a emergency, urgent, or trauma, and excluded those classified as elective as per the Centers for Medicare and Medicaid Services Claim Inpatient Admission Type Code definitions.

Predictor Variables for the Full‐Stay EHR Model

The full‐stay EHR model was iteratively developed from our group's previously derived and validated risk‐prediction model using EHR data available on admission (first‐day EHR model).[15] For the full‐stay EHR model, we included all predictor variables included in our published first‐day EHR model as candidate risk factors. Based on prior literature, we additionally expanded candidate predictors available on admission to include marital status (proxy for social isolation) and socioeconomic disadvantage (percent poverty, unemployment, median income, and educational attainment by zip code of residence as proxy measures of the social and built environment).[22, 23, 24, 25, 26, 27] We also expanded the ascertainment of prior hospitalization to include admissions at both the index hospital and any of 75 acute care hospitals from the same, separate all‐payer regional hospitalization database used to ascertain 30‐day readmissions.

Candidate predictors from the remainder of the hospital stay (ie, following the first 24 hours of admission) were included if they were: (1) available in the EHR of all participating hospitals, (2) routinely collected or available at the time of hospital discharge, and (3) plausible predictors of adverse outcomes based on prior literature and clinical expertise. These included length of stay, in‐hospital complications, transfer to an intensive or coronary care unit, blood transfusions, vital sign instabilities within 24 hours of discharge, select laboratory values at time of discharge, and disposition status. We also assessed trajectories of vital signs and selected laboratory values (defined as changes in these measures from admission to discharge).

Statistical Analysis

Model Derivation

Univariate relationships between readmission and each of the candidate predictors were assessed in the derivation cohort using a prespecified significance threshold of P 0.05. We included all factors from our previously derived and validated first‐day EHR model as candidate predictors.[15] Continuous laboratory and vital sign values at the time of discharge were categorized based on clinically meaningful cutoffs; predictors with missing values were assumed to be normal (<1% missing for each variable). Significant univariate candidate variables were entered in a multivariate logistic regression model using stepwise backward selection with a prespecified significance threshold of P 0.05. We performed several sensitivity analyses to confirm the robustness of our model. First, we alternately derived the full‐stay model using stepwise forward selection. Second, we forced in all significant variables from our first‐day EHR model, and entered the candidate variables from the remainder of the hospital stay using both stepwise backward and forward selection separately. Third, prespecified interactions between variables were evaluated for inclusion. Though final predictors varied slightly between the different approaches, discrimination of each model was similar to the model derived using our primary analytic approach (C statistics 0.01, data not shown).

Model Validation

We assessed model discrimination and calibration of the derived full‐stay EHR model using the validation cohort. Model discrimination was estimated by the C statistic. The C statistic represents the probability that, given 2 hospitalized individuals (1 who was readmitted and the other who was not), the model will predict a higher risk for the readmitted patient than for the nonreadmitted patient. Model calibration was assessed by comparing predicted to observed probabilities of readmission by quintiles of risk, and with the Hosmer‐Lemeshow goodness‐of‐fit test.

Comparison to Existing Models

We compared the full‐stay EHR model performance to 3 previously published models: our group's first‐day EHR model, and the LACE (includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year) and HOSPITAL (includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay) models, which were both derived to predict 30‐day readmissions among general medical inpatients and were intended to help clinicians identify high‐risk patients to target for discharge interventions.[9, 10, 15] We assessed each model's performance in our validation cohort, calculating the C statistic, integrated discrimination index (IDI), and net reclassification index (NRI) compared to the full‐stay model. IDI is a summary measure of both discrimination and reclassification, where more positive values suggest improvement in model performance in both these domains compared to a reference model.[28] The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.[29] The theoretical range of values is 2 to 2, with more positive values indicating improved net reclassification compared to a reference model. Here, we calculated a category‐based NRI to evaluate the performance of models in correctly classifying individuals with and without readmissions into the highest readmission risk quintile versus the lowest 4 risk quintiles compared to the full‐stay EHR model.[29] This prespecified cutoff is relevant for hospitals interested in identifying the highest‐risk individuals for targeted intervention.[6] Because some hospitals may be able to target a greater number of individuals for intervention, we performed a sensitivity analysis by assessing category‐based NRI for reclassification into the top 2 risk quintiles versus the lowest 3 risk quintiles and found no meaningful difference in our results (data not shown). Finally, we qualitatively assessed calibration of comparator models in our validation cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, TX). This study was approved by the UT Southwestern Medical Center institutional review board.

RESULTS

Overall, 32,922 index hospitalizations were included in our study cohort; 12.7% resulted in a 30‐day readmission (see Supporting Figure 1 in the online version of this article). Individuals had a mean age of 62 years and had diverse race/ethnicity and primary insurance status; half were female (Table 1). The study sample was randomly split into a derivation cohort (50%, n = 16,492) and validation cohort (50%, n = 16,430). Individuals in the derivation cohort with a 30‐day readmission had markedly different socioeconomic and clinical characteristics compared to those not readmitted (Table 1).

Baseline Characteristics and Candidate Variables for Risk‐Prediction Model
Entire Cohort, N = 32,922 Derivation Cohort, N = 16,492

No Readmission, N = 14,312

Readmission, N = 2,180

P Value
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit; IQR, interquartile range; SD, standard deviation. *20% poverty in zip code as per high poverty area US Census designation. Prior ED visit at site of index hospitalization within the past year. Prior hospitalization at any of 75 acute care hospitals in the North Texas region within the past year. Nonelective admission defined as hospitalization categorized as medical emergency, urgent, or trauma. ∥Calculated from diagnoses available within 1 year prior to index hospitalization. Conditions were considered complications if they were not listed as a principle diagnosis for hospitalization or as a previous diagnosis in the prior year. #On day of discharge or last known observation before discharge. Instabilities were defined as temperature 37.8C, heart rate >100 beats/minute, respiratory rate >24 breaths/minute, systolic blood pressure 90 mm Hg, or oxygen saturation <90%. **Discharges to nursing home, skilled nursing facility, or long‐term acute care hospital.

Demographic characteristics
Age, y, mean (SD) 62 (17.3) 61 (17.4) 64 (17.0) 0.001
Female, n (%) 17,715 (53.8) 7,694 (53.8) 1,163 (53.3) 0.72
Race/ethnicity 0.001
White 21,359 (64.9) 9,329 (65.2) 1,361 (62.4)
Black 5,964 (18.1) 2,520 (17.6) 434 (19.9)
Hispanic 4,452 (13.5) 1,931 (13.5) 338 (15.5)
Other 1,147 (3.5) 532 (3.7) 47 (2.2)
Marital status, n (%) 0.001
Single 8,076 (24.5) 3,516 (24.6) 514 (23.6)
Married 13,394 (40.7) 5,950 (41.6) 812 (37.3)
Separated/divorced 3,468 (10.5) 1,460 (10.2) 251 (11.5)
Widowed 4,487 (13.7) 1,868 (13.1) 388 (17.8)
Other 3,497 (10.6) 1,518 (10.6) 215 (9.9)
Primary payer, n (%) 0.001
Private 13,090 (39.8) 5,855 (40.9) 726 (33.3)
Medicare 13,015 (39.5) 5,597 (39.1) 987 (45.3)
Medicaid 2,204 (6.7) 852 (5.9) 242 (11.1)
Charity, self‐pay, or other 4,613 (14.0) 2,008 (14.0) 225 (10.3)
High‐poverty neighborhood, n (%)* 7,468 (22.7) 3,208 (22.4) 548 (25.1) 0.001
Utilization history
1 ED visits in past year, n (%) 9,299 (28.2) 3,793 (26.5) 823 (37.8) 0.001
1 hospitalizations in past year, n (%) 10,189 (30.9) 4,074 (28.5) 1,012 (46.4) 0.001
Clinical factors from first day of hospitalization
Nonelective admission, n (%) 27,818 (84.5) 11,960 (83.6) 1,960 (89.9) 0.001
Charlson Comorbidity Index, median (IQR)∥ 0 (01) 0 (00) 0 (03) 0.001
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 355 (1.1) 119 (0.8) 46 (2.1) 0.001
Albumin 23 g/dL 4,732 (14.4) 1,956 (13.7) 458 (21.0) 0.001
Aspartate aminotransferase >40 U/L 4,610 (14.0) 1,922 (13.4) 383 (17.6) 0.001
Creatine phosphokinase <60 g/L 3,728 (11.3) 1,536 (10.7) 330 (15.1) 0.001
Mean corpuscular volume >100 fL/red cell 1,346 (4.1) 537 (3.8) 134 (6.2) 0.001
Platelets <90 103/L 912 (2.8) 357 (2.5) 116 (5.3) 0.001
Platelets >350 103/L 3,332 (10.1) 1,433 (10.0) 283 (13.0) 0.001
Prothrombin time >35 seconds 248 (0.8) 90 (0.6) 35 (1.6) 0.001
Clinical factors from remainder of hospital stay
Length of stay, d, median (IQR) 4 (26) 4 (26) 5 (38) 0.001
ICU transfer after first 24 hours, n (%) 988 (3.0) 408 (2.9) 94 (4.3) 0.001
Hospital complications, n (%)
Clostridium difficile infection 119 (0.4) 44 (0.3) 24 (1.1) 0.001
Pressure ulcer 358 (1.1) 126 (0.9) 46 (2.1) 0.001
Venous thromboembolism 301 (0.9) 112 (0.8) 34 (1.6) 0.001
Respiratory failure 1,048 (3.2) 463 (3.2) 112 (5.1) 0.001
Central line‐associated bloodstream infection 22 (0.07) 6 (0.04) 5 (0.23) 0.005
Catheter‐associated urinary tract infection 47 (0.14) 20 (0.14) 6 (0.28) 0.15
Acute myocardial infarction 293 (0.9) 110 (0.8) 32 (1.5) 0.001
Pneumonia 1,754 (5.3) 719 (5.0) 154 (7.1) 0.001
Sepsis 853 (2.6) 368 (2.6) 73 (3.4) 0.04
Blood transfusion during hospitalization, n (%) 4,511 (13.7) 1,837 (12.8) 425 (19.5) 0.001
Laboratory abnormalities at discharge#
Blood urea nitrogen >20 mg/dL, n (%) 10,014 (30.4) 4,077 (28.5) 929 (42.6) 0.001
Sodium <135 mEq/L, n (%) 4,583 (13.9) 1,850 (12.9) 440 (20.2) 0.001
Hematocrit 27 3,104 (9.4) 1,231 (8.6) 287 (13.2) 0.001
1 vital sign instability at discharge, n (%)# 6,192 (18.8) 2,624 (18.3) 525 (24.1) 0.001
Discharge location, n (%) 0.001
Home 23,339 (70.9) 10,282 (71.8) 1,383 (63.4)
Home health 3,185 (9.7) 1,356 (9.5) 234 (10.7)
Postacute care** 5,990 (18.2) 2,496 (17.4) 549 (25.2)
Hospice 408 (1.2) 178 (1.2) 14 (0.6)

Derivation and Validation of the Full‐Stay EHR Model for 30‐Day Readmission

Our final model included 24 independent variables, including demographic characteristics, utilization history, clinical factors from the first day of admission, and clinical factors from the remainder of the hospital stay (Table 2). The strongest independent predictor of readmission was hospital‐acquired Clostridium difficile infection (adjusted odds ratio [AOR]: 2.03, 95% confidence interval [CI] 1.18‐3.48); other hospital‐acquired complications including pressure ulcers and venous thromboembolism were also significant predictors. Though having Medicaid was associated with increased odds of readmission (AOR: 1.55, 95% CI: 1.31‐1.83), other zip codelevel measures of socioeconomic disadvantage were not predictive and were not included in the final model. Being discharged to hospice was associated with markedly lower odds of readmission (AOR: 0.23, 95% CI: 0.13‐0.40).

Final Full‐Stay EHR Model Predicting 30‐Day Readmissions (Derivation Cohort, N = 16,492)
Odds Ratio (95% CI)
Univariate Multivariate*
  • NOTE: Abbreviations: CI, confidence interval; ED, emergency department. *Values shown reflect adjusted odds ratios and 95% CI for each factor after adjustment for all other factors listed in the table.

Demographic characteristics
Age, per 10 years 1.08 (1.051.11) 1.07 (1.041.10)
Medicaid 1.97 (1.702.29) 1.55 (1.311.83)
Widow 1.44 (1.281.63) 1.27 (1.111.45)
Utilization history
Prior ED visit, per visit 1.08 (1.061.10) 1.04 (1.021.06)
Prior hospitalization, per hospitalization 1.30 (1.271.34) 1.16 (1.121.20)
Hospital and clinical factors from first day of hospitalization
Nonelective admission 1.75 (1.512.03) 1.42 (1.221.65)
Charlson Comorbidity Index, per point 1.19 (1.171.21) 1.06 (1.041.09)
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 2.57 (1.823.62) 1.52 (1.052.21)
Albumin 23 g/dL 1.68 (1.501.88) 1.20 (1.061.36)
Aspartate aminotransferase >40 U/L 1.37 (1.221.55) 1.21 (1.061.38)
Creatine phosphokinase <60 g/L 1.48 (1.301.69) 1.28 (1.111.46)
Mean corpuscular volume >100 fL/red cell 1.68 (1.382.04) 1.32 (1.071.62)
Platelets <90 103/L 2.20 (1.772.72) 1.56 (1.231.97)
Platelets >350 103/L 1.34 (1.171.54) 1.24 (1.081.44)
Prothrombin time >35 seconds 2.58 (1.743.82) 1.92 (1.272.90)
Hospital and clinical factors from remainder of hospital stay
Length of stay, per day 1.08 (1.071.09) 1.06 (1.041.07)
Hospital complications
Clostridium difficile infection 3.61 (2.195.95) 2.03 (1.183.48)
Pressure ulcer 2.43 (1.733.41) 1.64 (1.152.34)
Venous thromboembolism 2.01 (1.362.96) 1.55 (1.032.32)
Laboratory abnormalities at discharge
Blood urea nitrogen >20 mg/dL 1.86 (1.702.04) 1.37 (1.241.52)
Sodium <135 mEq/L 1.70 (1.521.91) 1.34 (1.181.51)
Hematocrit 27 1.61 (1.401.85) 1.22 (1.051.41)
Vital sign instability at discharge, per instability 1.29 (1.201.40) 1.25 (1.151.36)
Discharged to hospice 0.51 (0.300.89) 0.23 (0.130.40)

In our validation cohort, the full‐stay EHR model had fair discrimination, with a C statistic of 0.69 (95% CI: 0.68‐0.70) (Table 3). The full‐stay EHR model was well calibrated across all quintiles of risk, with slight overestimation of predicted risk in the lowest and highest quintiles (Figure 1a) (see Supporting Table 5 in the online version of this article). It also effectively stratified individuals across a broad range of predicted readmission risk from 4.1% in the lowest decile to 36.5% in the highest decile (Table 3).

Comparison of the Discrimination and Reclassification of Different Readmission Models*
Model Name C‐Statistic (95% CI) IDI, % (95% CI) NRI (95% CI) Average Predicted Risk, %
Lowest Decile Highest Decile
  • NOTE: Abbreviations; CI, confidence interval; EHR, electronic health record; IDI, Integrated Discrimination Improvement; NRI, Net Reclassification Index. *All measures were assessed using the validation cohort (N = 16,430), except for estimating the C‐statistic for the derivation cohort. P value <0.001 for all pairwise comparisons of C‐statistic between full‐stay model and first‐day, LACE, and HOSPITAL models, respectively. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Full‐stay EHR model
Derivation cohort 0.72 (0.70 to 0.73) 4.1 36.5
Validation cohort 0.69 (0.68 to 0.70) [Reference] [Reference] 4.1 36.5
First‐day EHR model 0.67 (0.66 to 0.68) 1.2 (1.4 to 1.0) 0.020 (0.038 to 0.002) 5.8 31.9
LACE model 0.65 (0.64 to 0.66) 2.6 (2.9 to 2.3) 0.046 (0.067 to 0.024) 6.1 27.5
HOSPITAL model 0.64 (0.62 to 0.65) 3.2 (3.5 to 2.9) 0.058 (0.080 to 0.035) 6.7 26.6
Figure 1
Comparison of the calibration of different readmission models. Calibration graphs for full‐stay (a), first‐day (b), LACE (c), and HOSPITAL (d) models in the validation cohort. Each graph shows predicted probability compared to observed probability of readmission by quintiles of risk for each model. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Comparing the Performance of the Full‐Stay EHR Model to Other Models

The full‐stay EHR model had better discrimination compared to the first‐day EHR model and the LACE and HOSPITAL models, though the magnitude of improvement was modest (Table 3). The full‐stay EHR model also stratified individuals across a broader range of readmission risk, and was better able to discriminate and classify those in the highest quintile of risk from those in the lowest 4 quintiles of risk compared to other models as assessed by the IDI and NRI (Table 3) (see Supporting Tables 14 and Supporting Figure 2 in the online version of this article). In terms of model calibration, both the first‐day EHR and LACE models were also well calibrated, whereas the HOSPITAL model was less robust (Figure 1).

The diagnostic accuracy of the full‐stay EHR model in correctly predicting those in the highest quintile of risk was better than that of the first‐day, LACE, and HOSPITAL models, though overall improvements in the sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios were also modest (see Supporting Table 6 in the online version of this article).

DISCUSSION

In this study, we used clinically detailed EHR data from the entire hospitalization on 32,922 individuals treated in 6 diverse hospitals to develop an all‐payer, multicondition readmission risk‐prediction model. To our knowledge, this is the first 30‐day hospital readmission risk‐prediction model to use a comprehensive set of factors from EHR data from the entire hospital stay. Prior EHR‐based models have focused exclusively on data available on or prior to the first day of admission, which account for clinical severity on admission but do not account for factors uncovered during the inpatient stay that influence the chance of a postdischarge adverse outcome.[15, 30] We specifically assessed the prognostic impact of a comprehensive set of factors from the entire index hospitalization, including hospital‐acquired complications, clinical trajectory, and stability on discharge in predicting hospital readmissions. Our full‐stay EHR model had statistically better discrimination, calibration, and diagnostic accuracy than our existing all‐cause first‐day EHR model[15] and 2 previously published readmissions models that included more limited information from hospitalization (such as length of stay).[9, 10] However, although the more complicated full‐stay EHR model was statistically better than previously published models, we were surprised that the predictive performance was only modestly improved despite the inclusion of many additional clinically relevant prognostic factors.

Taken together, our study has several important implications. First, the added complexity and resource intensity of implementing a full‐stay EHR model yields only modestly improved readmission risk prediction. Thus, hospitals and healthcare systems interested in targeting their highest‐risk individuals for interventions to reduce 30‐day readmission should consider doing so within the first day of hospital admission. Our group's previously derived and validated first‐day EHR model, which used data only from the first day of admission, qualitatively performed nearly as well as the full‐stay EHR model.[15] Additionally, a recent study using only preadmission EHR data to predict 30‐day readmissions also achieved similar discrimination and diagnostic accuracy as our full‐stay model.[30]

Second, the field of readmissions risk‐prediction modeling may be reaching the maximum achievable model performance using data that are currently available in the EHR. Our limited ability to accurately predict all‐cause 30‐day readmission risk may reflect the influence of currently unmeasured patient, system, and community factors on readmissions.[31, 32, 33] Due to the constraints of data collected in the EHR, we were unable to include several patient‐level clinical characteristics associated with hospital readmission, including self‐perceived health status, functional impairment, and cognition.[33, 34, 35, 36] However, given their modest effect sizes (ORs ranging from 1.062.10), adequately measuring and including these risk factors in our model may not meaningfully improve model performance and diagnostic accuracy. Further, many social and behavioral patient‐level factors are also not consistently available in EHR data. Though we explored the role of several neighborhood‐level socioeconomic measuresincluding prevalence of poverty, median income, education, and unemploymentwe found that none were significantly associated with 30‐day readmissions. These particular measures may have been inadequate to characterize individual‐level social and behavioral factors, as several previous studies have demonstrated that patient‐level factors such as social support, substance abuse, and medication and visit adherence can influence readmission risk in heart failure and pneumonia.[11, 16, 22, 25] This underscores the need for more standardized routine collection of data across functional, social, and behavioral domains in clinical settings, as recently championed by the Institute of Medicine.[11, 37] Integrating data from outside the EHR on postdischarge health behaviors, self‐management, follow‐up care, recovery, and home environment may be another important but untapped strategy for further improving prediction of readmissions.[25, 38]

Third, a multicondition readmission risk‐prediction model may be a less effective strategy than more customized disease‐specific models for selected conditions associated with high 30‐day readmission rates. Our group's previously derived and internally validated models for heart failure and human immunodeficiency virus had superior discrimination compared to our full‐stay EHR model (C statistic of 0.72 for each).[11, 13] However, given differences in the included population and time periods studied, a head‐to‐head comparison of these different strategies is needed to assess differences in model performance and utility.

Our study had several strengths. To our knowledge, this is the first study to rigorously measure the additive influence of in‐hospital complications, clinical trajectory, and stability on discharge on the risk of 30‐day hospital readmission. Additionally, our study included a large, diverse study population that included all payers, all ages of adults, a mix of community, academic, and safety net hospitals, and individuals from a broad array of racial/ethnic and socioeconomic backgrounds.

Our results should be interpreted in light of several limitations. First, though we sought to represent a diverse group of hospitals, all study sites were located within north Texas and generalizability to other regions is uncertain. Second, our ascertainment of prior hospitalizations and readmissions was more inclusive than what could be typically accomplished in real time using only EHR data from a single clinical site. We performed a sensitivity analysis using only prior utilization data available within the EHR from the index hospital with no meaningful difference in our findings (data not shown). Additionally, a recent study found that 30‐day readmissions occur at the index hospital for over 75% of events, suggesting that 30‐day readmissions are fairly comprehensively captured even with only single‐site data.[39] Third, we were not able to include data on outpatient visits before or after the index hospitalization, which may influence the risk of readmission.[1, 40]

In conclusion, incorporating clinically granular EHR data from the entire course of hospitalization modestly improves prediction of 30‐day readmissions compared to models that only include information from the first 24 hours of hospital admission or models that use far fewer variables. However, given the limited improvement in prediction, our findings suggest that from the practical perspective of implementing real‐time models to identify those at highest risk for readmission, it may not be worth the added complexity of waiting until the end of a hospitalization to leverage additional data on hospital complications, and the trajectory of laboratory and vital sign values currently available in the EHR. Further improvement in prediction of readmissions will likely require accounting for psychosocial, functional, behavioral, and postdischarge factors not currently present in the inpatient EHR.

Disclosures: This study was presented at the Society of Hospital Medicine 2015 Annual Meeting in National Harbor, Maryland, and the Society of General Internal Medicine 2015 Annual Meeting in Toronto, Canada. This work was supported by the Agency for Healthcare Research and Qualityfunded UT Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01) and the Commonwealth Foundation (#20100323). Drs. Nguyen and Makam received funding from the UT Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). Dr. Halm was also supported in part by NIH/NCATS U54 RFA‐TR‐12‐006. The study sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

Files
References
  1. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee‐for‐service program. N Engl J Med. 2009;360(14):14181428.
  2. Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391E402.
  3. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital‐initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433440.
  4. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30‐day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520528.
  5. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  6. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real‐time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):9981005.
  7. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):11481154.
  8. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):16881698.
  9. Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551557.
  10. Donze J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30‐day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. 2013;173(8):632638.
  11. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  12. Singal AG, Rahimi RS, Clark C, et al. An automated model using electronic medical record data identifies patients with cirrhosis at high risk for readmission. Clin Gastroenterol Hepatol. 2013;11(10):13351341.e1331.
  13. Nijhawan AE, Clark C, Kaplan R, Moore B, Halm EA, Amarasingham R. An electronic medical record‐based model to predict 30‐day risk of readmission and death among HIV‐infected inpatients. J Acquir Immune Defic Syndr. 2012;61(3):349358.
  14. Horwitz LI, Partovian C, Lin Z, et al. Development and use of an administrative claims measure for profiling hospital‐wide performance on 30‐day unplanned readmission. Ann Intern Med. 2014;161(10 suppl):S66S75.
  15. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record‐based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15(1):39.
  16. Watson AJ, O'Rourke J, Jethwani K, et al. Linking electronic health record‐extracted psychosocial data in real‐time to risk of readmission for heart failure. Psychosomatics. 2011;52(4):319327.
  17. Ashton CM, Wray NP. A conceptual framework for the study of early readmission as an indicator of quality of care. Soc Sci Med. 1996;43(11):15331541.
  18. Dharmarajan K, Hsieh AF, Lin Z, et al. Hospital readmission performance and patterns of readmission: retrospective cohort study of Medicare admissions. BMJ. 2013;347:f6571.
  19. Cassel CK, Conway PH, Delbanco SF, Jha AK, Saunders RS, Lee TH. Getting more performance from performance measurement. N Engl J Med. 2014;371(23):21452147.
  20. Bradley EH, Sipsma H, Horwitz LI, et al. Hospital strategy uptake and reductions in unplanned readmission rates for patients with heart failure: a prospective study. J Gen Intern Med. 2015;30(5):605611.
  21. Krumholz HM. Post‐hospital syndrome—an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100102.
  22. Calvillo‐King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269282.
  23. Keyhani S, Myers LJ, Cheng E, Hebert P, Williams LS, Bravata DM. Effect of clinical and social risk factors on hospital profiling for stroke readmission: a cohort study. Ann Intern Med. 2014;161(11):775784.
  24. Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvantage and 30‐day rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161(11):765774.
  25. Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C. Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community‐dwelling Medicare beneficiaries. Gerontologist. 2008;48(4):495504.
  26. Hu J, Gonsahn MD, Nerenz DR. Socioeconomic status and readmissions: evidence from an urban teaching hospital. Health Aff (Millwood). 2014;33(5):778785.
  27. Nagasako EM, Reidhead M, Waterman B, Dunagan WC. Adding socioeconomic data to hospital readmissions calculations may produce more useful results. Health Aff (Millwood). 2014;33(5):786791.
  28. Pencina MJ, D'Agostino RB, D'Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157172; discussion 207–212.
  29. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med. 2014;160(2):122131.
  30. Shadmi E, Flaks‐Manov N, Hoshen M, Goldman O, Bitterman H, Balicer RD. Predicting 30‐day readmissions with preadmission electronic health record data. Med Care. 2015;53(3):283289.
  31. Kangovi S, Grande D. Hospital readmissions—not just a measure of quality. JAMA. 2011;306(16):17961797.
  32. Joynt KE, Jha AK. Thirty‐day readmissions—truth and consequences. N Engl J Med. 2012;366(15):13661369.
  33. Greysen SR, Stijacic Cenzer I, Auerbach AD, Covinsky KE. Functional impairment and hospital readmission in medicare seniors. JAMA Intern Med. 2015;175(4):559565.
  34. Holloway JJ, Thomas JW, Shapiro L. Clinical and sociodemographic risk factors for readmission of Medicare beneficiaries. Health Care Financ Rev. 1988;10(1):2736.
  35. Patel A, Parikh R, Howell EH, Hsich E, Landers SH, Gorodeski EZ. Mini‐cog performance: novel marker of post discharge risk among patients hospitalized for heart failure. Circ Heart Fail. 2015;8(1):816.
  36. Hoyer EH, Needham DM, Atanelov L, Knox B, Friedman M, Brotman DJ. Association of impaired functional status at hospital discharge and subsequent rehospitalization. J Hosp Med. 2014;9(5):277282.
  37. Adler NE, Stead WW. Patients in context—EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372(8):698701.
  38. Nguyen OK, Chan CV, Makam A, Stieglitz H, Amarasingham R. Envisioning a social‐health information exchange as a platform to support a patient‐centered medical neighborhood: a feasibility study. J Gen Intern Med. 2015;30(1):6067.
  39. Henke RM, Karaca Z, Lin H, Wier LM, Marder W, Wong HS. Patient factors contributing to variation in same‐hospital readmission rate. Med Care Res Review. 2015;72(3):338358.
  40. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):14411447.
Article PDF
Issue
Journal of Hospital Medicine - 11(7)
Publications
Page Number
473-480
Sections
Files
Files
Article PDF
Article PDF

Unplanned hospital readmissions are frequent, costly, and potentially avoidable.[1, 2] Due to major federal financial readmissions penalties targeting excessive 30‐day readmissions, there is increasing attention to implementing hospital‐initiated interventions to reduce readmissions.[3, 4] However, universal enrollment of all hospitalized patients into such programs may be too resource intensive for many hospitals.[5] To optimize efficiency and effectiveness, interventions should be targeted to individuals most likely to benefit.[6, 7] However, existing readmission risk‐prediction models have achieved only modest discrimination, have largely used administrative claims data not available until months after discharge, or are limited to only a subset of patients with Medicare or a specific clinical condition.[8, 9, 10, 11, 12, 13, 14] These limitations have precluded accurate identification of high‐risk individuals in an all‐payer general medical inpatient population to provide actionable information for intervention prior to discharge.

Approaches using electronic health record (EHR) data could allow early identification of high‐risk patients during the index hospitalization to enable initiation of interventions prior to discharge. To date, such strategies have relied largely on EHR data from the day of admission.[15, 16] However, given that variation in 30‐day readmission rates are thought to reflect the quality of in‐hospital care, incorporating EHR data from the entire hospital stay to reflect hospital care processes and clinical trajectory may more accurately identify at‐risk patients.[17, 18, 19, 20] Improved accuracy in risk prediction would help better target intervention efforts in the immediate postdischarge period, an interval characterized by heightened vulnerability for adverse events.[21]

To help hospitals target transitional care interventions more effectively to high‐risk individuals prior to discharge, we derived and validated a readmissions risk‐prediction model incorporating EHR data from the entire course of the index hospitalization, which we termed the full‐stay EHR model. We also compared the full‐stay EHR model performance to our group's previously derived prediction model based on EHR data on the day of admission, termed the first‐day EHR model, as well as to 2 other validated readmission models similarly intended to yield near real‐time risk predictions prior to or shortly after hospital discharge.[9, 10, 15]

METHODS

Study Design, Population, and Data Sources

We conducted an observational cohort study using EHR data from 6 hospitals in the DallasFort Worth metroplex between November 1, 2009 and October 30, 2010 using the same EHR system (Epic Systems Corp., Verona, WI). One site was a university‐affiliated safety net hospital; the remaining 5 sites were teaching and nonteaching community sites.

We included consecutive hospitalizations among adults 18 years old discharged alive from any medicine inpatient service. For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization, were transferred to another acute care facility, left against medical advice, or who died outside of the hospital within 30 days of discharge. For model derivation, we randomly split the sample into separate derivation (50%) and validation cohorts (50%).

Outcomes

The primary outcome was 30‐day hospital readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100‐mile radius of Dallas, ascertained from an all‐payer regional hospitalization database. Nonelective hospitalizations included all hospitalizations classified as a emergency, urgent, or trauma, and excluded those classified as elective as per the Centers for Medicare and Medicaid Services Claim Inpatient Admission Type Code definitions.

Predictor Variables for the Full‐Stay EHR Model

The full‐stay EHR model was iteratively developed from our group's previously derived and validated risk‐prediction model using EHR data available on admission (first‐day EHR model).[15] For the full‐stay EHR model, we included all predictor variables included in our published first‐day EHR model as candidate risk factors. Based on prior literature, we additionally expanded candidate predictors available on admission to include marital status (proxy for social isolation) and socioeconomic disadvantage (percent poverty, unemployment, median income, and educational attainment by zip code of residence as proxy measures of the social and built environment).[22, 23, 24, 25, 26, 27] We also expanded the ascertainment of prior hospitalization to include admissions at both the index hospital and any of 75 acute care hospitals from the same, separate all‐payer regional hospitalization database used to ascertain 30‐day readmissions.

Candidate predictors from the remainder of the hospital stay (ie, following the first 24 hours of admission) were included if they were: (1) available in the EHR of all participating hospitals, (2) routinely collected or available at the time of hospital discharge, and (3) plausible predictors of adverse outcomes based on prior literature and clinical expertise. These included length of stay, in‐hospital complications, transfer to an intensive or coronary care unit, blood transfusions, vital sign instabilities within 24 hours of discharge, select laboratory values at time of discharge, and disposition status. We also assessed trajectories of vital signs and selected laboratory values (defined as changes in these measures from admission to discharge).

Statistical Analysis

Model Derivation

Univariate relationships between readmission and each of the candidate predictors were assessed in the derivation cohort using a prespecified significance threshold of P 0.05. We included all factors from our previously derived and validated first‐day EHR model as candidate predictors.[15] Continuous laboratory and vital sign values at the time of discharge were categorized based on clinically meaningful cutoffs; predictors with missing values were assumed to be normal (<1% missing for each variable). Significant univariate candidate variables were entered in a multivariate logistic regression model using stepwise backward selection with a prespecified significance threshold of P 0.05. We performed several sensitivity analyses to confirm the robustness of our model. First, we alternately derived the full‐stay model using stepwise forward selection. Second, we forced in all significant variables from our first‐day EHR model, and entered the candidate variables from the remainder of the hospital stay using both stepwise backward and forward selection separately. Third, prespecified interactions between variables were evaluated for inclusion. Though final predictors varied slightly between the different approaches, discrimination of each model was similar to the model derived using our primary analytic approach (C statistics 0.01, data not shown).

Model Validation

We assessed model discrimination and calibration of the derived full‐stay EHR model using the validation cohort. Model discrimination was estimated by the C statistic. The C statistic represents the probability that, given 2 hospitalized individuals (1 who was readmitted and the other who was not), the model will predict a higher risk for the readmitted patient than for the nonreadmitted patient. Model calibration was assessed by comparing predicted to observed probabilities of readmission by quintiles of risk, and with the Hosmer‐Lemeshow goodness‐of‐fit test.

Comparison to Existing Models

We compared the full‐stay EHR model performance to 3 previously published models: our group's first‐day EHR model, and the LACE (includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year) and HOSPITAL (includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay) models, which were both derived to predict 30‐day readmissions among general medical inpatients and were intended to help clinicians identify high‐risk patients to target for discharge interventions.[9, 10, 15] We assessed each model's performance in our validation cohort, calculating the C statistic, integrated discrimination index (IDI), and net reclassification index (NRI) compared to the full‐stay model. IDI is a summary measure of both discrimination and reclassification, where more positive values suggest improvement in model performance in both these domains compared to a reference model.[28] The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.[29] The theoretical range of values is 2 to 2, with more positive values indicating improved net reclassification compared to a reference model. Here, we calculated a category‐based NRI to evaluate the performance of models in correctly classifying individuals with and without readmissions into the highest readmission risk quintile versus the lowest 4 risk quintiles compared to the full‐stay EHR model.[29] This prespecified cutoff is relevant for hospitals interested in identifying the highest‐risk individuals for targeted intervention.[6] Because some hospitals may be able to target a greater number of individuals for intervention, we performed a sensitivity analysis by assessing category‐based NRI for reclassification into the top 2 risk quintiles versus the lowest 3 risk quintiles and found no meaningful difference in our results (data not shown). Finally, we qualitatively assessed calibration of comparator models in our validation cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, TX). This study was approved by the UT Southwestern Medical Center institutional review board.

RESULTS

Overall, 32,922 index hospitalizations were included in our study cohort; 12.7% resulted in a 30‐day readmission (see Supporting Figure 1 in the online version of this article). Individuals had a mean age of 62 years and had diverse race/ethnicity and primary insurance status; half were female (Table 1). The study sample was randomly split into a derivation cohort (50%, n = 16,492) and validation cohort (50%, n = 16,430). Individuals in the derivation cohort with a 30‐day readmission had markedly different socioeconomic and clinical characteristics compared to those not readmitted (Table 1).

Baseline Characteristics and Candidate Variables for Risk‐Prediction Model
Entire Cohort, N = 32,922 Derivation Cohort, N = 16,492

No Readmission, N = 14,312

Readmission, N = 2,180

P Value
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit; IQR, interquartile range; SD, standard deviation. *20% poverty in zip code as per high poverty area US Census designation. Prior ED visit at site of index hospitalization within the past year. Prior hospitalization at any of 75 acute care hospitals in the North Texas region within the past year. Nonelective admission defined as hospitalization categorized as medical emergency, urgent, or trauma. ∥Calculated from diagnoses available within 1 year prior to index hospitalization. Conditions were considered complications if they were not listed as a principle diagnosis for hospitalization or as a previous diagnosis in the prior year. #On day of discharge or last known observation before discharge. Instabilities were defined as temperature 37.8C, heart rate >100 beats/minute, respiratory rate >24 breaths/minute, systolic blood pressure 90 mm Hg, or oxygen saturation <90%. **Discharges to nursing home, skilled nursing facility, or long‐term acute care hospital.

Demographic characteristics
Age, y, mean (SD) 62 (17.3) 61 (17.4) 64 (17.0) 0.001
Female, n (%) 17,715 (53.8) 7,694 (53.8) 1,163 (53.3) 0.72
Race/ethnicity 0.001
White 21,359 (64.9) 9,329 (65.2) 1,361 (62.4)
Black 5,964 (18.1) 2,520 (17.6) 434 (19.9)
Hispanic 4,452 (13.5) 1,931 (13.5) 338 (15.5)
Other 1,147 (3.5) 532 (3.7) 47 (2.2)
Marital status, n (%) 0.001
Single 8,076 (24.5) 3,516 (24.6) 514 (23.6)
Married 13,394 (40.7) 5,950 (41.6) 812 (37.3)
Separated/divorced 3,468 (10.5) 1,460 (10.2) 251 (11.5)
Widowed 4,487 (13.7) 1,868 (13.1) 388 (17.8)
Other 3,497 (10.6) 1,518 (10.6) 215 (9.9)
Primary payer, n (%) 0.001
Private 13,090 (39.8) 5,855 (40.9) 726 (33.3)
Medicare 13,015 (39.5) 5,597 (39.1) 987 (45.3)
Medicaid 2,204 (6.7) 852 (5.9) 242 (11.1)
Charity, self‐pay, or other 4,613 (14.0) 2,008 (14.0) 225 (10.3)
High‐poverty neighborhood, n (%)* 7,468 (22.7) 3,208 (22.4) 548 (25.1) 0.001
Utilization history
1 ED visits in past year, n (%) 9,299 (28.2) 3,793 (26.5) 823 (37.8) 0.001
1 hospitalizations in past year, n (%) 10,189 (30.9) 4,074 (28.5) 1,012 (46.4) 0.001
Clinical factors from first day of hospitalization
Nonelective admission, n (%) 27,818 (84.5) 11,960 (83.6) 1,960 (89.9) 0.001
Charlson Comorbidity Index, median (IQR)∥ 0 (01) 0 (00) 0 (03) 0.001
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 355 (1.1) 119 (0.8) 46 (2.1) 0.001
Albumin 23 g/dL 4,732 (14.4) 1,956 (13.7) 458 (21.0) 0.001
Aspartate aminotransferase >40 U/L 4,610 (14.0) 1,922 (13.4) 383 (17.6) 0.001
Creatine phosphokinase <60 g/L 3,728 (11.3) 1,536 (10.7) 330 (15.1) 0.001
Mean corpuscular volume >100 fL/red cell 1,346 (4.1) 537 (3.8) 134 (6.2) 0.001
Platelets <90 103/L 912 (2.8) 357 (2.5) 116 (5.3) 0.001
Platelets >350 103/L 3,332 (10.1) 1,433 (10.0) 283 (13.0) 0.001
Prothrombin time >35 seconds 248 (0.8) 90 (0.6) 35 (1.6) 0.001
Clinical factors from remainder of hospital stay
Length of stay, d, median (IQR) 4 (26) 4 (26) 5 (38) 0.001
ICU transfer after first 24 hours, n (%) 988 (3.0) 408 (2.9) 94 (4.3) 0.001
Hospital complications, n (%)
Clostridium difficile infection 119 (0.4) 44 (0.3) 24 (1.1) 0.001
Pressure ulcer 358 (1.1) 126 (0.9) 46 (2.1) 0.001
Venous thromboembolism 301 (0.9) 112 (0.8) 34 (1.6) 0.001
Respiratory failure 1,048 (3.2) 463 (3.2) 112 (5.1) 0.001
Central line‐associated bloodstream infection 22 (0.07) 6 (0.04) 5 (0.23) 0.005
Catheter‐associated urinary tract infection 47 (0.14) 20 (0.14) 6 (0.28) 0.15
Acute myocardial infarction 293 (0.9) 110 (0.8) 32 (1.5) 0.001
Pneumonia 1,754 (5.3) 719 (5.0) 154 (7.1) 0.001
Sepsis 853 (2.6) 368 (2.6) 73 (3.4) 0.04
Blood transfusion during hospitalization, n (%) 4,511 (13.7) 1,837 (12.8) 425 (19.5) 0.001
Laboratory abnormalities at discharge#
Blood urea nitrogen >20 mg/dL, n (%) 10,014 (30.4) 4,077 (28.5) 929 (42.6) 0.001
Sodium <135 mEq/L, n (%) 4,583 (13.9) 1,850 (12.9) 440 (20.2) 0.001
Hematocrit 27 3,104 (9.4) 1,231 (8.6) 287 (13.2) 0.001
1 vital sign instability at discharge, n (%)# 6,192 (18.8) 2,624 (18.3) 525 (24.1) 0.001
Discharge location, n (%) 0.001
Home 23,339 (70.9) 10,282 (71.8) 1,383 (63.4)
Home health 3,185 (9.7) 1,356 (9.5) 234 (10.7)
Postacute care** 5,990 (18.2) 2,496 (17.4) 549 (25.2)
Hospice 408 (1.2) 178 (1.2) 14 (0.6)

Derivation and Validation of the Full‐Stay EHR Model for 30‐Day Readmission

Our final model included 24 independent variables, including demographic characteristics, utilization history, clinical factors from the first day of admission, and clinical factors from the remainder of the hospital stay (Table 2). The strongest independent predictor of readmission was hospital‐acquired Clostridium difficile infection (adjusted odds ratio [AOR]: 2.03, 95% confidence interval [CI] 1.18‐3.48); other hospital‐acquired complications including pressure ulcers and venous thromboembolism were also significant predictors. Though having Medicaid was associated with increased odds of readmission (AOR: 1.55, 95% CI: 1.31‐1.83), other zip codelevel measures of socioeconomic disadvantage were not predictive and were not included in the final model. Being discharged to hospice was associated with markedly lower odds of readmission (AOR: 0.23, 95% CI: 0.13‐0.40).

Final Full‐Stay EHR Model Predicting 30‐Day Readmissions (Derivation Cohort, N = 16,492)
Odds Ratio (95% CI)
Univariate Multivariate*
  • NOTE: Abbreviations: CI, confidence interval; ED, emergency department. *Values shown reflect adjusted odds ratios and 95% CI for each factor after adjustment for all other factors listed in the table.

Demographic characteristics
Age, per 10 years 1.08 (1.051.11) 1.07 (1.041.10)
Medicaid 1.97 (1.702.29) 1.55 (1.311.83)
Widow 1.44 (1.281.63) 1.27 (1.111.45)
Utilization history
Prior ED visit, per visit 1.08 (1.061.10) 1.04 (1.021.06)
Prior hospitalization, per hospitalization 1.30 (1.271.34) 1.16 (1.121.20)
Hospital and clinical factors from first day of hospitalization
Nonelective admission 1.75 (1.512.03) 1.42 (1.221.65)
Charlson Comorbidity Index, per point 1.19 (1.171.21) 1.06 (1.041.09)
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 2.57 (1.823.62) 1.52 (1.052.21)
Albumin 23 g/dL 1.68 (1.501.88) 1.20 (1.061.36)
Aspartate aminotransferase >40 U/L 1.37 (1.221.55) 1.21 (1.061.38)
Creatine phosphokinase <60 g/L 1.48 (1.301.69) 1.28 (1.111.46)
Mean corpuscular volume >100 fL/red cell 1.68 (1.382.04) 1.32 (1.071.62)
Platelets <90 103/L 2.20 (1.772.72) 1.56 (1.231.97)
Platelets >350 103/L 1.34 (1.171.54) 1.24 (1.081.44)
Prothrombin time >35 seconds 2.58 (1.743.82) 1.92 (1.272.90)
Hospital and clinical factors from remainder of hospital stay
Length of stay, per day 1.08 (1.071.09) 1.06 (1.041.07)
Hospital complications
Clostridium difficile infection 3.61 (2.195.95) 2.03 (1.183.48)
Pressure ulcer 2.43 (1.733.41) 1.64 (1.152.34)
Venous thromboembolism 2.01 (1.362.96) 1.55 (1.032.32)
Laboratory abnormalities at discharge
Blood urea nitrogen >20 mg/dL 1.86 (1.702.04) 1.37 (1.241.52)
Sodium <135 mEq/L 1.70 (1.521.91) 1.34 (1.181.51)
Hematocrit 27 1.61 (1.401.85) 1.22 (1.051.41)
Vital sign instability at discharge, per instability 1.29 (1.201.40) 1.25 (1.151.36)
Discharged to hospice 0.51 (0.300.89) 0.23 (0.130.40)

In our validation cohort, the full‐stay EHR model had fair discrimination, with a C statistic of 0.69 (95% CI: 0.68‐0.70) (Table 3). The full‐stay EHR model was well calibrated across all quintiles of risk, with slight overestimation of predicted risk in the lowest and highest quintiles (Figure 1a) (see Supporting Table 5 in the online version of this article). It also effectively stratified individuals across a broad range of predicted readmission risk from 4.1% in the lowest decile to 36.5% in the highest decile (Table 3).

Comparison of the Discrimination and Reclassification of Different Readmission Models*
Model Name C‐Statistic (95% CI) IDI, % (95% CI) NRI (95% CI) Average Predicted Risk, %
Lowest Decile Highest Decile
  • NOTE: Abbreviations; CI, confidence interval; EHR, electronic health record; IDI, Integrated Discrimination Improvement; NRI, Net Reclassification Index. *All measures were assessed using the validation cohort (N = 16,430), except for estimating the C‐statistic for the derivation cohort. P value <0.001 for all pairwise comparisons of C‐statistic between full‐stay model and first‐day, LACE, and HOSPITAL models, respectively. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Full‐stay EHR model
Derivation cohort 0.72 (0.70 to 0.73) 4.1 36.5
Validation cohort 0.69 (0.68 to 0.70) [Reference] [Reference] 4.1 36.5
First‐day EHR model 0.67 (0.66 to 0.68) 1.2 (1.4 to 1.0) 0.020 (0.038 to 0.002) 5.8 31.9
LACE model 0.65 (0.64 to 0.66) 2.6 (2.9 to 2.3) 0.046 (0.067 to 0.024) 6.1 27.5
HOSPITAL model 0.64 (0.62 to 0.65) 3.2 (3.5 to 2.9) 0.058 (0.080 to 0.035) 6.7 26.6
Figure 1
Comparison of the calibration of different readmission models. Calibration graphs for full‐stay (a), first‐day (b), LACE (c), and HOSPITAL (d) models in the validation cohort. Each graph shows predicted probability compared to observed probability of readmission by quintiles of risk for each model. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Comparing the Performance of the Full‐Stay EHR Model to Other Models

The full‐stay EHR model had better discrimination compared to the first‐day EHR model and the LACE and HOSPITAL models, though the magnitude of improvement was modest (Table 3). The full‐stay EHR model also stratified individuals across a broader range of readmission risk, and was better able to discriminate and classify those in the highest quintile of risk from those in the lowest 4 quintiles of risk compared to other models as assessed by the IDI and NRI (Table 3) (see Supporting Tables 14 and Supporting Figure 2 in the online version of this article). In terms of model calibration, both the first‐day EHR and LACE models were also well calibrated, whereas the HOSPITAL model was less robust (Figure 1).

The diagnostic accuracy of the full‐stay EHR model in correctly predicting those in the highest quintile of risk was better than that of the first‐day, LACE, and HOSPITAL models, though overall improvements in the sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios were also modest (see Supporting Table 6 in the online version of this article).

DISCUSSION

In this study, we used clinically detailed EHR data from the entire hospitalization on 32,922 individuals treated in 6 diverse hospitals to develop an all‐payer, multicondition readmission risk‐prediction model. To our knowledge, this is the first 30‐day hospital readmission risk‐prediction model to use a comprehensive set of factors from EHR data from the entire hospital stay. Prior EHR‐based models have focused exclusively on data available on or prior to the first day of admission, which account for clinical severity on admission but do not account for factors uncovered during the inpatient stay that influence the chance of a postdischarge adverse outcome.[15, 30] We specifically assessed the prognostic impact of a comprehensive set of factors from the entire index hospitalization, including hospital‐acquired complications, clinical trajectory, and stability on discharge in predicting hospital readmissions. Our full‐stay EHR model had statistically better discrimination, calibration, and diagnostic accuracy than our existing all‐cause first‐day EHR model[15] and 2 previously published readmissions models that included more limited information from hospitalization (such as length of stay).[9, 10] However, although the more complicated full‐stay EHR model was statistically better than previously published models, we were surprised that the predictive performance was only modestly improved despite the inclusion of many additional clinically relevant prognostic factors.

Taken together, our study has several important implications. First, the added complexity and resource intensity of implementing a full‐stay EHR model yields only modestly improved readmission risk prediction. Thus, hospitals and healthcare systems interested in targeting their highest‐risk individuals for interventions to reduce 30‐day readmission should consider doing so within the first day of hospital admission. Our group's previously derived and validated first‐day EHR model, which used data only from the first day of admission, qualitatively performed nearly as well as the full‐stay EHR model.[15] Additionally, a recent study using only preadmission EHR data to predict 30‐day readmissions also achieved similar discrimination and diagnostic accuracy as our full‐stay model.[30]

Second, the field of readmissions risk‐prediction modeling may be reaching the maximum achievable model performance using data that are currently available in the EHR. Our limited ability to accurately predict all‐cause 30‐day readmission risk may reflect the influence of currently unmeasured patient, system, and community factors on readmissions.[31, 32, 33] Due to the constraints of data collected in the EHR, we were unable to include several patient‐level clinical characteristics associated with hospital readmission, including self‐perceived health status, functional impairment, and cognition.[33, 34, 35, 36] However, given their modest effect sizes (ORs ranging from 1.062.10), adequately measuring and including these risk factors in our model may not meaningfully improve model performance and diagnostic accuracy. Further, many social and behavioral patient‐level factors are also not consistently available in EHR data. Though we explored the role of several neighborhood‐level socioeconomic measuresincluding prevalence of poverty, median income, education, and unemploymentwe found that none were significantly associated with 30‐day readmissions. These particular measures may have been inadequate to characterize individual‐level social and behavioral factors, as several previous studies have demonstrated that patient‐level factors such as social support, substance abuse, and medication and visit adherence can influence readmission risk in heart failure and pneumonia.[11, 16, 22, 25] This underscores the need for more standardized routine collection of data across functional, social, and behavioral domains in clinical settings, as recently championed by the Institute of Medicine.[11, 37] Integrating data from outside the EHR on postdischarge health behaviors, self‐management, follow‐up care, recovery, and home environment may be another important but untapped strategy for further improving prediction of readmissions.[25, 38]

Third, a multicondition readmission risk‐prediction model may be a less effective strategy than more customized disease‐specific models for selected conditions associated with high 30‐day readmission rates. Our group's previously derived and internally validated models for heart failure and human immunodeficiency virus had superior discrimination compared to our full‐stay EHR model (C statistic of 0.72 for each).[11, 13] However, given differences in the included population and time periods studied, a head‐to‐head comparison of these different strategies is needed to assess differences in model performance and utility.

Our study had several strengths. To our knowledge, this is the first study to rigorously measure the additive influence of in‐hospital complications, clinical trajectory, and stability on discharge on the risk of 30‐day hospital readmission. Additionally, our study included a large, diverse study population that included all payers, all ages of adults, a mix of community, academic, and safety net hospitals, and individuals from a broad array of racial/ethnic and socioeconomic backgrounds.

Our results should be interpreted in light of several limitations. First, though we sought to represent a diverse group of hospitals, all study sites were located within north Texas and generalizability to other regions is uncertain. Second, our ascertainment of prior hospitalizations and readmissions was more inclusive than what could be typically accomplished in real time using only EHR data from a single clinical site. We performed a sensitivity analysis using only prior utilization data available within the EHR from the index hospital with no meaningful difference in our findings (data not shown). Additionally, a recent study found that 30‐day readmissions occur at the index hospital for over 75% of events, suggesting that 30‐day readmissions are fairly comprehensively captured even with only single‐site data.[39] Third, we were not able to include data on outpatient visits before or after the index hospitalization, which may influence the risk of readmission.[1, 40]

In conclusion, incorporating clinically granular EHR data from the entire course of hospitalization modestly improves prediction of 30‐day readmissions compared to models that only include information from the first 24 hours of hospital admission or models that use far fewer variables. However, given the limited improvement in prediction, our findings suggest that from the practical perspective of implementing real‐time models to identify those at highest risk for readmission, it may not be worth the added complexity of waiting until the end of a hospitalization to leverage additional data on hospital complications, and the trajectory of laboratory and vital sign values currently available in the EHR. Further improvement in prediction of readmissions will likely require accounting for psychosocial, functional, behavioral, and postdischarge factors not currently present in the inpatient EHR.

Disclosures: This study was presented at the Society of Hospital Medicine 2015 Annual Meeting in National Harbor, Maryland, and the Society of General Internal Medicine 2015 Annual Meeting in Toronto, Canada. This work was supported by the Agency for Healthcare Research and Qualityfunded UT Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01) and the Commonwealth Foundation (#20100323). Drs. Nguyen and Makam received funding from the UT Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). Dr. Halm was also supported in part by NIH/NCATS U54 RFA‐TR‐12‐006. The study sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

Unplanned hospital readmissions are frequent, costly, and potentially avoidable.[1, 2] Due to major federal financial readmissions penalties targeting excessive 30‐day readmissions, there is increasing attention to implementing hospital‐initiated interventions to reduce readmissions.[3, 4] However, universal enrollment of all hospitalized patients into such programs may be too resource intensive for many hospitals.[5] To optimize efficiency and effectiveness, interventions should be targeted to individuals most likely to benefit.[6, 7] However, existing readmission risk‐prediction models have achieved only modest discrimination, have largely used administrative claims data not available until months after discharge, or are limited to only a subset of patients with Medicare or a specific clinical condition.[8, 9, 10, 11, 12, 13, 14] These limitations have precluded accurate identification of high‐risk individuals in an all‐payer general medical inpatient population to provide actionable information for intervention prior to discharge.

Approaches using electronic health record (EHR) data could allow early identification of high‐risk patients during the index hospitalization to enable initiation of interventions prior to discharge. To date, such strategies have relied largely on EHR data from the day of admission.[15, 16] However, given that variation in 30‐day readmission rates are thought to reflect the quality of in‐hospital care, incorporating EHR data from the entire hospital stay to reflect hospital care processes and clinical trajectory may more accurately identify at‐risk patients.[17, 18, 19, 20] Improved accuracy in risk prediction would help better target intervention efforts in the immediate postdischarge period, an interval characterized by heightened vulnerability for adverse events.[21]

To help hospitals target transitional care interventions more effectively to high‐risk individuals prior to discharge, we derived and validated a readmissions risk‐prediction model incorporating EHR data from the entire course of the index hospitalization, which we termed the full‐stay EHR model. We also compared the full‐stay EHR model performance to our group's previously derived prediction model based on EHR data on the day of admission, termed the first‐day EHR model, as well as to 2 other validated readmission models similarly intended to yield near real‐time risk predictions prior to or shortly after hospital discharge.[9, 10, 15]

METHODS

Study Design, Population, and Data Sources

We conducted an observational cohort study using EHR data from 6 hospitals in the DallasFort Worth metroplex between November 1, 2009 and October 30, 2010 using the same EHR system (Epic Systems Corp., Verona, WI). One site was a university‐affiliated safety net hospital; the remaining 5 sites were teaching and nonteaching community sites.

We included consecutive hospitalizations among adults 18 years old discharged alive from any medicine inpatient service. For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization, were transferred to another acute care facility, left against medical advice, or who died outside of the hospital within 30 days of discharge. For model derivation, we randomly split the sample into separate derivation (50%) and validation cohorts (50%).

Outcomes

The primary outcome was 30‐day hospital readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100‐mile radius of Dallas, ascertained from an all‐payer regional hospitalization database. Nonelective hospitalizations included all hospitalizations classified as a emergency, urgent, or trauma, and excluded those classified as elective as per the Centers for Medicare and Medicaid Services Claim Inpatient Admission Type Code definitions.

Predictor Variables for the Full‐Stay EHR Model

The full‐stay EHR model was iteratively developed from our group's previously derived and validated risk‐prediction model using EHR data available on admission (first‐day EHR model).[15] For the full‐stay EHR model, we included all predictor variables included in our published first‐day EHR model as candidate risk factors. Based on prior literature, we additionally expanded candidate predictors available on admission to include marital status (proxy for social isolation) and socioeconomic disadvantage (percent poverty, unemployment, median income, and educational attainment by zip code of residence as proxy measures of the social and built environment).[22, 23, 24, 25, 26, 27] We also expanded the ascertainment of prior hospitalization to include admissions at both the index hospital and any of 75 acute care hospitals from the same, separate all‐payer regional hospitalization database used to ascertain 30‐day readmissions.

Candidate predictors from the remainder of the hospital stay (ie, following the first 24 hours of admission) were included if they were: (1) available in the EHR of all participating hospitals, (2) routinely collected or available at the time of hospital discharge, and (3) plausible predictors of adverse outcomes based on prior literature and clinical expertise. These included length of stay, in‐hospital complications, transfer to an intensive or coronary care unit, blood transfusions, vital sign instabilities within 24 hours of discharge, select laboratory values at time of discharge, and disposition status. We also assessed trajectories of vital signs and selected laboratory values (defined as changes in these measures from admission to discharge).

Statistical Analysis

Model Derivation

Univariate relationships between readmission and each of the candidate predictors were assessed in the derivation cohort using a prespecified significance threshold of P 0.05. We included all factors from our previously derived and validated first‐day EHR model as candidate predictors.[15] Continuous laboratory and vital sign values at the time of discharge were categorized based on clinically meaningful cutoffs; predictors with missing values were assumed to be normal (<1% missing for each variable). Significant univariate candidate variables were entered in a multivariate logistic regression model using stepwise backward selection with a prespecified significance threshold of P 0.05. We performed several sensitivity analyses to confirm the robustness of our model. First, we alternately derived the full‐stay model using stepwise forward selection. Second, we forced in all significant variables from our first‐day EHR model, and entered the candidate variables from the remainder of the hospital stay using both stepwise backward and forward selection separately. Third, prespecified interactions between variables were evaluated for inclusion. Though final predictors varied slightly between the different approaches, discrimination of each model was similar to the model derived using our primary analytic approach (C statistics 0.01, data not shown).

Model Validation

We assessed model discrimination and calibration of the derived full‐stay EHR model using the validation cohort. Model discrimination was estimated by the C statistic. The C statistic represents the probability that, given 2 hospitalized individuals (1 who was readmitted and the other who was not), the model will predict a higher risk for the readmitted patient than for the nonreadmitted patient. Model calibration was assessed by comparing predicted to observed probabilities of readmission by quintiles of risk, and with the Hosmer‐Lemeshow goodness‐of‐fit test.

Comparison to Existing Models

We compared the full‐stay EHR model performance to 3 previously published models: our group's first‐day EHR model, and the LACE (includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year) and HOSPITAL (includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay) models, which were both derived to predict 30‐day readmissions among general medical inpatients and were intended to help clinicians identify high‐risk patients to target for discharge interventions.[9, 10, 15] We assessed each model's performance in our validation cohort, calculating the C statistic, integrated discrimination index (IDI), and net reclassification index (NRI) compared to the full‐stay model. IDI is a summary measure of both discrimination and reclassification, where more positive values suggest improvement in model performance in both these domains compared to a reference model.[28] The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.[29] The theoretical range of values is 2 to 2, with more positive values indicating improved net reclassification compared to a reference model. Here, we calculated a category‐based NRI to evaluate the performance of models in correctly classifying individuals with and without readmissions into the highest readmission risk quintile versus the lowest 4 risk quintiles compared to the full‐stay EHR model.[29] This prespecified cutoff is relevant for hospitals interested in identifying the highest‐risk individuals for targeted intervention.[6] Because some hospitals may be able to target a greater number of individuals for intervention, we performed a sensitivity analysis by assessing category‐based NRI for reclassification into the top 2 risk quintiles versus the lowest 3 risk quintiles and found no meaningful difference in our results (data not shown). Finally, we qualitatively assessed calibration of comparator models in our validation cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, TX). This study was approved by the UT Southwestern Medical Center institutional review board.

RESULTS

Overall, 32,922 index hospitalizations were included in our study cohort; 12.7% resulted in a 30‐day readmission (see Supporting Figure 1 in the online version of this article). Individuals had a mean age of 62 years and had diverse race/ethnicity and primary insurance status; half were female (Table 1). The study sample was randomly split into a derivation cohort (50%, n = 16,492) and validation cohort (50%, n = 16,430). Individuals in the derivation cohort with a 30‐day readmission had markedly different socioeconomic and clinical characteristics compared to those not readmitted (Table 1).

Baseline Characteristics and Candidate Variables for Risk‐Prediction Model
Entire Cohort, N = 32,922 Derivation Cohort, N = 16,492

No Readmission, N = 14,312

Readmission, N = 2,180

P Value
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit; IQR, interquartile range; SD, standard deviation. *20% poverty in zip code as per high poverty area US Census designation. Prior ED visit at site of index hospitalization within the past year. Prior hospitalization at any of 75 acute care hospitals in the North Texas region within the past year. Nonelective admission defined as hospitalization categorized as medical emergency, urgent, or trauma. ∥Calculated from diagnoses available within 1 year prior to index hospitalization. Conditions were considered complications if they were not listed as a principle diagnosis for hospitalization or as a previous diagnosis in the prior year. #On day of discharge or last known observation before discharge. Instabilities were defined as temperature 37.8C, heart rate >100 beats/minute, respiratory rate >24 breaths/minute, systolic blood pressure 90 mm Hg, or oxygen saturation <90%. **Discharges to nursing home, skilled nursing facility, or long‐term acute care hospital.

Demographic characteristics
Age, y, mean (SD) 62 (17.3) 61 (17.4) 64 (17.0) 0.001
Female, n (%) 17,715 (53.8) 7,694 (53.8) 1,163 (53.3) 0.72
Race/ethnicity 0.001
White 21,359 (64.9) 9,329 (65.2) 1,361 (62.4)
Black 5,964 (18.1) 2,520 (17.6) 434 (19.9)
Hispanic 4,452 (13.5) 1,931 (13.5) 338 (15.5)
Other 1,147 (3.5) 532 (3.7) 47 (2.2)
Marital status, n (%) 0.001
Single 8,076 (24.5) 3,516 (24.6) 514 (23.6)
Married 13,394 (40.7) 5,950 (41.6) 812 (37.3)
Separated/divorced 3,468 (10.5) 1,460 (10.2) 251 (11.5)
Widowed 4,487 (13.7) 1,868 (13.1) 388 (17.8)
Other 3,497 (10.6) 1,518 (10.6) 215 (9.9)
Primary payer, n (%) 0.001
Private 13,090 (39.8) 5,855 (40.9) 726 (33.3)
Medicare 13,015 (39.5) 5,597 (39.1) 987 (45.3)
Medicaid 2,204 (6.7) 852 (5.9) 242 (11.1)
Charity, self‐pay, or other 4,613 (14.0) 2,008 (14.0) 225 (10.3)
High‐poverty neighborhood, n (%)* 7,468 (22.7) 3,208 (22.4) 548 (25.1) 0.001
Utilization history
1 ED visits in past year, n (%) 9,299 (28.2) 3,793 (26.5) 823 (37.8) 0.001
1 hospitalizations in past year, n (%) 10,189 (30.9) 4,074 (28.5) 1,012 (46.4) 0.001
Clinical factors from first day of hospitalization
Nonelective admission, n (%) 27,818 (84.5) 11,960 (83.6) 1,960 (89.9) 0.001
Charlson Comorbidity Index, median (IQR)∥ 0 (01) 0 (00) 0 (03) 0.001
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 355 (1.1) 119 (0.8) 46 (2.1) 0.001
Albumin 23 g/dL 4,732 (14.4) 1,956 (13.7) 458 (21.0) 0.001
Aspartate aminotransferase >40 U/L 4,610 (14.0) 1,922 (13.4) 383 (17.6) 0.001
Creatine phosphokinase <60 g/L 3,728 (11.3) 1,536 (10.7) 330 (15.1) 0.001
Mean corpuscular volume >100 fL/red cell 1,346 (4.1) 537 (3.8) 134 (6.2) 0.001
Platelets <90 103/L 912 (2.8) 357 (2.5) 116 (5.3) 0.001
Platelets >350 103/L 3,332 (10.1) 1,433 (10.0) 283 (13.0) 0.001
Prothrombin time >35 seconds 248 (0.8) 90 (0.6) 35 (1.6) 0.001
Clinical factors from remainder of hospital stay
Length of stay, d, median (IQR) 4 (26) 4 (26) 5 (38) 0.001
ICU transfer after first 24 hours, n (%) 988 (3.0) 408 (2.9) 94 (4.3) 0.001
Hospital complications, n (%)
Clostridium difficile infection 119 (0.4) 44 (0.3) 24 (1.1) 0.001
Pressure ulcer 358 (1.1) 126 (0.9) 46 (2.1) 0.001
Venous thromboembolism 301 (0.9) 112 (0.8) 34 (1.6) 0.001
Respiratory failure 1,048 (3.2) 463 (3.2) 112 (5.1) 0.001
Central line‐associated bloodstream infection 22 (0.07) 6 (0.04) 5 (0.23) 0.005
Catheter‐associated urinary tract infection 47 (0.14) 20 (0.14) 6 (0.28) 0.15
Acute myocardial infarction 293 (0.9) 110 (0.8) 32 (1.5) 0.001
Pneumonia 1,754 (5.3) 719 (5.0) 154 (7.1) 0.001
Sepsis 853 (2.6) 368 (2.6) 73 (3.4) 0.04
Blood transfusion during hospitalization, n (%) 4,511 (13.7) 1,837 (12.8) 425 (19.5) 0.001
Laboratory abnormalities at discharge#
Blood urea nitrogen >20 mg/dL, n (%) 10,014 (30.4) 4,077 (28.5) 929 (42.6) 0.001
Sodium <135 mEq/L, n (%) 4,583 (13.9) 1,850 (12.9) 440 (20.2) 0.001
Hematocrit 27 3,104 (9.4) 1,231 (8.6) 287 (13.2) 0.001
1 vital sign instability at discharge, n (%)# 6,192 (18.8) 2,624 (18.3) 525 (24.1) 0.001
Discharge location, n (%) 0.001
Home 23,339 (70.9) 10,282 (71.8) 1,383 (63.4)
Home health 3,185 (9.7) 1,356 (9.5) 234 (10.7)
Postacute care** 5,990 (18.2) 2,496 (17.4) 549 (25.2)
Hospice 408 (1.2) 178 (1.2) 14 (0.6)

Derivation and Validation of the Full‐Stay EHR Model for 30‐Day Readmission

Our final model included 24 independent variables, including demographic characteristics, utilization history, clinical factors from the first day of admission, and clinical factors from the remainder of the hospital stay (Table 2). The strongest independent predictor of readmission was hospital‐acquired Clostridium difficile infection (adjusted odds ratio [AOR]: 2.03, 95% confidence interval [CI] 1.18‐3.48); other hospital‐acquired complications including pressure ulcers and venous thromboembolism were also significant predictors. Though having Medicaid was associated with increased odds of readmission (AOR: 1.55, 95% CI: 1.31‐1.83), other zip codelevel measures of socioeconomic disadvantage were not predictive and were not included in the final model. Being discharged to hospice was associated with markedly lower odds of readmission (AOR: 0.23, 95% CI: 0.13‐0.40).

Final Full‐Stay EHR Model Predicting 30‐Day Readmissions (Derivation Cohort, N = 16,492)
Odds Ratio (95% CI)
Univariate Multivariate*
  • NOTE: Abbreviations: CI, confidence interval; ED, emergency department. *Values shown reflect adjusted odds ratios and 95% CI for each factor after adjustment for all other factors listed in the table.

Demographic characteristics
Age, per 10 years 1.08 (1.051.11) 1.07 (1.041.10)
Medicaid 1.97 (1.702.29) 1.55 (1.311.83)
Widow 1.44 (1.281.63) 1.27 (1.111.45)
Utilization history
Prior ED visit, per visit 1.08 (1.061.10) 1.04 (1.021.06)
Prior hospitalization, per hospitalization 1.30 (1.271.34) 1.16 (1.121.20)
Hospital and clinical factors from first day of hospitalization
Nonelective admission 1.75 (1.512.03) 1.42 (1.221.65)
Charlson Comorbidity Index, per point 1.19 (1.171.21) 1.06 (1.041.09)
Laboratory abnormalities within 24 hours of admission
Albumin <2 g/dL 2.57 (1.823.62) 1.52 (1.052.21)
Albumin 23 g/dL 1.68 (1.501.88) 1.20 (1.061.36)
Aspartate aminotransferase >40 U/L 1.37 (1.221.55) 1.21 (1.061.38)
Creatine phosphokinase <60 g/L 1.48 (1.301.69) 1.28 (1.111.46)
Mean corpuscular volume >100 fL/red cell 1.68 (1.382.04) 1.32 (1.071.62)
Platelets <90 103/L 2.20 (1.772.72) 1.56 (1.231.97)
Platelets >350 103/L 1.34 (1.171.54) 1.24 (1.081.44)
Prothrombin time >35 seconds 2.58 (1.743.82) 1.92 (1.272.90)
Hospital and clinical factors from remainder of hospital stay
Length of stay, per day 1.08 (1.071.09) 1.06 (1.041.07)
Hospital complications
Clostridium difficile infection 3.61 (2.195.95) 2.03 (1.183.48)
Pressure ulcer 2.43 (1.733.41) 1.64 (1.152.34)
Venous thromboembolism 2.01 (1.362.96) 1.55 (1.032.32)
Laboratory abnormalities at discharge
Blood urea nitrogen >20 mg/dL 1.86 (1.702.04) 1.37 (1.241.52)
Sodium <135 mEq/L 1.70 (1.521.91) 1.34 (1.181.51)
Hematocrit 27 1.61 (1.401.85) 1.22 (1.051.41)
Vital sign instability at discharge, per instability 1.29 (1.201.40) 1.25 (1.151.36)
Discharged to hospice 0.51 (0.300.89) 0.23 (0.130.40)

In our validation cohort, the full‐stay EHR model had fair discrimination, with a C statistic of 0.69 (95% CI: 0.68‐0.70) (Table 3). The full‐stay EHR model was well calibrated across all quintiles of risk, with slight overestimation of predicted risk in the lowest and highest quintiles (Figure 1a) (see Supporting Table 5 in the online version of this article). It also effectively stratified individuals across a broad range of predicted readmission risk from 4.1% in the lowest decile to 36.5% in the highest decile (Table 3).

Comparison of the Discrimination and Reclassification of Different Readmission Models*
Model Name C‐Statistic (95% CI) IDI, % (95% CI) NRI (95% CI) Average Predicted Risk, %
Lowest Decile Highest Decile
  • NOTE: Abbreviations; CI, confidence interval; EHR, electronic health record; IDI, Integrated Discrimination Improvement; NRI, Net Reclassification Index. *All measures were assessed using the validation cohort (N = 16,430), except for estimating the C‐statistic for the derivation cohort. P value <0.001 for all pairwise comparisons of C‐statistic between full‐stay model and first‐day, LACE, and HOSPITAL models, respectively. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Full‐stay EHR model
Derivation cohort 0.72 (0.70 to 0.73) 4.1 36.5
Validation cohort 0.69 (0.68 to 0.70) [Reference] [Reference] 4.1 36.5
First‐day EHR model 0.67 (0.66 to 0.68) 1.2 (1.4 to 1.0) 0.020 (0.038 to 0.002) 5.8 31.9
LACE model 0.65 (0.64 to 0.66) 2.6 (2.9 to 2.3) 0.046 (0.067 to 0.024) 6.1 27.5
HOSPITAL model 0.64 (0.62 to 0.65) 3.2 (3.5 to 2.9) 0.058 (0.080 to 0.035) 6.7 26.6
Figure 1
Comparison of the calibration of different readmission models. Calibration graphs for full‐stay (a), first‐day (b), LACE (c), and HOSPITAL (d) models in the validation cohort. Each graph shows predicted probability compared to observed probability of readmission by quintiles of risk for each model. The LACE model includes Length of stay, Acute (nonelective) admission status, Charlson Comorbidity Index, and Emergency department visits in the past year. The HOSPITAL model includes Hemoglobin at discharge, discharge from Oncology service, Sodium level at discharge, Procedure during index hospitalization, Index hospitalization Type (nonelective), number of Admissions in the past year, and Length of stay.

Comparing the Performance of the Full‐Stay EHR Model to Other Models

The full‐stay EHR model had better discrimination compared to the first‐day EHR model and the LACE and HOSPITAL models, though the magnitude of improvement was modest (Table 3). The full‐stay EHR model also stratified individuals across a broader range of readmission risk, and was better able to discriminate and classify those in the highest quintile of risk from those in the lowest 4 quintiles of risk compared to other models as assessed by the IDI and NRI (Table 3) (see Supporting Tables 14 and Supporting Figure 2 in the online version of this article). In terms of model calibration, both the first‐day EHR and LACE models were also well calibrated, whereas the HOSPITAL model was less robust (Figure 1).

The diagnostic accuracy of the full‐stay EHR model in correctly predicting those in the highest quintile of risk was better than that of the first‐day, LACE, and HOSPITAL models, though overall improvements in the sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios were also modest (see Supporting Table 6 in the online version of this article).

DISCUSSION

In this study, we used clinically detailed EHR data from the entire hospitalization on 32,922 individuals treated in 6 diverse hospitals to develop an all‐payer, multicondition readmission risk‐prediction model. To our knowledge, this is the first 30‐day hospital readmission risk‐prediction model to use a comprehensive set of factors from EHR data from the entire hospital stay. Prior EHR‐based models have focused exclusively on data available on or prior to the first day of admission, which account for clinical severity on admission but do not account for factors uncovered during the inpatient stay that influence the chance of a postdischarge adverse outcome.[15, 30] We specifically assessed the prognostic impact of a comprehensive set of factors from the entire index hospitalization, including hospital‐acquired complications, clinical trajectory, and stability on discharge in predicting hospital readmissions. Our full‐stay EHR model had statistically better discrimination, calibration, and diagnostic accuracy than our existing all‐cause first‐day EHR model[15] and 2 previously published readmissions models that included more limited information from hospitalization (such as length of stay).[9, 10] However, although the more complicated full‐stay EHR model was statistically better than previously published models, we were surprised that the predictive performance was only modestly improved despite the inclusion of many additional clinically relevant prognostic factors.

Taken together, our study has several important implications. First, the added complexity and resource intensity of implementing a full‐stay EHR model yields only modestly improved readmission risk prediction. Thus, hospitals and healthcare systems interested in targeting their highest‐risk individuals for interventions to reduce 30‐day readmission should consider doing so within the first day of hospital admission. Our group's previously derived and validated first‐day EHR model, which used data only from the first day of admission, qualitatively performed nearly as well as the full‐stay EHR model.[15] Additionally, a recent study using only preadmission EHR data to predict 30‐day readmissions also achieved similar discrimination and diagnostic accuracy as our full‐stay model.[30]

Second, the field of readmissions risk‐prediction modeling may be reaching the maximum achievable model performance using data that are currently available in the EHR. Our limited ability to accurately predict all‐cause 30‐day readmission risk may reflect the influence of currently unmeasured patient, system, and community factors on readmissions.[31, 32, 33] Due to the constraints of data collected in the EHR, we were unable to include several patient‐level clinical characteristics associated with hospital readmission, including self‐perceived health status, functional impairment, and cognition.[33, 34, 35, 36] However, given their modest effect sizes (ORs ranging from 1.062.10), adequately measuring and including these risk factors in our model may not meaningfully improve model performance and diagnostic accuracy. Further, many social and behavioral patient‐level factors are also not consistently available in EHR data. Though we explored the role of several neighborhood‐level socioeconomic measuresincluding prevalence of poverty, median income, education, and unemploymentwe found that none were significantly associated with 30‐day readmissions. These particular measures may have been inadequate to characterize individual‐level social and behavioral factors, as several previous studies have demonstrated that patient‐level factors such as social support, substance abuse, and medication and visit adherence can influence readmission risk in heart failure and pneumonia.[11, 16, 22, 25] This underscores the need for more standardized routine collection of data across functional, social, and behavioral domains in clinical settings, as recently championed by the Institute of Medicine.[11, 37] Integrating data from outside the EHR on postdischarge health behaviors, self‐management, follow‐up care, recovery, and home environment may be another important but untapped strategy for further improving prediction of readmissions.[25, 38]

Third, a multicondition readmission risk‐prediction model may be a less effective strategy than more customized disease‐specific models for selected conditions associated with high 30‐day readmission rates. Our group's previously derived and internally validated models for heart failure and human immunodeficiency virus had superior discrimination compared to our full‐stay EHR model (C statistic of 0.72 for each).[11, 13] However, given differences in the included population and time periods studied, a head‐to‐head comparison of these different strategies is needed to assess differences in model performance and utility.

Our study had several strengths. To our knowledge, this is the first study to rigorously measure the additive influence of in‐hospital complications, clinical trajectory, and stability on discharge on the risk of 30‐day hospital readmission. Additionally, our study included a large, diverse study population that included all payers, all ages of adults, a mix of community, academic, and safety net hospitals, and individuals from a broad array of racial/ethnic and socioeconomic backgrounds.

Our results should be interpreted in light of several limitations. First, though we sought to represent a diverse group of hospitals, all study sites were located within north Texas and generalizability to other regions is uncertain. Second, our ascertainment of prior hospitalizations and readmissions was more inclusive than what could be typically accomplished in real time using only EHR data from a single clinical site. We performed a sensitivity analysis using only prior utilization data available within the EHR from the index hospital with no meaningful difference in our findings (data not shown). Additionally, a recent study found that 30‐day readmissions occur at the index hospital for over 75% of events, suggesting that 30‐day readmissions are fairly comprehensively captured even with only single‐site data.[39] Third, we were not able to include data on outpatient visits before or after the index hospitalization, which may influence the risk of readmission.[1, 40]

In conclusion, incorporating clinically granular EHR data from the entire course of hospitalization modestly improves prediction of 30‐day readmissions compared to models that only include information from the first 24 hours of hospital admission or models that use far fewer variables. However, given the limited improvement in prediction, our findings suggest that from the practical perspective of implementing real‐time models to identify those at highest risk for readmission, it may not be worth the added complexity of waiting until the end of a hospitalization to leverage additional data on hospital complications, and the trajectory of laboratory and vital sign values currently available in the EHR. Further improvement in prediction of readmissions will likely require accounting for psychosocial, functional, behavioral, and postdischarge factors not currently present in the inpatient EHR.

Disclosures: This study was presented at the Society of Hospital Medicine 2015 Annual Meeting in National Harbor, Maryland, and the Society of General Internal Medicine 2015 Annual Meeting in Toronto, Canada. This work was supported by the Agency for Healthcare Research and Qualityfunded UT Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01) and the Commonwealth Foundation (#20100323). Drs. Nguyen and Makam received funding from the UT Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). Dr. Halm was also supported in part by NIH/NCATS U54 RFA‐TR‐12‐006. The study sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

References
  1. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee‐for‐service program. N Engl J Med. 2009;360(14):14181428.
  2. Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391E402.
  3. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital‐initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433440.
  4. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30‐day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520528.
  5. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  6. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real‐time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):9981005.
  7. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):11481154.
  8. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):16881698.
  9. Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551557.
  10. Donze J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30‐day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. 2013;173(8):632638.
  11. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  12. Singal AG, Rahimi RS, Clark C, et al. An automated model using electronic medical record data identifies patients with cirrhosis at high risk for readmission. Clin Gastroenterol Hepatol. 2013;11(10):13351341.e1331.
  13. Nijhawan AE, Clark C, Kaplan R, Moore B, Halm EA, Amarasingham R. An electronic medical record‐based model to predict 30‐day risk of readmission and death among HIV‐infected inpatients. J Acquir Immune Defic Syndr. 2012;61(3):349358.
  14. Horwitz LI, Partovian C, Lin Z, et al. Development and use of an administrative claims measure for profiling hospital‐wide performance on 30‐day unplanned readmission. Ann Intern Med. 2014;161(10 suppl):S66S75.
  15. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record‐based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15(1):39.
  16. Watson AJ, O'Rourke J, Jethwani K, et al. Linking electronic health record‐extracted psychosocial data in real‐time to risk of readmission for heart failure. Psychosomatics. 2011;52(4):319327.
  17. Ashton CM, Wray NP. A conceptual framework for the study of early readmission as an indicator of quality of care. Soc Sci Med. 1996;43(11):15331541.
  18. Dharmarajan K, Hsieh AF, Lin Z, et al. Hospital readmission performance and patterns of readmission: retrospective cohort study of Medicare admissions. BMJ. 2013;347:f6571.
  19. Cassel CK, Conway PH, Delbanco SF, Jha AK, Saunders RS, Lee TH. Getting more performance from performance measurement. N Engl J Med. 2014;371(23):21452147.
  20. Bradley EH, Sipsma H, Horwitz LI, et al. Hospital strategy uptake and reductions in unplanned readmission rates for patients with heart failure: a prospective study. J Gen Intern Med. 2015;30(5):605611.
  21. Krumholz HM. Post‐hospital syndrome—an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100102.
  22. Calvillo‐King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269282.
  23. Keyhani S, Myers LJ, Cheng E, Hebert P, Williams LS, Bravata DM. Effect of clinical and social risk factors on hospital profiling for stroke readmission: a cohort study. Ann Intern Med. 2014;161(11):775784.
  24. Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvantage and 30‐day rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161(11):765774.
  25. Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C. Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community‐dwelling Medicare beneficiaries. Gerontologist. 2008;48(4):495504.
  26. Hu J, Gonsahn MD, Nerenz DR. Socioeconomic status and readmissions: evidence from an urban teaching hospital. Health Aff (Millwood). 2014;33(5):778785.
  27. Nagasako EM, Reidhead M, Waterman B, Dunagan WC. Adding socioeconomic data to hospital readmissions calculations may produce more useful results. Health Aff (Millwood). 2014;33(5):786791.
  28. Pencina MJ, D'Agostino RB, D'Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157172; discussion 207–212.
  29. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med. 2014;160(2):122131.
  30. Shadmi E, Flaks‐Manov N, Hoshen M, Goldman O, Bitterman H, Balicer RD. Predicting 30‐day readmissions with preadmission electronic health record data. Med Care. 2015;53(3):283289.
  31. Kangovi S, Grande D. Hospital readmissions—not just a measure of quality. JAMA. 2011;306(16):17961797.
  32. Joynt KE, Jha AK. Thirty‐day readmissions—truth and consequences. N Engl J Med. 2012;366(15):13661369.
  33. Greysen SR, Stijacic Cenzer I, Auerbach AD, Covinsky KE. Functional impairment and hospital readmission in medicare seniors. JAMA Intern Med. 2015;175(4):559565.
  34. Holloway JJ, Thomas JW, Shapiro L. Clinical and sociodemographic risk factors for readmission of Medicare beneficiaries. Health Care Financ Rev. 1988;10(1):2736.
  35. Patel A, Parikh R, Howell EH, Hsich E, Landers SH, Gorodeski EZ. Mini‐cog performance: novel marker of post discharge risk among patients hospitalized for heart failure. Circ Heart Fail. 2015;8(1):816.
  36. Hoyer EH, Needham DM, Atanelov L, Knox B, Friedman M, Brotman DJ. Association of impaired functional status at hospital discharge and subsequent rehospitalization. J Hosp Med. 2014;9(5):277282.
  37. Adler NE, Stead WW. Patients in context—EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372(8):698701.
  38. Nguyen OK, Chan CV, Makam A, Stieglitz H, Amarasingham R. Envisioning a social‐health information exchange as a platform to support a patient‐centered medical neighborhood: a feasibility study. J Gen Intern Med. 2015;30(1):6067.
  39. Henke RM, Karaca Z, Lin H, Wier LM, Marder W, Wong HS. Patient factors contributing to variation in same‐hospital readmission rate. Med Care Res Review. 2015;72(3):338358.
  40. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):14411447.
References
  1. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee‐for‐service program. N Engl J Med. 2009;360(14):14181428.
  2. Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183(7):E391E402.
  3. Rennke S, Nguyen OK, Shoeb MH, Magan Y, Wachter RM, Ranji SR. Hospital‐initiated transitional care interventions as a patient safety strategy: a systematic review. Ann Intern Med. 2013;158(5 pt 2):433440.
  4. Hansen LO, Young RS, Hinami K, Leung A, Williams MV. Interventions to reduce 30‐day rehospitalization: a systematic review. Ann Intern Med. 2011;155(8):520528.
  5. Rennke S, Shoeb MH, Nguyen OK, Magan Y, Wachter RM, Ranji SR. Interventions to Improve Care Transitions at Hospital Discharge. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  6. Amarasingham R, Patel PC, Toto K, et al. Allocating scarce resources in real‐time to reduce heart failure readmissions: a prospective, controlled study. BMJ Qual Saf. 2013;22(12):9981005.
  7. Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):11481154.
  8. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):16881698.
  9. Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182(6):551557.
  10. Donze J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30‐day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. 2013;173(8):632638.
  11. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  12. Singal AG, Rahimi RS, Clark C, et al. An automated model using electronic medical record data identifies patients with cirrhosis at high risk for readmission. Clin Gastroenterol Hepatol. 2013;11(10):13351341.e1331.
  13. Nijhawan AE, Clark C, Kaplan R, Moore B, Halm EA, Amarasingham R. An electronic medical record‐based model to predict 30‐day risk of readmission and death among HIV‐infected inpatients. J Acquir Immune Defic Syndr. 2012;61(3):349358.
  14. Horwitz LI, Partovian C, Lin Z, et al. Development and use of an administrative claims measure for profiling hospital‐wide performance on 30‐day unplanned readmission. Ann Intern Med. 2014;161(10 suppl):S66S75.
  15. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record‐based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15(1):39.
  16. Watson AJ, O'Rourke J, Jethwani K, et al. Linking electronic health record‐extracted psychosocial data in real‐time to risk of readmission for heart failure. Psychosomatics. 2011;52(4):319327.
  17. Ashton CM, Wray NP. A conceptual framework for the study of early readmission as an indicator of quality of care. Soc Sci Med. 1996;43(11):15331541.
  18. Dharmarajan K, Hsieh AF, Lin Z, et al. Hospital readmission performance and patterns of readmission: retrospective cohort study of Medicare admissions. BMJ. 2013;347:f6571.
  19. Cassel CK, Conway PH, Delbanco SF, Jha AK, Saunders RS, Lee TH. Getting more performance from performance measurement. N Engl J Med. 2014;371(23):21452147.
  20. Bradley EH, Sipsma H, Horwitz LI, et al. Hospital strategy uptake and reductions in unplanned readmission rates for patients with heart failure: a prospective study. J Gen Intern Med. 2015;30(5):605611.
  21. Krumholz HM. Post‐hospital syndrome—an acquired, transient condition of generalized risk. N Engl J Med. 2013;368(2):100102.
  22. Calvillo‐King L, Arnold D, Eubank KJ, et al. Impact of social factors on risk of readmission or mortality in pneumonia and heart failure: systematic review. J Gen Intern Med. 2013;28(2):269282.
  23. Keyhani S, Myers LJ, Cheng E, Hebert P, Williams LS, Bravata DM. Effect of clinical and social risk factors on hospital profiling for stroke readmission: a cohort study. Ann Intern Med. 2014;161(11):775784.
  24. Kind AJ, Jencks S, Brock J, et al. Neighborhood socioeconomic disadvantage and 30‐day rehospitalization: a retrospective cohort study. Ann Intern Med. 2014;161(11):765774.
  25. Arbaje AI, Wolff JL, Yu Q, Powe NR, Anderson GF, Boult C. Postdischarge environmental and socioeconomic factors and the likelihood of early hospital readmission among community‐dwelling Medicare beneficiaries. Gerontologist. 2008;48(4):495504.
  26. Hu J, Gonsahn MD, Nerenz DR. Socioeconomic status and readmissions: evidence from an urban teaching hospital. Health Aff (Millwood). 2014;33(5):778785.
  27. Nagasako EM, Reidhead M, Waterman B, Dunagan WC. Adding socioeconomic data to hospital readmissions calculations may produce more useful results. Health Aff (Millwood). 2014;33(5):786791.
  28. Pencina MJ, D'Agostino RB, D'Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157172; discussion 207–212.
  29. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med. 2014;160(2):122131.
  30. Shadmi E, Flaks‐Manov N, Hoshen M, Goldman O, Bitterman H, Balicer RD. Predicting 30‐day readmissions with preadmission electronic health record data. Med Care. 2015;53(3):283289.
  31. Kangovi S, Grande D. Hospital readmissions—not just a measure of quality. JAMA. 2011;306(16):17961797.
  32. Joynt KE, Jha AK. Thirty‐day readmissions—truth and consequences. N Engl J Med. 2012;366(15):13661369.
  33. Greysen SR, Stijacic Cenzer I, Auerbach AD, Covinsky KE. Functional impairment and hospital readmission in medicare seniors. JAMA Intern Med. 2015;175(4):559565.
  34. Holloway JJ, Thomas JW, Shapiro L. Clinical and sociodemographic risk factors for readmission of Medicare beneficiaries. Health Care Financ Rev. 1988;10(1):2736.
  35. Patel A, Parikh R, Howell EH, Hsich E, Landers SH, Gorodeski EZ. Mini‐cog performance: novel marker of post discharge risk among patients hospitalized for heart failure. Circ Heart Fail. 2015;8(1):816.
  36. Hoyer EH, Needham DM, Atanelov L, Knox B, Friedman M, Brotman DJ. Association of impaired functional status at hospital discharge and subsequent rehospitalization. J Hosp Med. 2014;9(5):277282.
  37. Adler NE, Stead WW. Patients in context—EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372(8):698701.
  38. Nguyen OK, Chan CV, Makam A, Stieglitz H, Amarasingham R. Envisioning a social‐health information exchange as a platform to support a patient‐centered medical neighborhood: a feasibility study. J Gen Intern Med. 2015;30(1):6067.
  39. Henke RM, Karaca Z, Lin H, Wier LM, Marder W, Wong HS. Patient factors contributing to variation in same‐hospital readmission rate. Med Care Res Review. 2015;72(3):338358.
  40. Weinberger M, Oddone EZ, Henderson WG. Does increased access to primary care reduce hospital readmissions? Veterans Affairs Cooperative Study Group on Primary Care and Hospital Readmission. N Engl J Med. 1996;334(22):14411447.
Issue
Journal of Hospital Medicine - 11(7)
Issue
Journal of Hospital Medicine - 11(7)
Page Number
473-480
Page Number
473-480
Publications
Publications
Article Type
Display Headline
Predicting all‐cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison
Display Headline
Predicting all‐cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison
Sections
Article Source
© 2016 Society of Hospital Medicine
Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Oanh Kieu Nguyen, MD, 5323 Harry Hines Blvd., Dallas, Texas 75390‐9169; Telephone: 214‐648‐3135; Fax: 214‐648‐3232; E‐mail: oanhK.nguyen@UTSouthwestern.edu
Content Gating
Gated (full article locked unless allowed per User)
Gating Strategy
First Peek Free
Article PDF Media
Media Files

Financial Performance and Outcomes

Article Type
Changed
Mon, 01/02/2017 - 19:34
Display Headline
Relationship between hospital financial performance and publicly reported outcomes

Hospital care accounts for the single largest category of national healthcare expenditures, totaling $936.9 billion in 2013.[1] With ongoing scrutiny of US healthcare spending, hospitals are under increasing pressure to justify high costs and robust profits.[2] However, the dominant fee‐for‐service reimbursement model creates incentives for hospitals to prioritize high volume over high‐quality care to maximize profits.[3] Because hospitals may be reluctant to implement improvements if better quality is not accompanied by better payment or improved financial margins, an approach to stimulate quality improvement among hospitals has been to leverage consumer pressure through required public reporting of selected outcome metrics.[4, 5] Public reporting of outcomes is thought to influence hospital reputation; in turn, reputation affects patient perceptions and influences demand for hospital services, potentially enabling reputable hospitals to command higher prices for services to enhance hospital revenue.[6, 7]

Though improving outcomes is thought to reduce overall healthcare costs, it is unclear whether improving outcomes results in a hospital's financial return on investment.[4, 5, 8] Quality improvement can require substantial upfront investment, requiring that hospitals already have robust financial health to engage in such initiatives.[9, 10] Consequently, instead of stimulating broad efforts in quality improvement, public reporting may exacerbate existing disparities in hospital quality and finances, by rewarding already financially healthy hospitals, and by inadvertently penalizing hospitals without the means to invest in quality improvement.[11, 12, 13, 14, 15] Alternately, because fee‐for‐service remains the dominant reimbursement model for hospitals, loss of revenue through reducing readmissions may outweigh any financial gains from improved public reputation and result in worse overall financial performance, though robust evidence for this concern is lacking.[16, 17]

A small number of existing studies suggest a limited correlation between improved hospital financial performance and improved quality, patient safety, and lower readmission rates.[18, 19, 20] However, these studies had several limitations. They were conducted prior to public reporting of selected outcome metrics by the Centers for Medicare and Medicaid Services (CMS)[18, 19, 20]; used data from the Medicare Cost Report, which is not uniformly audited and thus prone to measurement error[19, 20]; used only relative measures of hospital financial performance (eg, operating margin), which do not capture the absolute amount of revenue potentially available for investment in quality improvement[18, 19]; or compared only hospitals at the extremes of financial performance, potentially exaggerating the magnitude of the relationship between hospital financial performance and quality outcomes.[19]

To address this gap in the literature, we sought to assess whether hospitals with robust financial performance have lower 30‐day risk‐standardized mortality and hospital readmission rates for acute myocardial infarction (AMI), congestive heart failure (CHF), and pneumonia (PNA). Given the concern that hospitals with the lowest mortality and readmission rates may experience a decrease in financial performance due to the lower volume of hospitalizations, we also assessed whether hospitals with the lowest readmission and mortality rates had a differential change in financial performance over time compared to hospitals with the highest rates.

METHODS

Data Sources and Study Population

This was an observational study using audited financial data from the 2008 and 2012 Hospital Annual Financial Data Files from the Office of Statewide Health Planning and Development (OSHPD) in the state of California, merged with data on outcome measures publicly reported by CMS via the Hospital Compare website for July 1, 2008 to June 30, 2011.[21, 22] We included all general acute care hospitals with available OSHPD data in 2008 and at least 1 publicly reported outcome from 2008 to 2011. We excluded hospitals without 1 year of audited financial data for 2008 and hospitals that closed during 2008 to 2011.

Measures of Financial Performance

Because we hypothesized that the absolute amount of revenue generated from clinical operations would influence investment in quality improvement programs more so than relative changes in revenue,[20] we used net revenue from operations (total operating revenue minus total operating expense) as our primary measure of hospital financial performance. We also performed 2 companion analyses using 2 commonly reported relative measures of financial performanceoperating margin (net revenue from operations divided by total operating revenue) and total margin (net total revenue divided by total revenue from all sources). Net revenue from operations for 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers.

Outcomes

For our primary analysis, the primary outcomes were publicly reported all‐cause 30‐day risk‐standardized mortality rates (RSMR) and readmission rates (RSRR) for AMI, CHF, and PNA aggregated over a 3‐year period. These measures were adjusted for key demographic and clinical characteristics available in Medicare data. CMS began publicly reporting 30‐day RSMR for AMI and CHF in June 2007, RSMR for PNA in June 2008, and RSRR for all 3 conditions in July 2009.[23, 24]

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion analysis where the primary outcome of interest was change in hospital financial performance over time, using the same definitions of financial performance outlined above. For this companion analysis, publicly reported 30‐day RSMR and RSRR for AMI, CHF, and PNA were assessed as predictors of subsequent financial performance.

Hospital Characteristics

Hospital characteristics were ascertained from the OSHPD data. Safety‐net status was defined as hospitals with an annual Medicaid caseload (number of Medicaid discharges divided by the total number of discharges) 1 standard deviation above the mean Medicaid caseload, as defined in previous studies.[25]

Statistical Analyses

Effect of Baseline Financial Performance on Subsequent Publicly Reported Outcomes

To estimate the relationship between baseline hospital financial performance in 2008 and subsequent RSMR and RSRR for AMI, CHF, and PNA from 2008 to 2011, we used linear regression adjusted for the following hospital characteristics: teaching status, rural location, bed size, safety‐net status, ownership, Medicare caseload, and volume of cases reported for the respective outcome. We accounted for clustering of hospitals by ownership. We adjusted for hospital volume of reported cases for each condition given that the risk‐standardization models used by CMS shrink outcomes for small hospitals to the mean, and therefore do not account for a potential volume‐outcome relationship.[26] We conducted a sensitivity analysis excluding hospitals at the extremes of financial performance, defined as hospitals with extreme outlier values for each financial performance measure (eg, values more than 3 times the interquartile range above the first quartile or below the third quartile).[27] Nonlinearity of financial performance measures was assessed using restricted cubic splines. For ease of interpretation, we scaled the estimated change in RSMR and RSRR per $50 million increase in net revenue from operations, and graphed nonparametric relationships using restricted cubic splines.

Effect of Public Reporting on Subsequent Hospital Financial Performance

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion hospital‐level difference‐in‐differences analysis to assess for differential changes in hospital financial performance between 2008 and 2012, stratified by tertiles of RSMR and RSRR rates from 2008 to 2011. This approach compares differences in an outcome of interest (hospital financial performance) within each group (where each group is a tertile of publicly reported rates of RSMR or RSRR), and then compares the difference in these differences between groups. Therefore, these analyses use each group as their own historical control and the opposite group as a concurrent control to account for potential secular trends. To conduct our difference‐in‐differences analysis, we compared the change in financial performance over time in the top tertile of hospitals to the change in financial performance over time in the bottom tertile of hospitals with respect to AMI, CHF, and PNA RSMR and RSRR. Our models therefore included year (2008 vs 2012), tertile of publicly reported rates for RSMR or RSRR, and the interaction between them as predictors, where the interaction was the difference‐in‐differences term and the primary predictor of interest. In addition to adjusting for hospital characteristics and accounting for clustering as mentioned above, we also included 3 separate interaction terms for year with bed size, safety‐net status, and Medicare caseload, to account for potential changes in the hospitals over time that may have independently influenced financial performance and publicly reported 30‐day measures. For sensitivity analyses, we repeated our difference‐in‐differences analyses excluding hospitals with a change in ownership and extreme outliers with respect to financial performance in 2008. We performed model diagnostics including assessment of functional form, linearity, normality, constant variance, and model misspecification. All analyses were conducted using Stata version 12.1 (StataCorp, College Station, TX). This study was deemed exempt from review by the UT Southwestern Medical Center institutional review board.

RESULTS

Among the 279 included hospitals (see Supporting Figure 1 in the online version of this article), 278 also had financial data available for 2012. In 2008, the median net revenue from operations was $1.6 million (interquartile range [IQR], $2.4 to $10.3 million), the median operating margin was 1.5% (IQR, 4.6% to 6%), and the median total margin was 2.5% (IQR, 2.2% to 7.5% (Table 1). The number of hospitals reporting each outcome, and median outcome rates, are shown in Table 2.

Hospital Characteristics and Financial Performance in 2008 and 2012
2008, n = 279 2012, n = 278
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation. *Medicaid caseload equivalent to 1 standard deviation above the mean (41.8% for 2008 and 42.1% for 2012). Operated by an investor‐individual, investor‐partnership, or investor‐corporation.

Hospital characteristics
Teaching, n (%) 28 (10.0) 28 (10.0)
Rural, n (%) 55 (19.7) 55 (19.7)
Bed size, n (%)
099 (small) 57 (20.4) 55 (19.8)
100299 (medium) 130 (46.6) 132 (47.5)
300 (large) 92 (33.0) 91 (32.7)
Safety‐net hospital, n (%)* 46 (16.5) 48 (17.3)
Hospital ownership, n (%)
City or county 15 (5.4) 16 (5.8)
District 42 (15.1) 39 (14.0)
Investor 66 (23.7) 66 (23.7)
Private nonprofit 156 (55.9) 157 (56.5)
Medicare caseload, mean % (SD) 41.6 (14.7) 43.6 (14.7)
Financial performance measures
Net revenue from operations, median $ in millions (IQR; range) 1.6 (2.4 to 10.3; 495.9 to 144.1) 3.2 (2.9 to 15.4; 396.2 to 276.8)
Operating margin, median % (IQR; range) 1.5 (4.6 to 6.8; 77.8 to 26.4) 2.3 (3.9 to 8.2; 134.8 to 21.1)
Total margin, median % (IQR; range) 2.5 (2.2 to 7.5; 101.0 to 26.3) 4.5 (0.7 to 9.8; 132.2 to 31.1)
Relationship Between Hospital Financial Performance and 30‐Day Mortality and Readmission Rates*
No. Median % (IQR) Adjusted % Change (95% CI) per $50 Million Increase in Net Revenue From Operations
Overall Extreme Outliers Excluded
  • NOTE: Abbreviations: CI, confidence interval; IQR, interquartile range. *Thirty‐day outcomes are risk standardized for age, sex, comorbidity count, and indicators of patient frailty.[3] Each outcome was modeled separately and adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. Twenty‐three hospitals were identified as extreme outliers with respect to net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million). There was a nonlinear and statistically significant relationship between net revenue from operations and readmission rate for myocardial infarction. Net revenue from operations was modeled as a cubic spline function. See Figure 1. The overall adjusted F statistic was 4.8 (P < 0.001). There was a nonlinear and statistically significant relationship between net revenue from operations and mortality rate for heart failure after exclusion of extreme outliers. Net revenue from operations was modeled as a cubic spline function. See Figure 1.The overall adjusted F statistic was 3.6 (P = 0.008).

Myocardial infarction
Mortality rate 211 15.2 (14.216.2) 0.07 (0.10 to 0.24) 0.63 (0.21 to 1.48)
Readmission rate 184 19.4 (18.520.2) Nonlinear 0.34 (1.17 to 0.50)
Congestive heart failure
Mortality rate 259 11.1 (10.112.1) 0.17 (0.01 to 0.35) Nonlinear
Readmission rate 264 24.5 (23.525.6) 0.07 (0.27 to 0.14) 0.45 (1.36 to 0.47)
Pneumonia
Mortality rate 268 11.6 (10.413.2) 0.17 (0.42 to 0.07) 0.35 (1.19 to 0.49)
Readmission rate 268 18.2 (17.319.1) 0.04 (0.20 to 0.11) 0.56 (1.27 to 0.16)

Relationship Between Financial Performance and Publicly Reported Outcomes

Acute Myocardial Infarction

We did not observe a consistent relationship between hospital financial performance and AMI mortality and readmission rates. In our overall adjusted analyses, net revenue from operations was not associated with mortality, but was significantly associated with a decrease in AMI readmissions among hospitals with net revenue from operations between approximately $5 million to $145 million (nonlinear relationship, F statistic = 4.8, P < 0.001 (Table 2, Figure 1A). However, after excluding 23 extreme outlying hospitals by net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million), this relationship was no longer observed. Using operating margin instead of net revenue from operations as the measure of hospital financial performance, we observed a 0.2% increase in AMI mortality (95% confidence interval [CI]: 0.06%‐0.35%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article) for each 10% increase in operating margin, which persisted with the exclusion of 5 outlying hospitals by operating margin (all 5 were underperformers, with operating margins <38.6%). However, using total margin as the measure of financial performance, there was no significant relationship with either mortality or readmissions (see Supporting Table 2 and Supporting Figure 3 in the online version of this article).

Figure 1
Relationship between financial performance and 30‐day readmission and mortality. The open circles represent individual hospitals. The bold dashed line and the bold solid line are the unadjusted and adjusted cubic spline curves, respectively, representing the nonlinear relationship between net revenue from operations and each outcome. The shaded grey area represents the 95% confidence interval for the adjusted cubic spline curve. Thin vertical dashed lines represent median values for net revenue from operations. Multivariate models were adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. *Twenty‐three hospitals were identified as outliers with respect to net revenue from clinical operations (10 “underperformers” with net revenue <−$49.4 million and 13 “overperformers” with net revenue >$52.1 million.

Congestive Heart Failure

In our primary analyses, we did not observe a significant relationship between net revenue from operations and CHF mortality and readmission rates. However, after excluding 23 extreme outliers, increasing net revenue from operations was associated with a modest increase in CHF mortality among hospitals, with net revenue between approximately $35 million and $20 million (nonlinear relationship, F statistic = 3.6, P = 0.008 (Table 2, Figure 1B). Using alternate measures of financial performance, we observed a consistent relationship between increasing hospital financial performance and higher 30‐day CHF mortality rate. Using operating margin, we observed a slight increase in the mortality rate for CHF (0.26% increase in CHF RSMR for every 10% increase in operating margin) (95% CI: 0.07%‐0.45%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article), which persisted after the exclusion of 5 extreme outliers. Using total margin, we observed a significant but modest association between improved hospital financial performance and increased mortality rate for CHF (nonlinear relationship, F statistic = 2.9, P = 0.03) (see Supporting Table 2 and Supporting Figure 3 in the online version of this article), which persisted after the exclusion of 3 extreme outliers (0.32% increase in CHF RSMR for every 10% increase in total margin) (95% CI: 0.03%‐0.62%).

Pneumonia

Hospital financial performance (using net revenue, operating margin, or total margin) was not associated with 30‐day PNA mortality or readmission rates.

Relationship of Readmission and Mortality Rates on Subsequent Hospital Financial Performance

Compared to hospitals in the highest tertile of readmission and mortality rates (ie, those with the worst rates), hospitals in the lowest tertile of readmission and mortality rates (ie, those with the best rates) had a similar magnitude of increase in net revenue from operations from 2008 to 2012 (Table 3). The difference‐in‐differences analyses showed no relationship between readmission or mortality rates for AMI, CHF, and PNA and changes in net revenue from operations from 2008 to 2012 (difference‐in‐differences estimates ranged from $8.61 to $6.77 million, P > 0.3 for all). These results were robust to the exclusion of hospitals with a change in ownership and extreme outliers by net revenue from operations (data not reported).

Difference in the Differences in Financial Performance Between the Worst‐ and the Best‐Performing Hospitals
Outcome Tertile With Highest Outcome Rates (Worst Hospitals) Tertile With Lowest Outcome Rates (Best Hospitals) Difference in Net From Operations Differences Between Highest and Lowest Outcome Rate Tertiles, $ Million (95% CI) P
Outcome, Median % (IQR) Gain/Loss in Net Revenue From Operations From 2008 to 2012, $ Million* Outcome, Median % (IQR) Gain/Loss in Net Revenue from Operations From 2008 to 2012, $ Million*
  • NOTE: Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; IQR, interquartile range; PNA, pneumonia. *Differences were calculated as net revenue from clinical operations in 2012 minus net revenue from clinical operations in 2008. Net revenue in 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers. Each outcome was modeled separately and adjusted for year, tertile of performance for the respective outcome, the interaction between year and tertile (difference‐in‐differences term), teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, volume of cases reported for the respective outcome, and interactions for year with bed size, safety‐net hospital status, and Medicare caseload, accounting for clustering of hospitals by owner.

AMI mortality 16.7 (16.217.4) +65.62 13.8 (13.314.2) +74.23 8.61 (27.95 to 10.73) 0.38
AMI readmit 20.7 (20.321.5) +38.62 18.3 (17.718.6) +31.85 +6.77 (13.24 to 26.77) 0.50
CHF mortality 13.0 (12.313.9) +45.66 9.6 (8.910.1) +48.60 2.94 (11.61 to 5.73) 0.50
CHF readmit 26.2 (25.726.9) +47.08 23.0 (22.323.5) +46.08 +0.99 (10.51 to 12.50) 0.87
PNA mortality 13.9 (13.314.7) +43.46 9.9 (9.310.4) +38.28 +5.18 (7.01 to 17.37) 0.40
PNA readmit 19.4 (19.120.1) +47.21 17.0 (16.517.3) +45.45 +1.76 (8.34 to 11.86) 0.73

DISCUSSION

Using audited financial data from California hospitals in 2008 and 2012, and CMS data on publicly reported outcomes from 2008 to 2011, we found no consistent relationship between hospital financial performance and publicly reported outcomes for AMI and PNA. However, better hospital financial performance was associated with a modest increase in 30‐day risk‐standardized CHF mortality rates, which was consistent across all 3 measures of hospital financial performance. Reassuringly, there was no difference in the change in net revenue from operations between 2008 and 2012 between hospitals in the highest and lowest tertiles of readmission and mortality rates for AMI, CHF, and PNA. In other words, hospitals with the lowest rates of 30‐day readmissions and mortality for AMI, CHF, and PNA did not experience a loss in net revenue from operations over time, compared to hospitals with the highest readmission and mortality rates.

Our study differs in several important ways from Ly et al., the only other study to our knowledge that investigated the relationship between hospital financial performance and outcomes for patients with AMI, CHF, and PNA.[19] First, outcomes in the Ly et al. study were ascertained in 2007, which preceded public reporting of outcomes. Second, the primary comparison was between hospitals in the bottom versus top decile of operating margin. Although Ly and colleagues also found no association between hospital financial performance and mortality rates for these 3 conditions, they found a significant absolute decrease of approximately 3% in readmission rates among hospitals in the top decile of operating margin versus those in bottom decile. However, readmission rates were comparable among the remaining 80% of hospitals, suggesting that these findings primarily reflected the influence of a few outlier hospitals. Third, the use of nonuniformly audited hospital financial data may have resulted in misclassification of financial performance. Our findings also differ from 2 previous studies that identified a modest association between improved hospital financial performance and decreased adverse patient safety events.[18, 20] However, publicly reported outcomes may not be fully representative of hospital quality and patient safety.[28, 29]

The limited association between hospital financial performance and publicly reported outcomes for AMI and PNA is noteworthy for several reasons. First, publicly reporting outcomes alone without concomitant changes to reimbursement may be inadequate to create strong financial incentives for hospital investment in quality improvement initiatives. Hospitals participating in both public reporting of outcomes and pay‐for‐performance have been shown to achieve greater improvements in outcomes than hospitals engaged only in public reporting.[30] Our time interval for ascertainment of outcomes preceded CMS implementation of the Hospital Readmissions Reduction Program (HRRP) in October 2012, which withholds up to 3% of Medicare hospital reimbursements for higher than expected mortality and readmission rates for AMI, CHF, and PNA. Once outcomes data become available for a 3‐year post‐HRRP implementation period, the impact of this combined approach can be assessed. Second, because adherence to many evidence‐based process measures for these conditions (ie, aspirin use in AMI) is already high, there may be a ceiling effect present that obviates the need for further hospital financial investment to optimize delivery of best practices.[31, 32] Third, hospitals themselves may contribute little to variation in mortality and readmission risk. Of the total variation in mortality and readmission rates among Texas Medicare beneficiaries, only about 1% is attributable to hospitals, whereas 42% to 56% of the variation is explained by differences in patient characteristics.[33, 34] Fourth, there is either low‐quality or insufficient evidence that transitional care interventions specifically targeted to patients with AMI or PNA result in better outcomes.[35] Thus, greater financial investment in hospital‐initiated and postdischarge transitional care interventions for these specific conditions may result in less than the desired effect. Lastly, many hospitalizations for these conditions are emergency hospitalizations that occur after patients present to the emergency department with unexpected and potentially life‐threatening symptoms. Thus, patients may not be able to incorporate the reputation or performance metrics of a hospital in their decisions for where they are hospitalized for AMI, CHF, or PNA despite the public reporting of outcomes.

Given the strong evidence that transitional care interventions reduce readmissions and mortality among patients hospitalized with CHF, we were surprised to find that improved hospital financial performance was associated with an increased risk‐adjusted CHF mortality rate.[36] This association held true for all 3 different measures of hospital financial performance, suggesting that this unexpected finding is unlikely to be the result of statistical chance, though potential reasons for this association remain unclear. One possibility is that the CMS model for CHF mortality may not adequately risk adjust for severity of illness.[37, 38] Thus, robust financial performance may be a marker for hospitals with more advanced heart failure services that care for more patients with severe illness.

Our findings should be interpreted in the context of certain limitations. Our study only included an analysis of outcomes for AMI, CHF, and PNA among older fee‐for‐service Medicare beneficiaries aggregated at the hospital level in California between 2008 and 2012, so generalizability to other populations, conditions, states, and time periods is uncertain. The observational design precludes a robust causal inference between financial performance and outcomes. For readmissions, rates were publicly reported for only the last 2 years of the 3‐year reporting period; thus, our findings may underestimate the association between hospital financial performance and publicly reported readmission rates.

CONCLUSION

There is no consistent relationship between hospital financial performance and subsequent publicly reported outcomes for AMI and PNA. However, for unclear reasons, hospitals with better financial performance had modestly higher CHF mortality rates. Given this limited association, public reporting of outcomes may have had less than the intended impact in motivating hospitals to invest in quality improvement. Additional financial incentives in addition to public reporting, such as readmissions penalties, may help motivate hospitals with robust financial performance to further improve outcomes. This would be a key area for future investigation once outcomes data are available for the 3‐year period following CMS implementation of readmissions penalties in 2012. Reassuringly, there was no association between low 30‐day mortality and readmissions rates and subsequent poor financial performance, suggesting that improved outcomes do not necessarily lead to loss of revenue.

Disclosures

Drs. Nguyen, Halm, and Makam were supported in part by the Agency for Healthcare Research and Quality University of Texas Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01). Drs. Nguyen and Makam received funding from the University of Texas Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

Files
References
  1. Centers for Medicare and Medicaid Services. Office of the Actuary. National Health Statistics Group. National Healthcare Expenditures Data. Baltimore, MD; 2013. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NationalHealthAccountsHistorical.html. Accessed February 16, 2016.
  2. Brill S. Bitter pill: why medical bills are killing us. Time Magazine. February 20, 2013:1655.
  3. Ginsburg PB. Fee‐for‐service will remain a feature of major payment reforms, requiring more changes in Medicare physician payment. Health Aff (Millwood). 2012;31(9):19771983.
  4. Leatherman S, Berwick D, Iles D, et al. The business case for quality: case studies and an analysis. Health Aff (Millwood). 2003;22(2):1730.
  5. Marshall MN, Shekelle PG, Davies HT, Smith PC. Public reporting on quality in the United States and the United Kingdom. Health Aff (Millwood). 2003;22(3):134148.
  6. Swensen SJ, Dilling JA, Mc Carty PM, Bolton JW, Harper CM. The business case for health‐care quality improvement. J Patient Saf. 2013;9(1):4452.
  7. Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood). 2005;24(4):11501160.
  8. Rauh SS, Wadsworth EB, Weeks WB, Weinstein JN. The savings illusion—why clinical quality improvement fails to deliver bottom‐line results. N Engl J Med. 2011;365(26):e48.
  9. Meyer JA, Silow‐Carroll S, Kutyla T, Stepnick LS, Rybowski LS. Hospital Quality: Ingredients for Success—Overview and Lessons Learned. New York, NY: Commonwealth Fund; 2004.
  10. Silow‐Carroll S, Alteras T, Meyer JA. Hospital Quality Improvement: Strategies and Lessons from U.S. Hospitals. New York, NY: Commonwealth Fund; 2007.
  11. Bazzoli GJ, Clement JP, Lindrooth RC, et al. Hospital financial condition and operational decisions related to the quality of hospital care. Med Care Res Rev. 2007;64(2):148168.
  12. Casalino LP, Elster A, Eisenberg A, Lewis E, Montgomery J, Ramos D. Will pay‐for‐performance and quality reporting affect health care disparities? Health Aff (Millwood). 2007;26(3):w405w414.
  13. Werner RM, Goldman LE, Dudley RA. Comparison of change in quality of care between safety‐net and non‐safety‐net hospitals. JAMA. 2008;299(18):21802187.
  14. Bhalla R, Kalkut G. Could Medicare readmission policy exacerbate health care system inequity? Ann Intern Med. 2010;152(2):114117.
  15. Hernandez AF, Curtis LH. Minding the gap between efforts to reduce readmissions and disparities. JAMA. 2011;305(7):715716.
  16. Terry DF, Moisuk S. Medicare Health Support Pilot Program. N Engl J Med. 2012;366(7):666; author reply 667–668.
  17. Fontanarosa PB, McNutt RA. Revisiting hospital readmissions. JAMA. 2013;309(4):398400.
  18. Encinosa WE, Bernard DM. Hospital finances and patient safety outcomes. Inquiry. 2005;42(1):6072.
  19. Ly DP, Jha AK, Epstein AM. The association between hospital margins, quality of care, and closure or other change in operating status. J Gen Intern Med. 2011;26(11):12911296.
  20. Bazzoli GJ, Chen HF, Zhao M, Lindrooth RC. Hospital financial condition and the quality of patient care. Health Econ. 2008;17(8):977995.
  21. State of California Office of Statewide Health Planning and Development. Healthcare Information Division. Annual financial data. Available at: http://www.oshpd.ca.gov/HID/Products/Hospitals/AnnFinanData/PivotProfles/default.asp. Accessed June 23, 2015.
  22. Centers for Medicare 4(1):1113.
  23. Ross JS, Cha SS, Epstein AJ, et al. Quality of care for acute myocardial infarction at urban safety‐net hospitals. Health Aff (Millwood). 2007;26(1):238248.
  24. Silber JH, Rosenbaum PR, Brachet TJ, et al. The Hospital Compare mortality model and the volume‐outcome relationship. Health Serv Res. 2010;45(5 Pt 1):11481167.
  25. Tukey J. Exploratory Data Analysis. Boston, MA: Addison‐Wesley; 1977.
  26. Press MJ, Scanlon DP, Ryan AM, et al. Limits of readmission rates in measuring hospital quality suggest the need for added metrics. Health Aff (Millwood). 2013;32(6):10831091.
  27. Stefan MS, Pekow PS, Nsa W, et al. Hospital performance measures and 30‐day readmission rates. J Gen Intern Med. 2013;28(3):377385.
  28. Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007;356(5):486496.
  29. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862869.
  30. Werner RM, Bradlow ET. Public reporting on hospital process improvements is linked to better patient outcomes. Health Aff (Millwood). 2010;29(7):13191324.
  31. Goodwin JS, Lin YL, Singh S, Kuo YF. Variation in length of stay and outcomes among hospitalized patients attributable to hospitals and hospitalists. J Gen Intern Med. 2013;28(3):370376.
  32. Singh S, Lin YL, Kuo YF, Nattinger AB, Goodwin JS. Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics. J Gen Intern Med. 2014;29(4):572578.
  33. Prvu Bettger J, Alexander KP, Dolor RJ, et al. Transitional care after hospitalization for acute stroke or myocardial infarction: a systematic review. Ann Intern Med. 2012;157(6):407416.
  34. Jha AK, Orav EJ, Li Z, Epstein AM. The inverse relationship between mortality rates and performance in the Hospital Quality Alliance measures. Health Aff (Millwood). 2007;26(4):11041110.
  35. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  36. Fuller RL, Atkinson G, Hughes JS. Indications of biased risk adjustment in the hospital readmission reduction program. J Ambul Care Manage. 2015;38(1):3947.
Article PDF
Issue
Journal of Hospital Medicine - 11(7)
Publications
Page Number
481-488
Sections
Files
Files
Article PDF
Article PDF

Hospital care accounts for the single largest category of national healthcare expenditures, totaling $936.9 billion in 2013.[1] With ongoing scrutiny of US healthcare spending, hospitals are under increasing pressure to justify high costs and robust profits.[2] However, the dominant fee‐for‐service reimbursement model creates incentives for hospitals to prioritize high volume over high‐quality care to maximize profits.[3] Because hospitals may be reluctant to implement improvements if better quality is not accompanied by better payment or improved financial margins, an approach to stimulate quality improvement among hospitals has been to leverage consumer pressure through required public reporting of selected outcome metrics.[4, 5] Public reporting of outcomes is thought to influence hospital reputation; in turn, reputation affects patient perceptions and influences demand for hospital services, potentially enabling reputable hospitals to command higher prices for services to enhance hospital revenue.[6, 7]

Though improving outcomes is thought to reduce overall healthcare costs, it is unclear whether improving outcomes results in a hospital's financial return on investment.[4, 5, 8] Quality improvement can require substantial upfront investment, requiring that hospitals already have robust financial health to engage in such initiatives.[9, 10] Consequently, instead of stimulating broad efforts in quality improvement, public reporting may exacerbate existing disparities in hospital quality and finances, by rewarding already financially healthy hospitals, and by inadvertently penalizing hospitals without the means to invest in quality improvement.[11, 12, 13, 14, 15] Alternately, because fee‐for‐service remains the dominant reimbursement model for hospitals, loss of revenue through reducing readmissions may outweigh any financial gains from improved public reputation and result in worse overall financial performance, though robust evidence for this concern is lacking.[16, 17]

A small number of existing studies suggest a limited correlation between improved hospital financial performance and improved quality, patient safety, and lower readmission rates.[18, 19, 20] However, these studies had several limitations. They were conducted prior to public reporting of selected outcome metrics by the Centers for Medicare and Medicaid Services (CMS)[18, 19, 20]; used data from the Medicare Cost Report, which is not uniformly audited and thus prone to measurement error[19, 20]; used only relative measures of hospital financial performance (eg, operating margin), which do not capture the absolute amount of revenue potentially available for investment in quality improvement[18, 19]; or compared only hospitals at the extremes of financial performance, potentially exaggerating the magnitude of the relationship between hospital financial performance and quality outcomes.[19]

To address this gap in the literature, we sought to assess whether hospitals with robust financial performance have lower 30‐day risk‐standardized mortality and hospital readmission rates for acute myocardial infarction (AMI), congestive heart failure (CHF), and pneumonia (PNA). Given the concern that hospitals with the lowest mortality and readmission rates may experience a decrease in financial performance due to the lower volume of hospitalizations, we also assessed whether hospitals with the lowest readmission and mortality rates had a differential change in financial performance over time compared to hospitals with the highest rates.

METHODS

Data Sources and Study Population

This was an observational study using audited financial data from the 2008 and 2012 Hospital Annual Financial Data Files from the Office of Statewide Health Planning and Development (OSHPD) in the state of California, merged with data on outcome measures publicly reported by CMS via the Hospital Compare website for July 1, 2008 to June 30, 2011.[21, 22] We included all general acute care hospitals with available OSHPD data in 2008 and at least 1 publicly reported outcome from 2008 to 2011. We excluded hospitals without 1 year of audited financial data for 2008 and hospitals that closed during 2008 to 2011.

Measures of Financial Performance

Because we hypothesized that the absolute amount of revenue generated from clinical operations would influence investment in quality improvement programs more so than relative changes in revenue,[20] we used net revenue from operations (total operating revenue minus total operating expense) as our primary measure of hospital financial performance. We also performed 2 companion analyses using 2 commonly reported relative measures of financial performanceoperating margin (net revenue from operations divided by total operating revenue) and total margin (net total revenue divided by total revenue from all sources). Net revenue from operations for 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers.

Outcomes

For our primary analysis, the primary outcomes were publicly reported all‐cause 30‐day risk‐standardized mortality rates (RSMR) and readmission rates (RSRR) for AMI, CHF, and PNA aggregated over a 3‐year period. These measures were adjusted for key demographic and clinical characteristics available in Medicare data. CMS began publicly reporting 30‐day RSMR for AMI and CHF in June 2007, RSMR for PNA in June 2008, and RSRR for all 3 conditions in July 2009.[23, 24]

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion analysis where the primary outcome of interest was change in hospital financial performance over time, using the same definitions of financial performance outlined above. For this companion analysis, publicly reported 30‐day RSMR and RSRR for AMI, CHF, and PNA were assessed as predictors of subsequent financial performance.

Hospital Characteristics

Hospital characteristics were ascertained from the OSHPD data. Safety‐net status was defined as hospitals with an annual Medicaid caseload (number of Medicaid discharges divided by the total number of discharges) 1 standard deviation above the mean Medicaid caseload, as defined in previous studies.[25]

Statistical Analyses

Effect of Baseline Financial Performance on Subsequent Publicly Reported Outcomes

To estimate the relationship between baseline hospital financial performance in 2008 and subsequent RSMR and RSRR for AMI, CHF, and PNA from 2008 to 2011, we used linear regression adjusted for the following hospital characteristics: teaching status, rural location, bed size, safety‐net status, ownership, Medicare caseload, and volume of cases reported for the respective outcome. We accounted for clustering of hospitals by ownership. We adjusted for hospital volume of reported cases for each condition given that the risk‐standardization models used by CMS shrink outcomes for small hospitals to the mean, and therefore do not account for a potential volume‐outcome relationship.[26] We conducted a sensitivity analysis excluding hospitals at the extremes of financial performance, defined as hospitals with extreme outlier values for each financial performance measure (eg, values more than 3 times the interquartile range above the first quartile or below the third quartile).[27] Nonlinearity of financial performance measures was assessed using restricted cubic splines. For ease of interpretation, we scaled the estimated change in RSMR and RSRR per $50 million increase in net revenue from operations, and graphed nonparametric relationships using restricted cubic splines.

Effect of Public Reporting on Subsequent Hospital Financial Performance

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion hospital‐level difference‐in‐differences analysis to assess for differential changes in hospital financial performance between 2008 and 2012, stratified by tertiles of RSMR and RSRR rates from 2008 to 2011. This approach compares differences in an outcome of interest (hospital financial performance) within each group (where each group is a tertile of publicly reported rates of RSMR or RSRR), and then compares the difference in these differences between groups. Therefore, these analyses use each group as their own historical control and the opposite group as a concurrent control to account for potential secular trends. To conduct our difference‐in‐differences analysis, we compared the change in financial performance over time in the top tertile of hospitals to the change in financial performance over time in the bottom tertile of hospitals with respect to AMI, CHF, and PNA RSMR and RSRR. Our models therefore included year (2008 vs 2012), tertile of publicly reported rates for RSMR or RSRR, and the interaction between them as predictors, where the interaction was the difference‐in‐differences term and the primary predictor of interest. In addition to adjusting for hospital characteristics and accounting for clustering as mentioned above, we also included 3 separate interaction terms for year with bed size, safety‐net status, and Medicare caseload, to account for potential changes in the hospitals over time that may have independently influenced financial performance and publicly reported 30‐day measures. For sensitivity analyses, we repeated our difference‐in‐differences analyses excluding hospitals with a change in ownership and extreme outliers with respect to financial performance in 2008. We performed model diagnostics including assessment of functional form, linearity, normality, constant variance, and model misspecification. All analyses were conducted using Stata version 12.1 (StataCorp, College Station, TX). This study was deemed exempt from review by the UT Southwestern Medical Center institutional review board.

RESULTS

Among the 279 included hospitals (see Supporting Figure 1 in the online version of this article), 278 also had financial data available for 2012. In 2008, the median net revenue from operations was $1.6 million (interquartile range [IQR], $2.4 to $10.3 million), the median operating margin was 1.5% (IQR, 4.6% to 6%), and the median total margin was 2.5% (IQR, 2.2% to 7.5% (Table 1). The number of hospitals reporting each outcome, and median outcome rates, are shown in Table 2.

Hospital Characteristics and Financial Performance in 2008 and 2012
2008, n = 279 2012, n = 278
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation. *Medicaid caseload equivalent to 1 standard deviation above the mean (41.8% for 2008 and 42.1% for 2012). Operated by an investor‐individual, investor‐partnership, or investor‐corporation.

Hospital characteristics
Teaching, n (%) 28 (10.0) 28 (10.0)
Rural, n (%) 55 (19.7) 55 (19.7)
Bed size, n (%)
099 (small) 57 (20.4) 55 (19.8)
100299 (medium) 130 (46.6) 132 (47.5)
300 (large) 92 (33.0) 91 (32.7)
Safety‐net hospital, n (%)* 46 (16.5) 48 (17.3)
Hospital ownership, n (%)
City or county 15 (5.4) 16 (5.8)
District 42 (15.1) 39 (14.0)
Investor 66 (23.7) 66 (23.7)
Private nonprofit 156 (55.9) 157 (56.5)
Medicare caseload, mean % (SD) 41.6 (14.7) 43.6 (14.7)
Financial performance measures
Net revenue from operations, median $ in millions (IQR; range) 1.6 (2.4 to 10.3; 495.9 to 144.1) 3.2 (2.9 to 15.4; 396.2 to 276.8)
Operating margin, median % (IQR; range) 1.5 (4.6 to 6.8; 77.8 to 26.4) 2.3 (3.9 to 8.2; 134.8 to 21.1)
Total margin, median % (IQR; range) 2.5 (2.2 to 7.5; 101.0 to 26.3) 4.5 (0.7 to 9.8; 132.2 to 31.1)
Relationship Between Hospital Financial Performance and 30‐Day Mortality and Readmission Rates*
No. Median % (IQR) Adjusted % Change (95% CI) per $50 Million Increase in Net Revenue From Operations
Overall Extreme Outliers Excluded
  • NOTE: Abbreviations: CI, confidence interval; IQR, interquartile range. *Thirty‐day outcomes are risk standardized for age, sex, comorbidity count, and indicators of patient frailty.[3] Each outcome was modeled separately and adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. Twenty‐three hospitals were identified as extreme outliers with respect to net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million). There was a nonlinear and statistically significant relationship between net revenue from operations and readmission rate for myocardial infarction. Net revenue from operations was modeled as a cubic spline function. See Figure 1. The overall adjusted F statistic was 4.8 (P < 0.001). There was a nonlinear and statistically significant relationship between net revenue from operations and mortality rate for heart failure after exclusion of extreme outliers. Net revenue from operations was modeled as a cubic spline function. See Figure 1.The overall adjusted F statistic was 3.6 (P = 0.008).

Myocardial infarction
Mortality rate 211 15.2 (14.216.2) 0.07 (0.10 to 0.24) 0.63 (0.21 to 1.48)
Readmission rate 184 19.4 (18.520.2) Nonlinear 0.34 (1.17 to 0.50)
Congestive heart failure
Mortality rate 259 11.1 (10.112.1) 0.17 (0.01 to 0.35) Nonlinear
Readmission rate 264 24.5 (23.525.6) 0.07 (0.27 to 0.14) 0.45 (1.36 to 0.47)
Pneumonia
Mortality rate 268 11.6 (10.413.2) 0.17 (0.42 to 0.07) 0.35 (1.19 to 0.49)
Readmission rate 268 18.2 (17.319.1) 0.04 (0.20 to 0.11) 0.56 (1.27 to 0.16)

Relationship Between Financial Performance and Publicly Reported Outcomes

Acute Myocardial Infarction

We did not observe a consistent relationship between hospital financial performance and AMI mortality and readmission rates. In our overall adjusted analyses, net revenue from operations was not associated with mortality, but was significantly associated with a decrease in AMI readmissions among hospitals with net revenue from operations between approximately $5 million to $145 million (nonlinear relationship, F statistic = 4.8, P < 0.001 (Table 2, Figure 1A). However, after excluding 23 extreme outlying hospitals by net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million), this relationship was no longer observed. Using operating margin instead of net revenue from operations as the measure of hospital financial performance, we observed a 0.2% increase in AMI mortality (95% confidence interval [CI]: 0.06%‐0.35%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article) for each 10% increase in operating margin, which persisted with the exclusion of 5 outlying hospitals by operating margin (all 5 were underperformers, with operating margins <38.6%). However, using total margin as the measure of financial performance, there was no significant relationship with either mortality or readmissions (see Supporting Table 2 and Supporting Figure 3 in the online version of this article).

Figure 1
Relationship between financial performance and 30‐day readmission and mortality. The open circles represent individual hospitals. The bold dashed line and the bold solid line are the unadjusted and adjusted cubic spline curves, respectively, representing the nonlinear relationship between net revenue from operations and each outcome. The shaded grey area represents the 95% confidence interval for the adjusted cubic spline curve. Thin vertical dashed lines represent median values for net revenue from operations. Multivariate models were adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. *Twenty‐three hospitals were identified as outliers with respect to net revenue from clinical operations (10 “underperformers” with net revenue <−$49.4 million and 13 “overperformers” with net revenue >$52.1 million.

Congestive Heart Failure

In our primary analyses, we did not observe a significant relationship between net revenue from operations and CHF mortality and readmission rates. However, after excluding 23 extreme outliers, increasing net revenue from operations was associated with a modest increase in CHF mortality among hospitals, with net revenue between approximately $35 million and $20 million (nonlinear relationship, F statistic = 3.6, P = 0.008 (Table 2, Figure 1B). Using alternate measures of financial performance, we observed a consistent relationship between increasing hospital financial performance and higher 30‐day CHF mortality rate. Using operating margin, we observed a slight increase in the mortality rate for CHF (0.26% increase in CHF RSMR for every 10% increase in operating margin) (95% CI: 0.07%‐0.45%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article), which persisted after the exclusion of 5 extreme outliers. Using total margin, we observed a significant but modest association between improved hospital financial performance and increased mortality rate for CHF (nonlinear relationship, F statistic = 2.9, P = 0.03) (see Supporting Table 2 and Supporting Figure 3 in the online version of this article), which persisted after the exclusion of 3 extreme outliers (0.32% increase in CHF RSMR for every 10% increase in total margin) (95% CI: 0.03%‐0.62%).

Pneumonia

Hospital financial performance (using net revenue, operating margin, or total margin) was not associated with 30‐day PNA mortality or readmission rates.

Relationship of Readmission and Mortality Rates on Subsequent Hospital Financial Performance

Compared to hospitals in the highest tertile of readmission and mortality rates (ie, those with the worst rates), hospitals in the lowest tertile of readmission and mortality rates (ie, those with the best rates) had a similar magnitude of increase in net revenue from operations from 2008 to 2012 (Table 3). The difference‐in‐differences analyses showed no relationship between readmission or mortality rates for AMI, CHF, and PNA and changes in net revenue from operations from 2008 to 2012 (difference‐in‐differences estimates ranged from $8.61 to $6.77 million, P > 0.3 for all). These results were robust to the exclusion of hospitals with a change in ownership and extreme outliers by net revenue from operations (data not reported).

Difference in the Differences in Financial Performance Between the Worst‐ and the Best‐Performing Hospitals
Outcome Tertile With Highest Outcome Rates (Worst Hospitals) Tertile With Lowest Outcome Rates (Best Hospitals) Difference in Net From Operations Differences Between Highest and Lowest Outcome Rate Tertiles, $ Million (95% CI) P
Outcome, Median % (IQR) Gain/Loss in Net Revenue From Operations From 2008 to 2012, $ Million* Outcome, Median % (IQR) Gain/Loss in Net Revenue from Operations From 2008 to 2012, $ Million*
  • NOTE: Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; IQR, interquartile range; PNA, pneumonia. *Differences were calculated as net revenue from clinical operations in 2012 minus net revenue from clinical operations in 2008. Net revenue in 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers. Each outcome was modeled separately and adjusted for year, tertile of performance for the respective outcome, the interaction between year and tertile (difference‐in‐differences term), teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, volume of cases reported for the respective outcome, and interactions for year with bed size, safety‐net hospital status, and Medicare caseload, accounting for clustering of hospitals by owner.

AMI mortality 16.7 (16.217.4) +65.62 13.8 (13.314.2) +74.23 8.61 (27.95 to 10.73) 0.38
AMI readmit 20.7 (20.321.5) +38.62 18.3 (17.718.6) +31.85 +6.77 (13.24 to 26.77) 0.50
CHF mortality 13.0 (12.313.9) +45.66 9.6 (8.910.1) +48.60 2.94 (11.61 to 5.73) 0.50
CHF readmit 26.2 (25.726.9) +47.08 23.0 (22.323.5) +46.08 +0.99 (10.51 to 12.50) 0.87
PNA mortality 13.9 (13.314.7) +43.46 9.9 (9.310.4) +38.28 +5.18 (7.01 to 17.37) 0.40
PNA readmit 19.4 (19.120.1) +47.21 17.0 (16.517.3) +45.45 +1.76 (8.34 to 11.86) 0.73

DISCUSSION

Using audited financial data from California hospitals in 2008 and 2012, and CMS data on publicly reported outcomes from 2008 to 2011, we found no consistent relationship between hospital financial performance and publicly reported outcomes for AMI and PNA. However, better hospital financial performance was associated with a modest increase in 30‐day risk‐standardized CHF mortality rates, which was consistent across all 3 measures of hospital financial performance. Reassuringly, there was no difference in the change in net revenue from operations between 2008 and 2012 between hospitals in the highest and lowest tertiles of readmission and mortality rates for AMI, CHF, and PNA. In other words, hospitals with the lowest rates of 30‐day readmissions and mortality for AMI, CHF, and PNA did not experience a loss in net revenue from operations over time, compared to hospitals with the highest readmission and mortality rates.

Our study differs in several important ways from Ly et al., the only other study to our knowledge that investigated the relationship between hospital financial performance and outcomes for patients with AMI, CHF, and PNA.[19] First, outcomes in the Ly et al. study were ascertained in 2007, which preceded public reporting of outcomes. Second, the primary comparison was between hospitals in the bottom versus top decile of operating margin. Although Ly and colleagues also found no association between hospital financial performance and mortality rates for these 3 conditions, they found a significant absolute decrease of approximately 3% in readmission rates among hospitals in the top decile of operating margin versus those in bottom decile. However, readmission rates were comparable among the remaining 80% of hospitals, suggesting that these findings primarily reflected the influence of a few outlier hospitals. Third, the use of nonuniformly audited hospital financial data may have resulted in misclassification of financial performance. Our findings also differ from 2 previous studies that identified a modest association between improved hospital financial performance and decreased adverse patient safety events.[18, 20] However, publicly reported outcomes may not be fully representative of hospital quality and patient safety.[28, 29]

The limited association between hospital financial performance and publicly reported outcomes for AMI and PNA is noteworthy for several reasons. First, publicly reporting outcomes alone without concomitant changes to reimbursement may be inadequate to create strong financial incentives for hospital investment in quality improvement initiatives. Hospitals participating in both public reporting of outcomes and pay‐for‐performance have been shown to achieve greater improvements in outcomes than hospitals engaged only in public reporting.[30] Our time interval for ascertainment of outcomes preceded CMS implementation of the Hospital Readmissions Reduction Program (HRRP) in October 2012, which withholds up to 3% of Medicare hospital reimbursements for higher than expected mortality and readmission rates for AMI, CHF, and PNA. Once outcomes data become available for a 3‐year post‐HRRP implementation period, the impact of this combined approach can be assessed. Second, because adherence to many evidence‐based process measures for these conditions (ie, aspirin use in AMI) is already high, there may be a ceiling effect present that obviates the need for further hospital financial investment to optimize delivery of best practices.[31, 32] Third, hospitals themselves may contribute little to variation in mortality and readmission risk. Of the total variation in mortality and readmission rates among Texas Medicare beneficiaries, only about 1% is attributable to hospitals, whereas 42% to 56% of the variation is explained by differences in patient characteristics.[33, 34] Fourth, there is either low‐quality or insufficient evidence that transitional care interventions specifically targeted to patients with AMI or PNA result in better outcomes.[35] Thus, greater financial investment in hospital‐initiated and postdischarge transitional care interventions for these specific conditions may result in less than the desired effect. Lastly, many hospitalizations for these conditions are emergency hospitalizations that occur after patients present to the emergency department with unexpected and potentially life‐threatening symptoms. Thus, patients may not be able to incorporate the reputation or performance metrics of a hospital in their decisions for where they are hospitalized for AMI, CHF, or PNA despite the public reporting of outcomes.

Given the strong evidence that transitional care interventions reduce readmissions and mortality among patients hospitalized with CHF, we were surprised to find that improved hospital financial performance was associated with an increased risk‐adjusted CHF mortality rate.[36] This association held true for all 3 different measures of hospital financial performance, suggesting that this unexpected finding is unlikely to be the result of statistical chance, though potential reasons for this association remain unclear. One possibility is that the CMS model for CHF mortality may not adequately risk adjust for severity of illness.[37, 38] Thus, robust financial performance may be a marker for hospitals with more advanced heart failure services that care for more patients with severe illness.

Our findings should be interpreted in the context of certain limitations. Our study only included an analysis of outcomes for AMI, CHF, and PNA among older fee‐for‐service Medicare beneficiaries aggregated at the hospital level in California between 2008 and 2012, so generalizability to other populations, conditions, states, and time periods is uncertain. The observational design precludes a robust causal inference between financial performance and outcomes. For readmissions, rates were publicly reported for only the last 2 years of the 3‐year reporting period; thus, our findings may underestimate the association between hospital financial performance and publicly reported readmission rates.

CONCLUSION

There is no consistent relationship between hospital financial performance and subsequent publicly reported outcomes for AMI and PNA. However, for unclear reasons, hospitals with better financial performance had modestly higher CHF mortality rates. Given this limited association, public reporting of outcomes may have had less than the intended impact in motivating hospitals to invest in quality improvement. Additional financial incentives in addition to public reporting, such as readmissions penalties, may help motivate hospitals with robust financial performance to further improve outcomes. This would be a key area for future investigation once outcomes data are available for the 3‐year period following CMS implementation of readmissions penalties in 2012. Reassuringly, there was no association between low 30‐day mortality and readmissions rates and subsequent poor financial performance, suggesting that improved outcomes do not necessarily lead to loss of revenue.

Disclosures

Drs. Nguyen, Halm, and Makam were supported in part by the Agency for Healthcare Research and Quality University of Texas Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01). Drs. Nguyen and Makam received funding from the University of Texas Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

Hospital care accounts for the single largest category of national healthcare expenditures, totaling $936.9 billion in 2013.[1] With ongoing scrutiny of US healthcare spending, hospitals are under increasing pressure to justify high costs and robust profits.[2] However, the dominant fee‐for‐service reimbursement model creates incentives for hospitals to prioritize high volume over high‐quality care to maximize profits.[3] Because hospitals may be reluctant to implement improvements if better quality is not accompanied by better payment or improved financial margins, an approach to stimulate quality improvement among hospitals has been to leverage consumer pressure through required public reporting of selected outcome metrics.[4, 5] Public reporting of outcomes is thought to influence hospital reputation; in turn, reputation affects patient perceptions and influences demand for hospital services, potentially enabling reputable hospitals to command higher prices for services to enhance hospital revenue.[6, 7]

Though improving outcomes is thought to reduce overall healthcare costs, it is unclear whether improving outcomes results in a hospital's financial return on investment.[4, 5, 8] Quality improvement can require substantial upfront investment, requiring that hospitals already have robust financial health to engage in such initiatives.[9, 10] Consequently, instead of stimulating broad efforts in quality improvement, public reporting may exacerbate existing disparities in hospital quality and finances, by rewarding already financially healthy hospitals, and by inadvertently penalizing hospitals without the means to invest in quality improvement.[11, 12, 13, 14, 15] Alternately, because fee‐for‐service remains the dominant reimbursement model for hospitals, loss of revenue through reducing readmissions may outweigh any financial gains from improved public reputation and result in worse overall financial performance, though robust evidence for this concern is lacking.[16, 17]

A small number of existing studies suggest a limited correlation between improved hospital financial performance and improved quality, patient safety, and lower readmission rates.[18, 19, 20] However, these studies had several limitations. They were conducted prior to public reporting of selected outcome metrics by the Centers for Medicare and Medicaid Services (CMS)[18, 19, 20]; used data from the Medicare Cost Report, which is not uniformly audited and thus prone to measurement error[19, 20]; used only relative measures of hospital financial performance (eg, operating margin), which do not capture the absolute amount of revenue potentially available for investment in quality improvement[18, 19]; or compared only hospitals at the extremes of financial performance, potentially exaggerating the magnitude of the relationship between hospital financial performance and quality outcomes.[19]

To address this gap in the literature, we sought to assess whether hospitals with robust financial performance have lower 30‐day risk‐standardized mortality and hospital readmission rates for acute myocardial infarction (AMI), congestive heart failure (CHF), and pneumonia (PNA). Given the concern that hospitals with the lowest mortality and readmission rates may experience a decrease in financial performance due to the lower volume of hospitalizations, we also assessed whether hospitals with the lowest readmission and mortality rates had a differential change in financial performance over time compared to hospitals with the highest rates.

METHODS

Data Sources and Study Population

This was an observational study using audited financial data from the 2008 and 2012 Hospital Annual Financial Data Files from the Office of Statewide Health Planning and Development (OSHPD) in the state of California, merged with data on outcome measures publicly reported by CMS via the Hospital Compare website for July 1, 2008 to June 30, 2011.[21, 22] We included all general acute care hospitals with available OSHPD data in 2008 and at least 1 publicly reported outcome from 2008 to 2011. We excluded hospitals without 1 year of audited financial data for 2008 and hospitals that closed during 2008 to 2011.

Measures of Financial Performance

Because we hypothesized that the absolute amount of revenue generated from clinical operations would influence investment in quality improvement programs more so than relative changes in revenue,[20] we used net revenue from operations (total operating revenue minus total operating expense) as our primary measure of hospital financial performance. We also performed 2 companion analyses using 2 commonly reported relative measures of financial performanceoperating margin (net revenue from operations divided by total operating revenue) and total margin (net total revenue divided by total revenue from all sources). Net revenue from operations for 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers.

Outcomes

For our primary analysis, the primary outcomes were publicly reported all‐cause 30‐day risk‐standardized mortality rates (RSMR) and readmission rates (RSRR) for AMI, CHF, and PNA aggregated over a 3‐year period. These measures were adjusted for key demographic and clinical characteristics available in Medicare data. CMS began publicly reporting 30‐day RSMR for AMI and CHF in June 2007, RSMR for PNA in June 2008, and RSRR for all 3 conditions in July 2009.[23, 24]

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion analysis where the primary outcome of interest was change in hospital financial performance over time, using the same definitions of financial performance outlined above. For this companion analysis, publicly reported 30‐day RSMR and RSRR for AMI, CHF, and PNA were assessed as predictors of subsequent financial performance.

Hospital Characteristics

Hospital characteristics were ascertained from the OSHPD data. Safety‐net status was defined as hospitals with an annual Medicaid caseload (number of Medicaid discharges divided by the total number of discharges) 1 standard deviation above the mean Medicaid caseload, as defined in previous studies.[25]

Statistical Analyses

Effect of Baseline Financial Performance on Subsequent Publicly Reported Outcomes

To estimate the relationship between baseline hospital financial performance in 2008 and subsequent RSMR and RSRR for AMI, CHF, and PNA from 2008 to 2011, we used linear regression adjusted for the following hospital characteristics: teaching status, rural location, bed size, safety‐net status, ownership, Medicare caseload, and volume of cases reported for the respective outcome. We accounted for clustering of hospitals by ownership. We adjusted for hospital volume of reported cases for each condition given that the risk‐standardization models used by CMS shrink outcomes for small hospitals to the mean, and therefore do not account for a potential volume‐outcome relationship.[26] We conducted a sensitivity analysis excluding hospitals at the extremes of financial performance, defined as hospitals with extreme outlier values for each financial performance measure (eg, values more than 3 times the interquartile range above the first quartile or below the third quartile).[27] Nonlinearity of financial performance measures was assessed using restricted cubic splines. For ease of interpretation, we scaled the estimated change in RSMR and RSRR per $50 million increase in net revenue from operations, and graphed nonparametric relationships using restricted cubic splines.

Effect of Public Reporting on Subsequent Hospital Financial Performance

To assess whether public reporting had an effect on subsequent hospital financial performance, we conducted a companion hospital‐level difference‐in‐differences analysis to assess for differential changes in hospital financial performance between 2008 and 2012, stratified by tertiles of RSMR and RSRR rates from 2008 to 2011. This approach compares differences in an outcome of interest (hospital financial performance) within each group (where each group is a tertile of publicly reported rates of RSMR or RSRR), and then compares the difference in these differences between groups. Therefore, these analyses use each group as their own historical control and the opposite group as a concurrent control to account for potential secular trends. To conduct our difference‐in‐differences analysis, we compared the change in financial performance over time in the top tertile of hospitals to the change in financial performance over time in the bottom tertile of hospitals with respect to AMI, CHF, and PNA RSMR and RSRR. Our models therefore included year (2008 vs 2012), tertile of publicly reported rates for RSMR or RSRR, and the interaction between them as predictors, where the interaction was the difference‐in‐differences term and the primary predictor of interest. In addition to adjusting for hospital characteristics and accounting for clustering as mentioned above, we also included 3 separate interaction terms for year with bed size, safety‐net status, and Medicare caseload, to account for potential changes in the hospitals over time that may have independently influenced financial performance and publicly reported 30‐day measures. For sensitivity analyses, we repeated our difference‐in‐differences analyses excluding hospitals with a change in ownership and extreme outliers with respect to financial performance in 2008. We performed model diagnostics including assessment of functional form, linearity, normality, constant variance, and model misspecification. All analyses were conducted using Stata version 12.1 (StataCorp, College Station, TX). This study was deemed exempt from review by the UT Southwestern Medical Center institutional review board.

RESULTS

Among the 279 included hospitals (see Supporting Figure 1 in the online version of this article), 278 also had financial data available for 2012. In 2008, the median net revenue from operations was $1.6 million (interquartile range [IQR], $2.4 to $10.3 million), the median operating margin was 1.5% (IQR, 4.6% to 6%), and the median total margin was 2.5% (IQR, 2.2% to 7.5% (Table 1). The number of hospitals reporting each outcome, and median outcome rates, are shown in Table 2.

Hospital Characteristics and Financial Performance in 2008 and 2012
2008, n = 279 2012, n = 278
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation. *Medicaid caseload equivalent to 1 standard deviation above the mean (41.8% for 2008 and 42.1% for 2012). Operated by an investor‐individual, investor‐partnership, or investor‐corporation.

Hospital characteristics
Teaching, n (%) 28 (10.0) 28 (10.0)
Rural, n (%) 55 (19.7) 55 (19.7)
Bed size, n (%)
099 (small) 57 (20.4) 55 (19.8)
100299 (medium) 130 (46.6) 132 (47.5)
300 (large) 92 (33.0) 91 (32.7)
Safety‐net hospital, n (%)* 46 (16.5) 48 (17.3)
Hospital ownership, n (%)
City or county 15 (5.4) 16 (5.8)
District 42 (15.1) 39 (14.0)
Investor 66 (23.7) 66 (23.7)
Private nonprofit 156 (55.9) 157 (56.5)
Medicare caseload, mean % (SD) 41.6 (14.7) 43.6 (14.7)
Financial performance measures
Net revenue from operations, median $ in millions (IQR; range) 1.6 (2.4 to 10.3; 495.9 to 144.1) 3.2 (2.9 to 15.4; 396.2 to 276.8)
Operating margin, median % (IQR; range) 1.5 (4.6 to 6.8; 77.8 to 26.4) 2.3 (3.9 to 8.2; 134.8 to 21.1)
Total margin, median % (IQR; range) 2.5 (2.2 to 7.5; 101.0 to 26.3) 4.5 (0.7 to 9.8; 132.2 to 31.1)
Relationship Between Hospital Financial Performance and 30‐Day Mortality and Readmission Rates*
No. Median % (IQR) Adjusted % Change (95% CI) per $50 Million Increase in Net Revenue From Operations
Overall Extreme Outliers Excluded
  • NOTE: Abbreviations: CI, confidence interval; IQR, interquartile range. *Thirty‐day outcomes are risk standardized for age, sex, comorbidity count, and indicators of patient frailty.[3] Each outcome was modeled separately and adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. Twenty‐three hospitals were identified as extreme outliers with respect to net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million). There was a nonlinear and statistically significant relationship between net revenue from operations and readmission rate for myocardial infarction. Net revenue from operations was modeled as a cubic spline function. See Figure 1. The overall adjusted F statistic was 4.8 (P < 0.001). There was a nonlinear and statistically significant relationship between net revenue from operations and mortality rate for heart failure after exclusion of extreme outliers. Net revenue from operations was modeled as a cubic spline function. See Figure 1.The overall adjusted F statistic was 3.6 (P = 0.008).

Myocardial infarction
Mortality rate 211 15.2 (14.216.2) 0.07 (0.10 to 0.24) 0.63 (0.21 to 1.48)
Readmission rate 184 19.4 (18.520.2) Nonlinear 0.34 (1.17 to 0.50)
Congestive heart failure
Mortality rate 259 11.1 (10.112.1) 0.17 (0.01 to 0.35) Nonlinear
Readmission rate 264 24.5 (23.525.6) 0.07 (0.27 to 0.14) 0.45 (1.36 to 0.47)
Pneumonia
Mortality rate 268 11.6 (10.413.2) 0.17 (0.42 to 0.07) 0.35 (1.19 to 0.49)
Readmission rate 268 18.2 (17.319.1) 0.04 (0.20 to 0.11) 0.56 (1.27 to 0.16)

Relationship Between Financial Performance and Publicly Reported Outcomes

Acute Myocardial Infarction

We did not observe a consistent relationship between hospital financial performance and AMI mortality and readmission rates. In our overall adjusted analyses, net revenue from operations was not associated with mortality, but was significantly associated with a decrease in AMI readmissions among hospitals with net revenue from operations between approximately $5 million to $145 million (nonlinear relationship, F statistic = 4.8, P < 0.001 (Table 2, Figure 1A). However, after excluding 23 extreme outlying hospitals by net revenue from operations (10 underperformers with net revenue <$49.4 million and 13 overperformers with net revenue >$52.1 million), this relationship was no longer observed. Using operating margin instead of net revenue from operations as the measure of hospital financial performance, we observed a 0.2% increase in AMI mortality (95% confidence interval [CI]: 0.06%‐0.35%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article) for each 10% increase in operating margin, which persisted with the exclusion of 5 outlying hospitals by operating margin (all 5 were underperformers, with operating margins <38.6%). However, using total margin as the measure of financial performance, there was no significant relationship with either mortality or readmissions (see Supporting Table 2 and Supporting Figure 3 in the online version of this article).

Figure 1
Relationship between financial performance and 30‐day readmission and mortality. The open circles represent individual hospitals. The bold dashed line and the bold solid line are the unadjusted and adjusted cubic spline curves, respectively, representing the nonlinear relationship between net revenue from operations and each outcome. The shaded grey area represents the 95% confidence interval for the adjusted cubic spline curve. Thin vertical dashed lines represent median values for net revenue from operations. Multivariate models were adjusted for teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership, Medicare caseload, and volume of cases reported for the respective outcome, accounting for clustering of hospitals by owner. *Twenty‐three hospitals were identified as outliers with respect to net revenue from clinical operations (10 “underperformers” with net revenue <−$49.4 million and 13 “overperformers” with net revenue >$52.1 million.

Congestive Heart Failure

In our primary analyses, we did not observe a significant relationship between net revenue from operations and CHF mortality and readmission rates. However, after excluding 23 extreme outliers, increasing net revenue from operations was associated with a modest increase in CHF mortality among hospitals, with net revenue between approximately $35 million and $20 million (nonlinear relationship, F statistic = 3.6, P = 0.008 (Table 2, Figure 1B). Using alternate measures of financial performance, we observed a consistent relationship between increasing hospital financial performance and higher 30‐day CHF mortality rate. Using operating margin, we observed a slight increase in the mortality rate for CHF (0.26% increase in CHF RSMR for every 10% increase in operating margin) (95% CI: 0.07%‐0.45%) (see Supporting Table 1 and Supporting Figure 2 in the online version of this article), which persisted after the exclusion of 5 extreme outliers. Using total margin, we observed a significant but modest association between improved hospital financial performance and increased mortality rate for CHF (nonlinear relationship, F statistic = 2.9, P = 0.03) (see Supporting Table 2 and Supporting Figure 3 in the online version of this article), which persisted after the exclusion of 3 extreme outliers (0.32% increase in CHF RSMR for every 10% increase in total margin) (95% CI: 0.03%‐0.62%).

Pneumonia

Hospital financial performance (using net revenue, operating margin, or total margin) was not associated with 30‐day PNA mortality or readmission rates.

Relationship of Readmission and Mortality Rates on Subsequent Hospital Financial Performance

Compared to hospitals in the highest tertile of readmission and mortality rates (ie, those with the worst rates), hospitals in the lowest tertile of readmission and mortality rates (ie, those with the best rates) had a similar magnitude of increase in net revenue from operations from 2008 to 2012 (Table 3). The difference‐in‐differences analyses showed no relationship between readmission or mortality rates for AMI, CHF, and PNA and changes in net revenue from operations from 2008 to 2012 (difference‐in‐differences estimates ranged from $8.61 to $6.77 million, P > 0.3 for all). These results were robust to the exclusion of hospitals with a change in ownership and extreme outliers by net revenue from operations (data not reported).

Difference in the Differences in Financial Performance Between the Worst‐ and the Best‐Performing Hospitals
Outcome Tertile With Highest Outcome Rates (Worst Hospitals) Tertile With Lowest Outcome Rates (Best Hospitals) Difference in Net From Operations Differences Between Highest and Lowest Outcome Rate Tertiles, $ Million (95% CI) P
Outcome, Median % (IQR) Gain/Loss in Net Revenue From Operations From 2008 to 2012, $ Million* Outcome, Median % (IQR) Gain/Loss in Net Revenue from Operations From 2008 to 2012, $ Million*
  • NOTE: Abbreviations: AMI, acute myocardial infarction; CHF, congestive heart failure; CI, confidence interval; IQR, interquartile range; PNA, pneumonia. *Differences were calculated as net revenue from clinical operations in 2012 minus net revenue from clinical operations in 2008. Net revenue in 2008 was adjusted to 2012 US dollars using the chained Consumer Price Index for all urban consumers. Each outcome was modeled separately and adjusted for year, tertile of performance for the respective outcome, the interaction between year and tertile (difference‐in‐differences term), teaching status, metropolitan status (urban vs rural), bed size, safety‐net hospital status, hospital ownership type, Medicare caseload, volume of cases reported for the respective outcome, and interactions for year with bed size, safety‐net hospital status, and Medicare caseload, accounting for clustering of hospitals by owner.

AMI mortality 16.7 (16.217.4) +65.62 13.8 (13.314.2) +74.23 8.61 (27.95 to 10.73) 0.38
AMI readmit 20.7 (20.321.5) +38.62 18.3 (17.718.6) +31.85 +6.77 (13.24 to 26.77) 0.50
CHF mortality 13.0 (12.313.9) +45.66 9.6 (8.910.1) +48.60 2.94 (11.61 to 5.73) 0.50
CHF readmit 26.2 (25.726.9) +47.08 23.0 (22.323.5) +46.08 +0.99 (10.51 to 12.50) 0.87
PNA mortality 13.9 (13.314.7) +43.46 9.9 (9.310.4) +38.28 +5.18 (7.01 to 17.37) 0.40
PNA readmit 19.4 (19.120.1) +47.21 17.0 (16.517.3) +45.45 +1.76 (8.34 to 11.86) 0.73

DISCUSSION

Using audited financial data from California hospitals in 2008 and 2012, and CMS data on publicly reported outcomes from 2008 to 2011, we found no consistent relationship between hospital financial performance and publicly reported outcomes for AMI and PNA. However, better hospital financial performance was associated with a modest increase in 30‐day risk‐standardized CHF mortality rates, which was consistent across all 3 measures of hospital financial performance. Reassuringly, there was no difference in the change in net revenue from operations between 2008 and 2012 between hospitals in the highest and lowest tertiles of readmission and mortality rates for AMI, CHF, and PNA. In other words, hospitals with the lowest rates of 30‐day readmissions and mortality for AMI, CHF, and PNA did not experience a loss in net revenue from operations over time, compared to hospitals with the highest readmission and mortality rates.

Our study differs in several important ways from Ly et al., the only other study to our knowledge that investigated the relationship between hospital financial performance and outcomes for patients with AMI, CHF, and PNA.[19] First, outcomes in the Ly et al. study were ascertained in 2007, which preceded public reporting of outcomes. Second, the primary comparison was between hospitals in the bottom versus top decile of operating margin. Although Ly and colleagues also found no association between hospital financial performance and mortality rates for these 3 conditions, they found a significant absolute decrease of approximately 3% in readmission rates among hospitals in the top decile of operating margin versus those in bottom decile. However, readmission rates were comparable among the remaining 80% of hospitals, suggesting that these findings primarily reflected the influence of a few outlier hospitals. Third, the use of nonuniformly audited hospital financial data may have resulted in misclassification of financial performance. Our findings also differ from 2 previous studies that identified a modest association between improved hospital financial performance and decreased adverse patient safety events.[18, 20] However, publicly reported outcomes may not be fully representative of hospital quality and patient safety.[28, 29]

The limited association between hospital financial performance and publicly reported outcomes for AMI and PNA is noteworthy for several reasons. First, publicly reporting outcomes alone without concomitant changes to reimbursement may be inadequate to create strong financial incentives for hospital investment in quality improvement initiatives. Hospitals participating in both public reporting of outcomes and pay‐for‐performance have been shown to achieve greater improvements in outcomes than hospitals engaged only in public reporting.[30] Our time interval for ascertainment of outcomes preceded CMS implementation of the Hospital Readmissions Reduction Program (HRRP) in October 2012, which withholds up to 3% of Medicare hospital reimbursements for higher than expected mortality and readmission rates for AMI, CHF, and PNA. Once outcomes data become available for a 3‐year post‐HRRP implementation period, the impact of this combined approach can be assessed. Second, because adherence to many evidence‐based process measures for these conditions (ie, aspirin use in AMI) is already high, there may be a ceiling effect present that obviates the need for further hospital financial investment to optimize delivery of best practices.[31, 32] Third, hospitals themselves may contribute little to variation in mortality and readmission risk. Of the total variation in mortality and readmission rates among Texas Medicare beneficiaries, only about 1% is attributable to hospitals, whereas 42% to 56% of the variation is explained by differences in patient characteristics.[33, 34] Fourth, there is either low‐quality or insufficient evidence that transitional care interventions specifically targeted to patients with AMI or PNA result in better outcomes.[35] Thus, greater financial investment in hospital‐initiated and postdischarge transitional care interventions for these specific conditions may result in less than the desired effect. Lastly, many hospitalizations for these conditions are emergency hospitalizations that occur after patients present to the emergency department with unexpected and potentially life‐threatening symptoms. Thus, patients may not be able to incorporate the reputation or performance metrics of a hospital in their decisions for where they are hospitalized for AMI, CHF, or PNA despite the public reporting of outcomes.

Given the strong evidence that transitional care interventions reduce readmissions and mortality among patients hospitalized with CHF, we were surprised to find that improved hospital financial performance was associated with an increased risk‐adjusted CHF mortality rate.[36] This association held true for all 3 different measures of hospital financial performance, suggesting that this unexpected finding is unlikely to be the result of statistical chance, though potential reasons for this association remain unclear. One possibility is that the CMS model for CHF mortality may not adequately risk adjust for severity of illness.[37, 38] Thus, robust financial performance may be a marker for hospitals with more advanced heart failure services that care for more patients with severe illness.

Our findings should be interpreted in the context of certain limitations. Our study only included an analysis of outcomes for AMI, CHF, and PNA among older fee‐for‐service Medicare beneficiaries aggregated at the hospital level in California between 2008 and 2012, so generalizability to other populations, conditions, states, and time periods is uncertain. The observational design precludes a robust causal inference between financial performance and outcomes. For readmissions, rates were publicly reported for only the last 2 years of the 3‐year reporting period; thus, our findings may underestimate the association between hospital financial performance and publicly reported readmission rates.

CONCLUSION

There is no consistent relationship between hospital financial performance and subsequent publicly reported outcomes for AMI and PNA. However, for unclear reasons, hospitals with better financial performance had modestly higher CHF mortality rates. Given this limited association, public reporting of outcomes may have had less than the intended impact in motivating hospitals to invest in quality improvement. Additional financial incentives in addition to public reporting, such as readmissions penalties, may help motivate hospitals with robust financial performance to further improve outcomes. This would be a key area for future investigation once outcomes data are available for the 3‐year period following CMS implementation of readmissions penalties in 2012. Reassuringly, there was no association between low 30‐day mortality and readmissions rates and subsequent poor financial performance, suggesting that improved outcomes do not necessarily lead to loss of revenue.

Disclosures

Drs. Nguyen, Halm, and Makam were supported in part by the Agency for Healthcare Research and Quality University of Texas Southwestern Center for Patient‐Centered Outcomes Research (1R24HS022418‐01). Drs. Nguyen and Makam received funding from the University of Texas Southwestern KL2 Scholars Program (NIH/NCATS KL2 TR001103). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no conflicts of interest to disclose.

References
  1. Centers for Medicare and Medicaid Services. Office of the Actuary. National Health Statistics Group. National Healthcare Expenditures Data. Baltimore, MD; 2013. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NationalHealthAccountsHistorical.html. Accessed February 16, 2016.
  2. Brill S. Bitter pill: why medical bills are killing us. Time Magazine. February 20, 2013:1655.
  3. Ginsburg PB. Fee‐for‐service will remain a feature of major payment reforms, requiring more changes in Medicare physician payment. Health Aff (Millwood). 2012;31(9):19771983.
  4. Leatherman S, Berwick D, Iles D, et al. The business case for quality: case studies and an analysis. Health Aff (Millwood). 2003;22(2):1730.
  5. Marshall MN, Shekelle PG, Davies HT, Smith PC. Public reporting on quality in the United States and the United Kingdom. Health Aff (Millwood). 2003;22(3):134148.
  6. Swensen SJ, Dilling JA, Mc Carty PM, Bolton JW, Harper CM. The business case for health‐care quality improvement. J Patient Saf. 2013;9(1):4452.
  7. Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood). 2005;24(4):11501160.
  8. Rauh SS, Wadsworth EB, Weeks WB, Weinstein JN. The savings illusion—why clinical quality improvement fails to deliver bottom‐line results. N Engl J Med. 2011;365(26):e48.
  9. Meyer JA, Silow‐Carroll S, Kutyla T, Stepnick LS, Rybowski LS. Hospital Quality: Ingredients for Success—Overview and Lessons Learned. New York, NY: Commonwealth Fund; 2004.
  10. Silow‐Carroll S, Alteras T, Meyer JA. Hospital Quality Improvement: Strategies and Lessons from U.S. Hospitals. New York, NY: Commonwealth Fund; 2007.
  11. Bazzoli GJ, Clement JP, Lindrooth RC, et al. Hospital financial condition and operational decisions related to the quality of hospital care. Med Care Res Rev. 2007;64(2):148168.
  12. Casalino LP, Elster A, Eisenberg A, Lewis E, Montgomery J, Ramos D. Will pay‐for‐performance and quality reporting affect health care disparities? Health Aff (Millwood). 2007;26(3):w405w414.
  13. Werner RM, Goldman LE, Dudley RA. Comparison of change in quality of care between safety‐net and non‐safety‐net hospitals. JAMA. 2008;299(18):21802187.
  14. Bhalla R, Kalkut G. Could Medicare readmission policy exacerbate health care system inequity? Ann Intern Med. 2010;152(2):114117.
  15. Hernandez AF, Curtis LH. Minding the gap between efforts to reduce readmissions and disparities. JAMA. 2011;305(7):715716.
  16. Terry DF, Moisuk S. Medicare Health Support Pilot Program. N Engl J Med. 2012;366(7):666; author reply 667–668.
  17. Fontanarosa PB, McNutt RA. Revisiting hospital readmissions. JAMA. 2013;309(4):398400.
  18. Encinosa WE, Bernard DM. Hospital finances and patient safety outcomes. Inquiry. 2005;42(1):6072.
  19. Ly DP, Jha AK, Epstein AM. The association between hospital margins, quality of care, and closure or other change in operating status. J Gen Intern Med. 2011;26(11):12911296.
  20. Bazzoli GJ, Chen HF, Zhao M, Lindrooth RC. Hospital financial condition and the quality of patient care. Health Econ. 2008;17(8):977995.
  21. State of California Office of Statewide Health Planning and Development. Healthcare Information Division. Annual financial data. Available at: http://www.oshpd.ca.gov/HID/Products/Hospitals/AnnFinanData/PivotProfles/default.asp. Accessed June 23, 2015.
  22. Centers for Medicare 4(1):1113.
  23. Ross JS, Cha SS, Epstein AJ, et al. Quality of care for acute myocardial infarction at urban safety‐net hospitals. Health Aff (Millwood). 2007;26(1):238248.
  24. Silber JH, Rosenbaum PR, Brachet TJ, et al. The Hospital Compare mortality model and the volume‐outcome relationship. Health Serv Res. 2010;45(5 Pt 1):11481167.
  25. Tukey J. Exploratory Data Analysis. Boston, MA: Addison‐Wesley; 1977.
  26. Press MJ, Scanlon DP, Ryan AM, et al. Limits of readmission rates in measuring hospital quality suggest the need for added metrics. Health Aff (Millwood). 2013;32(6):10831091.
  27. Stefan MS, Pekow PS, Nsa W, et al. Hospital performance measures and 30‐day readmission rates. J Gen Intern Med. 2013;28(3):377385.
  28. Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007;356(5):486496.
  29. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862869.
  30. Werner RM, Bradlow ET. Public reporting on hospital process improvements is linked to better patient outcomes. Health Aff (Millwood). 2010;29(7):13191324.
  31. Goodwin JS, Lin YL, Singh S, Kuo YF. Variation in length of stay and outcomes among hospitalized patients attributable to hospitals and hospitalists. J Gen Intern Med. 2013;28(3):370376.
  32. Singh S, Lin YL, Kuo YF, Nattinger AB, Goodwin JS. Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics. J Gen Intern Med. 2014;29(4):572578.
  33. Prvu Bettger J, Alexander KP, Dolor RJ, et al. Transitional care after hospitalization for acute stroke or myocardial infarction: a systematic review. Ann Intern Med. 2012;157(6):407416.
  34. Jha AK, Orav EJ, Li Z, Epstein AM. The inverse relationship between mortality rates and performance in the Hospital Quality Alliance measures. Health Aff (Millwood). 2007;26(4):11041110.
  35. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  36. Fuller RL, Atkinson G, Hughes JS. Indications of biased risk adjustment in the hospital readmission reduction program. J Ambul Care Manage. 2015;38(1):3947.
References
  1. Centers for Medicare and Medicaid Services. Office of the Actuary. National Health Statistics Group. National Healthcare Expenditures Data. Baltimore, MD; 2013. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NationalHealthAccountsHistorical.html. Accessed February 16, 2016.
  2. Brill S. Bitter pill: why medical bills are killing us. Time Magazine. February 20, 2013:1655.
  3. Ginsburg PB. Fee‐for‐service will remain a feature of major payment reforms, requiring more changes in Medicare physician payment. Health Aff (Millwood). 2012;31(9):19771983.
  4. Leatherman S, Berwick D, Iles D, et al. The business case for quality: case studies and an analysis. Health Aff (Millwood). 2003;22(2):1730.
  5. Marshall MN, Shekelle PG, Davies HT, Smith PC. Public reporting on quality in the United States and the United Kingdom. Health Aff (Millwood). 2003;22(3):134148.
  6. Swensen SJ, Dilling JA, Mc Carty PM, Bolton JW, Harper CM. The business case for health‐care quality improvement. J Patient Saf. 2013;9(1):4452.
  7. Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood). 2005;24(4):11501160.
  8. Rauh SS, Wadsworth EB, Weeks WB, Weinstein JN. The savings illusion—why clinical quality improvement fails to deliver bottom‐line results. N Engl J Med. 2011;365(26):e48.
  9. Meyer JA, Silow‐Carroll S, Kutyla T, Stepnick LS, Rybowski LS. Hospital Quality: Ingredients for Success—Overview and Lessons Learned. New York, NY: Commonwealth Fund; 2004.
  10. Silow‐Carroll S, Alteras T, Meyer JA. Hospital Quality Improvement: Strategies and Lessons from U.S. Hospitals. New York, NY: Commonwealth Fund; 2007.
  11. Bazzoli GJ, Clement JP, Lindrooth RC, et al. Hospital financial condition and operational decisions related to the quality of hospital care. Med Care Res Rev. 2007;64(2):148168.
  12. Casalino LP, Elster A, Eisenberg A, Lewis E, Montgomery J, Ramos D. Will pay‐for‐performance and quality reporting affect health care disparities? Health Aff (Millwood). 2007;26(3):w405w414.
  13. Werner RM, Goldman LE, Dudley RA. Comparison of change in quality of care between safety‐net and non‐safety‐net hospitals. JAMA. 2008;299(18):21802187.
  14. Bhalla R, Kalkut G. Could Medicare readmission policy exacerbate health care system inequity? Ann Intern Med. 2010;152(2):114117.
  15. Hernandez AF, Curtis LH. Minding the gap between efforts to reduce readmissions and disparities. JAMA. 2011;305(7):715716.
  16. Terry DF, Moisuk S. Medicare Health Support Pilot Program. N Engl J Med. 2012;366(7):666; author reply 667–668.
  17. Fontanarosa PB, McNutt RA. Revisiting hospital readmissions. JAMA. 2013;309(4):398400.
  18. Encinosa WE, Bernard DM. Hospital finances and patient safety outcomes. Inquiry. 2005;42(1):6072.
  19. Ly DP, Jha AK, Epstein AM. The association between hospital margins, quality of care, and closure or other change in operating status. J Gen Intern Med. 2011;26(11):12911296.
  20. Bazzoli GJ, Chen HF, Zhao M, Lindrooth RC. Hospital financial condition and the quality of patient care. Health Econ. 2008;17(8):977995.
  21. State of California Office of Statewide Health Planning and Development. Healthcare Information Division. Annual financial data. Available at: http://www.oshpd.ca.gov/HID/Products/Hospitals/AnnFinanData/PivotProfles/default.asp. Accessed June 23, 2015.
  22. Centers for Medicare 4(1):1113.
  23. Ross JS, Cha SS, Epstein AJ, et al. Quality of care for acute myocardial infarction at urban safety‐net hospitals. Health Aff (Millwood). 2007;26(1):238248.
  24. Silber JH, Rosenbaum PR, Brachet TJ, et al. The Hospital Compare mortality model and the volume‐outcome relationship. Health Serv Res. 2010;45(5 Pt 1):11481167.
  25. Tukey J. Exploratory Data Analysis. Boston, MA: Addison‐Wesley; 1977.
  26. Press MJ, Scanlon DP, Ryan AM, et al. Limits of readmission rates in measuring hospital quality suggest the need for added metrics. Health Aff (Millwood). 2013;32(6):10831091.
  27. Stefan MS, Pekow PS, Nsa W, et al. Hospital performance measures and 30‐day readmission rates. J Gen Intern Med. 2013;28(3):377385.
  28. Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007;356(5):486496.
  29. Spatz ES, Sheth SD, Gosch KL, et al. Usual source of care and outcomes following acute myocardial infarction. J Gen Intern Med. 2014;29(6):862869.
  30. Werner RM, Bradlow ET. Public reporting on hospital process improvements is linked to better patient outcomes. Health Aff (Millwood). 2010;29(7):13191324.
  31. Goodwin JS, Lin YL, Singh S, Kuo YF. Variation in length of stay and outcomes among hospitalized patients attributable to hospitals and hospitalists. J Gen Intern Med. 2013;28(3):370376.
  32. Singh S, Lin YL, Kuo YF, Nattinger AB, Goodwin JS. Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics. J Gen Intern Med. 2014;29(4):572578.
  33. Prvu Bettger J, Alexander KP, Dolor RJ, et al. Transitional care after hospitalization for acute stroke or myocardial infarction: a systematic review. Ann Intern Med. 2012;157(6):407416.
  34. Jha AK, Orav EJ, Li Z, Epstein AM. The inverse relationship between mortality rates and performance in the Hospital Quality Alliance measures. Health Aff (Millwood). 2007;26(4):11041110.
  35. Amarasingham R, Moore BJ, Tabak YP, et al. An automated model to identify heart failure patients at risk for 30‐day readmission or death using electronic medical record data. Med Care. 2010;48(11):981988.
  36. Fuller RL, Atkinson G, Hughes JS. Indications of biased risk adjustment in the hospital readmission reduction program. J Ambul Care Manage. 2015;38(1):3947.
Issue
Journal of Hospital Medicine - 11(7)
Issue
Journal of Hospital Medicine - 11(7)
Page Number
481-488
Page Number
481-488
Publications
Publications
Article Type
Display Headline
Relationship between hospital financial performance and publicly reported outcomes
Display Headline
Relationship between hospital financial performance and publicly reported outcomes
Sections
Article Source
© 2016 Society of Hospital Medicine
Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Oanh Kieu Nguyen, MD, 5323 Harry Hines Blvd., Dallas, Texas 75390‐9169; Telephone: 214‐648‐3135; Fax: 214‐648‐3232; E‐mail: oanhk.nguyen@utsouthwestern.edu
Content Gating
Gated (full article locked unless allowed per User)
Gating Strategy
First Peek Free
Article PDF Media
Media Files

Automated Sepsis Alert Systems

Article Type
Changed
Mon, 01/02/2017 - 19:34
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya
  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb
  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)
  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing
  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality
  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

Files
References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):18.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):15461554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):13681377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):16831693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):1319.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):22942303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):20962101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):4045.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):1925.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):22422243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331345.
Article PDF
Issue
Journal of Hospital Medicine - 10(6)
Publications
Page Number
396-402
Sections
Files
Files
Article PDF
Article PDF

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya
  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb
  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)
  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing
  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality
  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

MATERIALS AND METHODS

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).

Study Selection

Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).

RESULTS

Description of Studies

Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Site No./Type Setting Alert Threshold Gold Standard Definition Gold Standard Measurement No. Study Qualitya
  • NOTE: Abbreviations: ED, emergency department; ICD‐9, International Classification of Diseases, Ninth Revision; ICU, intensive care unit; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[10]

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 1/academic MICU 2 SIRS criteriab Reviewer judgment, not otherwise specified Chart review 560 High
Meurer et al., 200913 1/academic ED 2 SIRS criteria Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria Chart review 248 Low
Nelson J. et al., 201114 1/academic ED 2 SIRS criteria and 2 SBP measurements <90 mm Hg Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved Chart review 1,386 High
Nguyen et al., 201415 1/academic ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Reviewer judgment to confirm SIRS, shock, and presence of a serious infection Chart review 1,095 Low
Thiel et al., 201016 1/academic Wards Recursive partitioning tree analysis including vitals and laboratory resultsc Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer 27,674 Low
Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts
Source Design Site No./ Type Setting No. Alert System Type Alert Threshold Alert Notificationa Treatment Recommendation Study Qualityb
  • NOTE: Abbreviations: CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICD‐9, International Classification of Diseases, Ninth Revision; MICU, medical intensive care unit; RCT, randomized control trial; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

  • Assessed by prespecified criteria of study design and intention‐to‐treat protocol.

  • Not an intention‐to‐treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • Not an intention‐to‐treat analysis. Only medical records reviewed of individuals with an ICD‐9 discharge code of sepsis.

  • Not an intention‐to‐treat analysis. Three hundred ninety‐eight patients triggered the alert, but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

  • Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Berger et al., 201017 Before‐after (6 months pre and 6 months post) 1/academic ED 5796c CPOE system 2 SIRS criteria CPOE passive alert Yes: lactate collection Low
Hooper et al., 201210 RCT 1/academic MICU 443 EHR 2 SIRS criteriad Text page and EHR passive alert No High
McRee et al., 201418 Before‐after (6 months pre and 6 months post) 1/academic Wards 171e EHR 2 SIRS criteria Notified nurse, specifics unclear No, but the nurse completed a sepsis risk evaluation flow sheet Low
Nelson et al., 201114 Before‐after (3 months pre and 3 months post) 1/academic ED 184f EHR 2 SIRS criteria and 2 or more SBP readings <90 mm Hg Text page and EHR passive alert Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others Low
Sawyer et al., 201119 Prospective, nonrandomized (2 intervention and 4 control wards) 1/academic Wards 300 EHR Recursive partitioning regression tree algorithm including vitals and lab valuesg Text page to charge nurse who then assessed patient and informed treating physicianh No High

Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).

Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Source Setting Alert Threshold Prevalence, % Sensitivity, % (95% CI) Specificity, % (95% CI) PPV, % (95% CI) NPV, % (95% CI) LR+, (95% CI) LR, (95% CI)
  • NOTE: Abbreviations: ED, emergency department; MICU, medical intensive care unit; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome.

  • Recorded within a 24‐hour period, mandating either abnormal temperature or white blood cell count.

  • False negative and true negatives unknown due to random sampling of nonalert cases.

  • Estimated value based on random sample of 300 non‐alert cases.

  • Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Hooper et al., 201210 MICU 2 SIRS criteriaa 36.3 98.9 (95.799.8) 18.1 (14.222.9) 40.7 (36.145.5) 96.7 (87.599.4) 1.21 (1.14‐1.27) 0.06 (0.01‐0.25)
Meurer et al., 200913 ED 2 SIRS criteria 27.8 36.2 (25.348.8) 79.9 (73.185.3) 41.0 (28.854.3) 76.5 (69.682.2) 1.80 (1.17‐2.76) 0.80 (0.67‐0.96)
Nelson et al., 201114 ED 2 SIRS criteria and 2 SBP measurements<90 mm Hg 0.8 63.6 (31.687.8) 99.6 (99.099.8) 53.8 (26.179.6) 99.7 (99.299.9) 145.8 (58.4364.1) 0.37 (0.17‐0.80)
Nguyen et al., 201415 ED 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) Unable to estimateb Unable to estimateb Unable to estimateb 44.7 (41.248.2) 100.0c (98.8100.0) Unable to estimateb Unable to estimateb
Thiel et al., 201016 Wards Recursive partitioning tree analysis including vitals and laboratory resultsd 4.7 17.1 (15.119.3) 96.7 (96.596.9) 20.5 (18.223.0) 95.9 (95.796.2) 5.22 (4.56‐5.98) 0.86 (0.84‐0.88)

All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]

Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts
Study Patient Selection Index Test Reference Standard Flow and Timing
  • Determined by 2 independent abstractors using the revised Quality Assessment of Diagnostic Accuracy Studies revised tool.11 Three plus signs indicate the lowest risk for bias and 1 plus sign indicates highest risk for bias.

  • Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert.

  • Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

  • Consecutive selection for cases, but random selection of noncases greatly limited evaluation of diagnostic accuracy.

  • Gold standard was interpreted with knowledge of the results of the sepsis alert.

  • Discharge International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

  • There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD‐9 diagnoses), which may result in misclassification.

Hooper et al., 201210 +++ +++ ++b +++
Meurer et al., 200913 +++ +++ ++b +c
Nelson et al., 201114 +++ +++ ++b +++
Nguyen et al., 201415 +d +++ +e +++
Thiel et al., 201016 +++ +++ +f +g

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]

Effectiveness of Automated Electronic Sepsis Alerts
Source Outcomes Evaluated Key Findings Quality
  • NOTE: Abbreviations: CI, confidence interval; CXR, chest radiograph; ED, emergency department; HR, hazard ratio; ICU, intensive care unit; IV, intravenous; OR, odds ratio.

Hooper et al., 201210 Primary: time to receipt of antibiotic (new or changed) No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) High
Secondary: sepsis‐related process measures and outcomes No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3)
Sawyer et al., 201119 Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) High
Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7)
Berger et al., 201017 Primary: lactate collection in ED Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) Low
Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6)
McRee et al., 201418 Stage of sepsis, length of stay, mortality, discharge location Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) Low
Nelson et al., 201114 Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). Low

Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]

DISCUSSION

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.

CONCLUSION

Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.

Acknowledgements

The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.

Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.

References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):18.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):15461554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):13681377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):16831693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):1319.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):22942303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):20962101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):4045.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):1925.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):22422243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331345.
References
  1. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
  2. Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):18.
  3. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):15461554.
  4. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580637.
  5. Rivers E, Nguyen B, Havstad S, et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):13681377.
  6. Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):16831693.
  7. Turi SK, Ah D. Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):1319.
  8. Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929936.
  9. Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):22942303.
  10. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):20962101.
  11. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529536.
  12. Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513523.
  13. Meurer WJ, Smith BL, Losman ED, et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):4045.
  14. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500504.
  15. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
  16. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):1925.
  17. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394407.
  18. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546549.
  19. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469473.
  20. Brun‐Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64S74.
  21. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487491.
  22. Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451456.
  23. Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):22422243.
  24. Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806810.
  25. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331345.
Issue
Journal of Hospital Medicine - 10(6)
Issue
Journal of Hospital Medicine - 10(6)
Page Number
396-402
Page Number
396-402
Publications
Publications
Article Type
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review
Display Headline
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: A systematic review
Sections
Article Source
© 2015 Society of Hospital Medicine
Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Anil N. Makam, MD, 5323 Harry Hines Blvd., Dallas, TX 75390‐9169; Telephone: 214‐648‐3272; Fax: 314‐648‐3232; E‐mail: anil.makam@utsouthwestern.edu
Content Gating
Gated (full article locked unless allowed per User)
Gating Strategy
First Peek Free
Article PDF Media
Media Files

Blood Cultures in Nonpneumonia Illness

Article Type
Changed
Mon, 01/02/2017 - 19:34
Display Headline
Blood culture use in the emergency department in patients hospitalized with respiratory symptoms due to a nonpneumonia illness

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

Study Design, Setting, and Participants

We performed a cross‐sectional analysis using data from the 2002 to 2010 National Hospital Ambulatory Medical Care Surveys (NHAMCS), a probability sample of visits to EDs of noninstitutional general and short‐stay hospitals in the United States, excluding federal, military, and Veterans Administration hospitals.[10] The NHAMCS data are derived through multistage sampling and estimation procedures that produce unbiased national estimates.[11] Further details regarding the sampling and estimation procedures can be found on the US Centers for Disease Control and Prevention website.[10, 11] Years 2005 and 2006 are omitted because NHAMCS did not collect blood culture use during this period. We included all visits by patients aged 18 years or older who were subsequently hospitalized.

Measurements

Trained hospital staff collected data with oversight from US Census Bureau field representatives.[12] Blood culture collection during the visit was recorded as a checkbox on the NHAMCS data collection form if at least 1 culture was ordered or collected in the ED. Visits for conditions that may resemble pneumonia were defined as visits with a respiratory symptom listed for at least 1 of the 3 reason for visit fields, excluding those visits admitted with a diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification [ICD‐9‐CM] codes 481.xx‐486.xx). The reason for visit field captures the patient's complaints, symptoms, or other reasons for the visit in the patient's own words. CAP was defined by having 1 of the 3 ED provider's diagnosis fields coded as pneumonia (ICD‐9‐CM 481486), excluding patients with suspected hospital‐acquired pneumonia (nursing home or institutionalized resident, seen in the ED in the past 72 hours, or discharged from any hospital within the past 7 days) or those with a follow‐up visit for the same problem.[8]

Data Analysis

All analyses accounted for the complex survey design, including weights, to reflect national estimates. To examine for potential spillover effects of the blood culture recommendations for CAP on other conditions that may present similarly, we used linear regression to examine the trend in collecting blood cultures in patients admitted to the hospital with respiratory symptoms due to a nonpneumonia illness.

The data were analyzed using Stata statistical software, version 12.0 (StataCorp, College Station, TX). This study was exempt from review by the institutional review board of the University of California, San Francisco and the San Francisco Veterans Affairs Medical Center.

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

Characteristics of Visits to the ED by Patients Hospitalized With Respiratory Symptoms Due to a Nonpneumonia Illness From 2002 to 2010
Years 20022004, Weighted % (Unweighted N=2,175)b Years 20072008, Weighted % (Unweighted N=1,346)b Years 20092010, Weighted % (Unweighted N=1,333)b
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Percentages shown are weighted to reflect complex survey design. All estimates are considered to be reliable (standard errors below the 30% threshold recommended by the National Hospital Ambulatory Medical Care Survey for reporting data and 30 or more unweighted observations per subgroup).

  • Excludes year 2002 due to incomplete ethnicity ascertainment (unweighted number for race/ethnicity ascertainment=1,496).

  • Only for years 2007 to 2010, which included oxygen saturation in the survey.

Blood culture collected 9.8 14.4 19.9
Demographics
Age 65 years 56.9 55.1 50.9
Female 54.0 57.5 51.3
Race/ethnicity
White, non‐Hispanic 71.5c 69.5 67.2
Black, non‐Hispanic 17.1c 20.8 22.2
Other 11.3c 9.7 10.6
Primary payer
Private insurance 23.4 19.1 19.1
Medicare 55.2 58.0 54.2
Medicaid 10.0 10.5 13.8
Other/unknown 11.4 12.4 13.0
Visit characteristics
Disposition status
Non‐ICU 86.8 85.5 83.3
ICU 13.2 14.5 16.7
Fever (38.0C) 6.1 5.3 4.8
Hypoxia (<90%)d 11.5 10.9
Emergent status by triage 46.1 44.5 35.8
Administered antibiotics 19.6 24.6 24.8
Tests/services ordered in ED
05 29.9 29.1 22.3
610 43.5 58.3 56.1
>10 26.6 12.6 21.6
ED characteristics
Region
West 16.6 18.2 15.8
Midwest 27.1 25.2 22.8
South 32.8 36.4 38.6
Northeast 23.5 20.2 22.7
Hospital owner
Nonprofit 80.6 84.6 80.7
Government 12.1 6.4 13.0
Private 7.4 9.0 6.3

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P<0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

Emergency Department Visits With a Blood Culture Collected in Patients Subsequently Hospitalized, Stratified by Select Conditions
National Weighted Estimates (95% CI)
  • NOTE: Abbreviations: CAP, community‐acquired pneumonia; CI, confidence interval; ICD‐9, International Classification of Diseases, 9th Revision.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Linear trend analysis.

  • Respiratory symptoms were defined by the patient's reason for visit. Excludes visits with an emergency department provider's diagnosis of pneumonia (ICD‐9 481486).

Condition 2002 2003 2004 2007 2008 2009 2010 P Valueb
Respiratory symptomc
% 9.9 (7.113.5) 9.2 (6.912.2) 10.6 (7.914.1) 13.5 (10.117.8) 15.2 (12.118.8) 19.4 (15.923.5) 20.4 (16.125.6) <0.001
No., thousands 211 (126296) 229 (140319) 212 (140285) 287 (191382) 418 (288548) 486 (344627) 526 (361692)
CAP
% 29.4 (21.938.3) 34.2 (25.943.6) 38.4 (31.045.4) 45.7 (35.456.4) 44.1 (34.154.6) 46.7 (37.456.1) 51.1 (41.860.3) 0.027
No., thousands 155 (100210) 287 (177397) 276 (192361) 277 (173381) 361 (255467) 350 (237464) 428 (283574)

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

ACKNOWLEDGEMENT

Disclosures: Dr. Makam's work on this project was completed while he was a Primary Care Research Fellow at the University of California San Francisco, funded by an NRSA training grant (T32HP19025‐07‐00). The authors report no conflicts of interest.

Files
References
  1. Bartlett JG, Dowell SF, Mandell LA, File TM, Musher DM, Fine MJ. Practice guidelines for the management of community‐acquired pneumonia in adults. Infectious Diseases Society of America. Clin Infect Dis. 2000;31(2):347382.
  2. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  3. Campbell SG, Marrie TJ, Anstey R, Dickinson G, Ackroyd‐Stolarz S. The contribution of blood cultures to the clinical management of adult patients admitted to the hospital with community‐acquired pneumonia: a prospective observational study. Chest. 2003;123(4):11421150.
  4. Kennedy M, Bates DW, Wright SB, Ruiz R, Wolfe RE, Shapiro NI. Do emergency department blood cultures change practice in patients with pneumonia? Ann Emerg Med. 2005;46(5):393400.
  5. Metersky ML, Ma A, Bratzler DW, Houck PM. Predicting bacteremia in patients with community‐acquired pneumonia. Am J Respir Crit Care Med. 2004;169(3):342347.
  6. Waterer GW, Wunderink RG. The influence of the severity of community‐acquired pneumonia on the usefulness of blood cultures. Respir Med. 2001;95(1):7882.
  7. Walls RM, Resnick J. The Joint Commission on Accreditation of Healthcare Organizations and Center for Medicare and Medicaid Services community‐acquired pneumonia initiative: what went wrong? Ann Emerg Med. 2005;46(5):409411.
  8. Makam AN, Auerbach AD, Steinman MA. Blood culture use in the emergency department in patients hospitalized for community‐acquired pneumonia [published online ahead of print March 10, 2014]. JAMA Intern Med. doi: 10.1001/jamainternmed.2013.13808.
  9. Heckerling PS, Tape TG, Wigton RS, et al. Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664670.
  10. Centers for Disease Control and Prevention. NHAMCS scope and sample design. Available at: http://www.cdc.gov/nchs/ahcd/ahcd_scope.htm#nhamcs_scope. Accessed May 27, 2013.
  11. Centers for Disease Control and Prevention. NHAMCS estimation procedures. http://www.cdc.gov/nchs/ahcd/ahcd_estimation_procedures.htm#nhamcs_procedures. Updated January 15, 2010. Accessed May 27, 2013.
  12. McCaig LF, Burt CW, Schappert SM, et al. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2013;62(5):549551.
  13. Metlay JP, Kapoor WN, Fine MJ. Does this patient have community‐acquired pneumonia? Diagnosing pneumonia by history and physical examination. JAMA. 1997;278(17):14401445.
  14. Kanwar M, Brar N, Khatib R, Fakih MG. Misdiagnosis of community‐acquired pneumonia and inappropriate utilization of antibiotics: side effects of the 4‐h antibiotic administration rule. Chest. 2007;131(6):18651869.
  15. Bates DW, Goldman L, Lee TH. Contaminant blood cultures and resource utilization. The true consequences of false‐positive results. JAMA. 1991;265(3):365369.
  16. Zwang O, Albert RK. Analysis of strategies to improve cost effectiveness of blood cultures. J Hosp Med. 2006;1(5):272276.
  17. Cooper RJ. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2012;60(6):722725.
Article PDF
Issue
Journal of Hospital Medicine - 9(8)
Publications
Page Number
521-524
Sections
Files
Files
Article PDF
Article PDF

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

Study Design, Setting, and Participants

We performed a cross‐sectional analysis using data from the 2002 to 2010 National Hospital Ambulatory Medical Care Surveys (NHAMCS), a probability sample of visits to EDs of noninstitutional general and short‐stay hospitals in the United States, excluding federal, military, and Veterans Administration hospitals.[10] The NHAMCS data are derived through multistage sampling and estimation procedures that produce unbiased national estimates.[11] Further details regarding the sampling and estimation procedures can be found on the US Centers for Disease Control and Prevention website.[10, 11] Years 2005 and 2006 are omitted because NHAMCS did not collect blood culture use during this period. We included all visits by patients aged 18 years or older who were subsequently hospitalized.

Measurements

Trained hospital staff collected data with oversight from US Census Bureau field representatives.[12] Blood culture collection during the visit was recorded as a checkbox on the NHAMCS data collection form if at least 1 culture was ordered or collected in the ED. Visits for conditions that may resemble pneumonia were defined as visits with a respiratory symptom listed for at least 1 of the 3 reason for visit fields, excluding those visits admitted with a diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification [ICD‐9‐CM] codes 481.xx‐486.xx). The reason for visit field captures the patient's complaints, symptoms, or other reasons for the visit in the patient's own words. CAP was defined by having 1 of the 3 ED provider's diagnosis fields coded as pneumonia (ICD‐9‐CM 481486), excluding patients with suspected hospital‐acquired pneumonia (nursing home or institutionalized resident, seen in the ED in the past 72 hours, or discharged from any hospital within the past 7 days) or those with a follow‐up visit for the same problem.[8]

Data Analysis

All analyses accounted for the complex survey design, including weights, to reflect national estimates. To examine for potential spillover effects of the blood culture recommendations for CAP on other conditions that may present similarly, we used linear regression to examine the trend in collecting blood cultures in patients admitted to the hospital with respiratory symptoms due to a nonpneumonia illness.

The data were analyzed using Stata statistical software, version 12.0 (StataCorp, College Station, TX). This study was exempt from review by the institutional review board of the University of California, San Francisco and the San Francisco Veterans Affairs Medical Center.

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

Characteristics of Visits to the ED by Patients Hospitalized With Respiratory Symptoms Due to a Nonpneumonia Illness From 2002 to 2010
Years 20022004, Weighted % (Unweighted N=2,175)b Years 20072008, Weighted % (Unweighted N=1,346)b Years 20092010, Weighted % (Unweighted N=1,333)b
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Percentages shown are weighted to reflect complex survey design. All estimates are considered to be reliable (standard errors below the 30% threshold recommended by the National Hospital Ambulatory Medical Care Survey for reporting data and 30 or more unweighted observations per subgroup).

  • Excludes year 2002 due to incomplete ethnicity ascertainment (unweighted number for race/ethnicity ascertainment=1,496).

  • Only for years 2007 to 2010, which included oxygen saturation in the survey.

Blood culture collected 9.8 14.4 19.9
Demographics
Age 65 years 56.9 55.1 50.9
Female 54.0 57.5 51.3
Race/ethnicity
White, non‐Hispanic 71.5c 69.5 67.2
Black, non‐Hispanic 17.1c 20.8 22.2
Other 11.3c 9.7 10.6
Primary payer
Private insurance 23.4 19.1 19.1
Medicare 55.2 58.0 54.2
Medicaid 10.0 10.5 13.8
Other/unknown 11.4 12.4 13.0
Visit characteristics
Disposition status
Non‐ICU 86.8 85.5 83.3
ICU 13.2 14.5 16.7
Fever (38.0C) 6.1 5.3 4.8
Hypoxia (<90%)d 11.5 10.9
Emergent status by triage 46.1 44.5 35.8
Administered antibiotics 19.6 24.6 24.8
Tests/services ordered in ED
05 29.9 29.1 22.3
610 43.5 58.3 56.1
>10 26.6 12.6 21.6
ED characteristics
Region
West 16.6 18.2 15.8
Midwest 27.1 25.2 22.8
South 32.8 36.4 38.6
Northeast 23.5 20.2 22.7
Hospital owner
Nonprofit 80.6 84.6 80.7
Government 12.1 6.4 13.0
Private 7.4 9.0 6.3

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P<0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

Emergency Department Visits With a Blood Culture Collected in Patients Subsequently Hospitalized, Stratified by Select Conditions
National Weighted Estimates (95% CI)
  • NOTE: Abbreviations: CAP, community‐acquired pneumonia; CI, confidence interval; ICD‐9, International Classification of Diseases, 9th Revision.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Linear trend analysis.

  • Respiratory symptoms were defined by the patient's reason for visit. Excludes visits with an emergency department provider's diagnosis of pneumonia (ICD‐9 481486).

Condition 2002 2003 2004 2007 2008 2009 2010 P Valueb
Respiratory symptomc
% 9.9 (7.113.5) 9.2 (6.912.2) 10.6 (7.914.1) 13.5 (10.117.8) 15.2 (12.118.8) 19.4 (15.923.5) 20.4 (16.125.6) <0.001
No., thousands 211 (126296) 229 (140319) 212 (140285) 287 (191382) 418 (288548) 486 (344627) 526 (361692)
CAP
% 29.4 (21.938.3) 34.2 (25.943.6) 38.4 (31.045.4) 45.7 (35.456.4) 44.1 (34.154.6) 46.7 (37.456.1) 51.1 (41.860.3) 0.027
No., thousands 155 (100210) 287 (177397) 276 (192361) 277 (173381) 361 (255467) 350 (237464) 428 (283574)

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

ACKNOWLEDGEMENT

Disclosures: Dr. Makam's work on this project was completed while he was a Primary Care Research Fellow at the University of California San Francisco, funded by an NRSA training grant (T32HP19025‐07‐00). The authors report no conflicts of interest.

In 2002, based on consensus practice guidelines,[1] the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) announced a core measure mandating the collection of routine blood cultures in the emergency department (ED) for all patients hospitalized with community‐acquired pneumonia (CAP) to benchmark the quality of hospital care. However, due to the limited utility and false‐positive results of routine blood cultures,[2, 3, 4, 5, 6] performance measures and practice guidelines were modified in 2005 and 2007, respectively, to recommend routine collection in only the sickest patients with CAP.[2, 7] Despite recommendations for a more narrow set of indications, the collection of blood cultures in patients hospitalized with CAP continued to increase.[8]

Distinguishing CAP from other respiratory illnesses may be challenging. Among patients presenting to the ED with an acute respiratory illness, only a minority of patients (10%30%) are diagnosed with pneumonia.[9] Therefore, the harms and costs of inappropriate diagnostic tests for CAP may be further magnified if applied to a larger population of patients who present to the ED with similar clinical signs and symptoms as pneumonia. Using a national sample of ED visits, we examined whether there was a similar increase in the frequency of blood culture collection among patients who were hospitalized with respiratory symptoms due to an illness other than pneumonia.

METHOD

Study Design, Setting, and Participants

We performed a cross‐sectional analysis using data from the 2002 to 2010 National Hospital Ambulatory Medical Care Surveys (NHAMCS), a probability sample of visits to EDs of noninstitutional general and short‐stay hospitals in the United States, excluding federal, military, and Veterans Administration hospitals.[10] The NHAMCS data are derived through multistage sampling and estimation procedures that produce unbiased national estimates.[11] Further details regarding the sampling and estimation procedures can be found on the US Centers for Disease Control and Prevention website.[10, 11] Years 2005 and 2006 are omitted because NHAMCS did not collect blood culture use during this period. We included all visits by patients aged 18 years or older who were subsequently hospitalized.

Measurements

Trained hospital staff collected data with oversight from US Census Bureau field representatives.[12] Blood culture collection during the visit was recorded as a checkbox on the NHAMCS data collection form if at least 1 culture was ordered or collected in the ED. Visits for conditions that may resemble pneumonia were defined as visits with a respiratory symptom listed for at least 1 of the 3 reason for visit fields, excluding those visits admitted with a diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification [ICD‐9‐CM] codes 481.xx‐486.xx). The reason for visit field captures the patient's complaints, symptoms, or other reasons for the visit in the patient's own words. CAP was defined by having 1 of the 3 ED provider's diagnosis fields coded as pneumonia (ICD‐9‐CM 481486), excluding patients with suspected hospital‐acquired pneumonia (nursing home or institutionalized resident, seen in the ED in the past 72 hours, or discharged from any hospital within the past 7 days) or those with a follow‐up visit for the same problem.[8]

Data Analysis

All analyses accounted for the complex survey design, including weights, to reflect national estimates. To examine for potential spillover effects of the blood culture recommendations for CAP on other conditions that may present similarly, we used linear regression to examine the trend in collecting blood cultures in patients admitted to the hospital with respiratory symptoms due to a nonpneumonia illness.

The data were analyzed using Stata statistical software, version 12.0 (StataCorp, College Station, TX). This study was exempt from review by the institutional review board of the University of California, San Francisco and the San Francisco Veterans Affairs Medical Center.

RESULTS

This study included 4854 ED visits, representing approximately 17 million visits by adult patients hospitalized with respiratory symptoms due to a nonpneumonia illness. The most common primary ED provider's diagnoses for these visits included heart failure (15.9%), chronic obstructive pulmonary disease (12.6%), chest pain (11.9%), respiratory insufficiency or failure (8.8%), and asthma (5.5%). The characteristics of these visits are shown in Table 1.

Characteristics of Visits to the ED by Patients Hospitalized With Respiratory Symptoms Due to a Nonpneumonia Illness From 2002 to 2010
Years 20022004, Weighted % (Unweighted N=2,175)b Years 20072008, Weighted % (Unweighted N=1,346)b Years 20092010, Weighted % (Unweighted N=1,333)b
  • NOTE: Abbreviations: ED, emergency department; ICU, intensive care unit.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Percentages shown are weighted to reflect complex survey design. All estimates are considered to be reliable (standard errors below the 30% threshold recommended by the National Hospital Ambulatory Medical Care Survey for reporting data and 30 or more unweighted observations per subgroup).

  • Excludes year 2002 due to incomplete ethnicity ascertainment (unweighted number for race/ethnicity ascertainment=1,496).

  • Only for years 2007 to 2010, which included oxygen saturation in the survey.

Blood culture collected 9.8 14.4 19.9
Demographics
Age 65 years 56.9 55.1 50.9
Female 54.0 57.5 51.3
Race/ethnicity
White, non‐Hispanic 71.5c 69.5 67.2
Black, non‐Hispanic 17.1c 20.8 22.2
Other 11.3c 9.7 10.6
Primary payer
Private insurance 23.4 19.1 19.1
Medicare 55.2 58.0 54.2
Medicaid 10.0 10.5 13.8
Other/unknown 11.4 12.4 13.0
Visit characteristics
Disposition status
Non‐ICU 86.8 85.5 83.3
ICU 13.2 14.5 16.7
Fever (38.0C) 6.1 5.3 4.8
Hypoxia (<90%)d 11.5 10.9
Emergent status by triage 46.1 44.5 35.8
Administered antibiotics 19.6 24.6 24.8
Tests/services ordered in ED
05 29.9 29.1 22.3
610 43.5 58.3 56.1
>10 26.6 12.6 21.6
ED characteristics
Region
West 16.6 18.2 15.8
Midwest 27.1 25.2 22.8
South 32.8 36.4 38.6
Northeast 23.5 20.2 22.7
Hospital owner
Nonprofit 80.6 84.6 80.7
Government 12.1 6.4 13.0
Private 7.4 9.0 6.3

The proportion of blood cultures collected in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness increased from 9.9% (95% confidence interval [CI]: 7.1%‐13.5%) in 2002 to 20.4% (95% CI: 16.1%‐25.6%) in 2010 (P<0.001 for the trend). This observed increase paralleled the increase in the frequency of culture collection in patients hospitalized with CAP (P=0.12 for the difference in temporal trends). The estimated absolute number of visits for respiratory symptoms due a nonpneumonia illness with a blood culture collected increased from 211,000 (95% CI: 126,000296,000) in 2002 to 526,000 (95% CI: 361,000692,000) in 2010, which was similar in magnitude to the estimated number of visits for CAP with a culture collected (Table 2).

Emergency Department Visits With a Blood Culture Collected in Patients Subsequently Hospitalized, Stratified by Select Conditions
National Weighted Estimates (95% CI)
  • NOTE: Abbreviations: CAP, community‐acquired pneumonia; CI, confidence interval; ICD‐9, International Classification of Diseases, 9th Revision.

  • Years 2005 and 2006 are omitted for missing the blood culture field in the survey.

  • Linear trend analysis.

  • Respiratory symptoms were defined by the patient's reason for visit. Excludes visits with an emergency department provider's diagnosis of pneumonia (ICD‐9 481486).

Condition 2002 2003 2004 2007 2008 2009 2010 P Valueb
Respiratory symptomc
% 9.9 (7.113.5) 9.2 (6.912.2) 10.6 (7.914.1) 13.5 (10.117.8) 15.2 (12.118.8) 19.4 (15.923.5) 20.4 (16.125.6) <0.001
No., thousands 211 (126296) 229 (140319) 212 (140285) 287 (191382) 418 (288548) 486 (344627) 526 (361692)
CAP
% 29.4 (21.938.3) 34.2 (25.943.6) 38.4 (31.045.4) 45.7 (35.456.4) 44.1 (34.154.6) 46.7 (37.456.1) 51.1 (41.860.3) 0.027
No., thousands 155 (100210) 287 (177397) 276 (192361) 277 (173381) 361 (255467) 350 (237464) 428 (283574)

DISCUSSION

In this national study of ED visits, we found that the collection of blood cultures in patients hospitalized with respiratory symptoms due to an illness other than pneumonia continued to increase from 2002 to 2010 in a parallel fashion to the trend observed for patients hospitalized with CAP. Our findings suggest that the heightened attention of collecting blood cultures for suspected pneumonia had unintended consequences, which led to an increase in the collection of blood cultures in patients hospitalized with conditions that mimic pneumonia in the ED.

There can be a great deal of diagnostic uncertainty when treating patients in the ED who present with acute respiratory symptoms. Unfortunately, the initial history and physical exam are often insufficient to effectively rule in CAP.[13] Furthermore, the challenge of diagnosing pneumonia is amplified in the subset of patients who present with evolving, atypical, or occult disease. Faced with this diagnostic uncertainty, ED providers may feel pressured to comply with performance measures for CAP, promoting the overuse of inappropriate diagnostic tests and treatments. For instance, efforts to comply with early antibiotic administration in patients with CAP have led to an increase in unnecessary antibiotic use among patients with a diagnosis other than CAP.[14] Due to concerns for these unintended consequences, the core measure for early antibiotic administration was effectively retired in 2012.

Although a smaller percentage of ED visits for respiratory symptoms had a blood culture collected compared to CAP visits, there was a similar absolute number of visits with a blood culture collected during the study period. While a fraction of these patients may present with an infectious etiology aside from pneumonia, the majority of these cases likely represent situations where blood cultures add little diagnostic value at the expense of potentially longer hospital stays and broad spectrum antimicrobial use due to false‐positive results,[5, 15] as well as higher costs incurred by the test itself.[15, 16]

Although recommendations for routine culture collection for all patients hospitalized with CAP have been revised, the JCAHO/CMS core measure (PN‐3b) announced in 2002 mandates that if a culture is collected in the ED, it should be collected prior to antibiotic administration. Due to inherent uncertainty and challenges in making a timely diagnosis of pneumonia, this measure may encourage providers to reflexively order cultures in all patients presenting with respiratory symptoms in whom antibiotic administration is anticipated. The observed increasing trend in culture collection in patients hospitalized with respiratory symptoms due to a nonpneumonia illness should prompt JCAHO and CMS to reevaluate the risks and benefits of this core measure, with consideration of eliminating it altogether to discourage overuse in this population.

Our study had certain limitations. First, the omission of 2005 and 2006 data prohibited an evaluation of whether culture rates slowed down among patients hospitalized with respiratory symptoms due to a nonpneumonia illness after revisions in recommendations for obtaining cultures in patients with CAP. Second, there may have been misclassification of culture collection due to errors in chart abstraction. However, abstraction errors in the NHAMCS typically result in undercoding.[17] Therefore, our findings likely underestimate the magnitude and frequency of culture collection in this population.

In conclusion, collecting blood cultures in the ED for patients hospitalized with respiratory symptoms due to a nonpneumonia illness has increased in a parallel fashion compared to the trend in culture collection in patients hospitalized with CAP from 2002 to 2010. This suggests an important potential unintended consequence of blood culture recommendations for CAP on patients who present with conditions that resemble pneumonia. More attention to the judicious use of blood cultures in these patients to reduce harm and costs is needed.

ACKNOWLEDGEMENT

Disclosures: Dr. Makam's work on this project was completed while he was a Primary Care Research Fellow at the University of California San Francisco, funded by an NRSA training grant (T32HP19025‐07‐00). The authors report no conflicts of interest.

References
  1. Bartlett JG, Dowell SF, Mandell LA, File TM, Musher DM, Fine MJ. Practice guidelines for the management of community‐acquired pneumonia in adults. Infectious Diseases Society of America. Clin Infect Dis. 2000;31(2):347382.
  2. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  3. Campbell SG, Marrie TJ, Anstey R, Dickinson G, Ackroyd‐Stolarz S. The contribution of blood cultures to the clinical management of adult patients admitted to the hospital with community‐acquired pneumonia: a prospective observational study. Chest. 2003;123(4):11421150.
  4. Kennedy M, Bates DW, Wright SB, Ruiz R, Wolfe RE, Shapiro NI. Do emergency department blood cultures change practice in patients with pneumonia? Ann Emerg Med. 2005;46(5):393400.
  5. Metersky ML, Ma A, Bratzler DW, Houck PM. Predicting bacteremia in patients with community‐acquired pneumonia. Am J Respir Crit Care Med. 2004;169(3):342347.
  6. Waterer GW, Wunderink RG. The influence of the severity of community‐acquired pneumonia on the usefulness of blood cultures. Respir Med. 2001;95(1):7882.
  7. Walls RM, Resnick J. The Joint Commission on Accreditation of Healthcare Organizations and Center for Medicare and Medicaid Services community‐acquired pneumonia initiative: what went wrong? Ann Emerg Med. 2005;46(5):409411.
  8. Makam AN, Auerbach AD, Steinman MA. Blood culture use in the emergency department in patients hospitalized for community‐acquired pneumonia [published online ahead of print March 10, 2014]. JAMA Intern Med. doi: 10.1001/jamainternmed.2013.13808.
  9. Heckerling PS, Tape TG, Wigton RS, et al. Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664670.
  10. Centers for Disease Control and Prevention. NHAMCS scope and sample design. Available at: http://www.cdc.gov/nchs/ahcd/ahcd_scope.htm#nhamcs_scope. Accessed May 27, 2013.
  11. Centers for Disease Control and Prevention. NHAMCS estimation procedures. http://www.cdc.gov/nchs/ahcd/ahcd_estimation_procedures.htm#nhamcs_procedures. Updated January 15, 2010. Accessed May 27, 2013.
  12. McCaig LF, Burt CW, Schappert SM, et al. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2013;62(5):549551.
  13. Metlay JP, Kapoor WN, Fine MJ. Does this patient have community‐acquired pneumonia? Diagnosing pneumonia by history and physical examination. JAMA. 1997;278(17):14401445.
  14. Kanwar M, Brar N, Khatib R, Fakih MG. Misdiagnosis of community‐acquired pneumonia and inappropriate utilization of antibiotics: side effects of the 4‐h antibiotic administration rule. Chest. 2007;131(6):18651869.
  15. Bates DW, Goldman L, Lee TH. Contaminant blood cultures and resource utilization. The true consequences of false‐positive results. JAMA. 1991;265(3):365369.
  16. Zwang O, Albert RK. Analysis of strategies to improve cost effectiveness of blood cultures. J Hosp Med. 2006;1(5):272276.
  17. Cooper RJ. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2012;60(6):722725.
References
  1. Bartlett JG, Dowell SF, Mandell LA, File TM, Musher DM, Fine MJ. Practice guidelines for the management of community‐acquired pneumonia in adults. Infectious Diseases Society of America. Clin Infect Dis. 2000;31(2):347382.
  2. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  3. Campbell SG, Marrie TJ, Anstey R, Dickinson G, Ackroyd‐Stolarz S. The contribution of blood cultures to the clinical management of adult patients admitted to the hospital with community‐acquired pneumonia: a prospective observational study. Chest. 2003;123(4):11421150.
  4. Kennedy M, Bates DW, Wright SB, Ruiz R, Wolfe RE, Shapiro NI. Do emergency department blood cultures change practice in patients with pneumonia? Ann Emerg Med. 2005;46(5):393400.
  5. Metersky ML, Ma A, Bratzler DW, Houck PM. Predicting bacteremia in patients with community‐acquired pneumonia. Am J Respir Crit Care Med. 2004;169(3):342347.
  6. Waterer GW, Wunderink RG. The influence of the severity of community‐acquired pneumonia on the usefulness of blood cultures. Respir Med. 2001;95(1):7882.
  7. Walls RM, Resnick J. The Joint Commission on Accreditation of Healthcare Organizations and Center for Medicare and Medicaid Services community‐acquired pneumonia initiative: what went wrong? Ann Emerg Med. 2005;46(5):409411.
  8. Makam AN, Auerbach AD, Steinman MA. Blood culture use in the emergency department in patients hospitalized for community‐acquired pneumonia [published online ahead of print March 10, 2014]. JAMA Intern Med. doi: 10.1001/jamainternmed.2013.13808.
  9. Heckerling PS, Tape TG, Wigton RS, et al. Clinical prediction rule for pulmonary infiltrates. Ann Intern Med. 1990;113(9):664670.
  10. Centers for Disease Control and Prevention. NHAMCS scope and sample design. Available at: http://www.cdc.gov/nchs/ahcd/ahcd_scope.htm#nhamcs_scope. Accessed May 27, 2013.
  11. Centers for Disease Control and Prevention. NHAMCS estimation procedures. http://www.cdc.gov/nchs/ahcd/ahcd_estimation_procedures.htm#nhamcs_procedures. Updated January 15, 2010. Accessed May 27, 2013.
  12. McCaig LF, Burt CW, Schappert SM, et al. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2013;62(5):549551.
  13. Metlay JP, Kapoor WN, Fine MJ. Does this patient have community‐acquired pneumonia? Diagnosing pneumonia by history and physical examination. JAMA. 1997;278(17):14401445.
  14. Kanwar M, Brar N, Khatib R, Fakih MG. Misdiagnosis of community‐acquired pneumonia and inappropriate utilization of antibiotics: side effects of the 4‐h antibiotic administration rule. Chest. 2007;131(6):18651869.
  15. Bates DW, Goldman L, Lee TH. Contaminant blood cultures and resource utilization. The true consequences of false‐positive results. JAMA. 1991;265(3):365369.
  16. Zwang O, Albert RK. Analysis of strategies to improve cost effectiveness of blood cultures. J Hosp Med. 2006;1(5):272276.
  17. Cooper RJ. NHAMCS: does it hold up to scrutiny? Ann Emerg Med. 2012;60(6):722725.
Issue
Journal of Hospital Medicine - 9(8)
Issue
Journal of Hospital Medicine - 9(8)
Page Number
521-524
Page Number
521-524
Publications
Publications
Article Type
Display Headline
Blood culture use in the emergency department in patients hospitalized with respiratory symptoms due to a nonpneumonia illness
Display Headline
Blood culture use in the emergency department in patients hospitalized with respiratory symptoms due to a nonpneumonia illness
Sections
Article Source
© 2014 Society of Hospital Medicine
Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Anil N. Makam, MD, 5323 Harry Hines Blvd., Dallas, TX 75390‐9169; Telephone: 214‐648‐3272; Fax: 214‐648‐3232; E‐mail: anil.makam@utsouthwestern.edu
Content Gating
Gated (full article locked unless allowed per User)
Gating Strategy
First Peek Free
Article PDF Media
Media Files